[Spark] Scala Test (Library, Sbt Test)

• spark-testing-base
• scala-test
• scala-check

"org.scalatest" %% "scalatest" % "3.0.1" % "test",
"org.scalacheck" %% "scalacheck" % "1.13.4" % "test",
"com.holdenkarau" %% "spark-testing-base" % "2.3.0_0.9.0" % "test"

• spark-test-samples

sbt 테스트

• 아래 명령어를 입력하면 Test 전체를 수행한다

sbt assembly

• 아래 명령도 전체를 Test를 수행

sbt
> test

• build하지 않고 테스트만

sbt test

• 특정 테스트만 실행하기 (build를 따로 할필요가 없음)

 sbt "testOnly package.TestClass"

test를 진행할때마다 잘보면 아래와 같이 뜨는데... 테스트 마다 Session을 생성하는듯 (매우느림)

20/02/28 12:18:33 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/02/28 12:18:34 WARN SparkContext: Using an existing SparkContext; some configuration may not take effect.

이때 Spark의 경우 Session을 생성을 해야하기 때문에 sbt tool에 들어가서 test를 하면 매번 Session을 생성해서 불필요한 시간을 사용하게 된다.

Spark Test Sample

Project Main Modules:

• SharedSparkContext: Provides SparkContext to be used in testing.
• RDDComparisons: Compares two RDDs.
• DataFrameSuiteBase: Checks for dataframe equality.
• DatasetSuiteBase: Checks for dataset equality.
• StreamingSuiteBase: Applies given operation on given input stream and compare with expected output.
• StreamingActionBase: Tests actions that don't have output on input streams.
• RDDGenerator: Generates arbitrary RDDs.
• DataFrameGenerator: Generates arbitrary DataFrames.
• DatasetGenerator: Generates arbitrary Datasets.

참고

• https://medium.com/@mrpowers/creating-a-spark-project-with-sbt-intellij-sbt-spark-package-and-friends-cc9108751c28
• https://www.scala-sbt.org/1.x/docs/Testing.html