linux中sbt 中单元测试并发执行例子

此次研究的目的原本是要使得 Play Framwork 2 中单元测试能够并发执行, 包括 JUnit 和 Spec 的测试用例, Play 2 的 activator 就是一个 sbt 的包装. 开发中发现我们 Play 2 中的单元测试是按序执行的, 实际上 sbt 下测试用例默认是并发执行的. 之所以 Play 2 的单元测试是按序的, 是因为 activator 设置了把 sbt 的两个属性 fork in Test := true 和 parallelExecution in Test := false, 见 PlaySettings.scala, 它们默认分别为 false 和 true. 这使得默认设置下 Play 2 中的所有测试无法并发执行.

sbt 默认的 fork 是 false, Play 2 改为 true 之后便可以使用 javaOptions in Test := "-Dkey1=value1" (注: 如果 fork 为 false 的话, javaOptions 将无效.) 往单元测试中参数了, 这也是为什么在 Play 2 的单元测试中无法获得启动 sbt 时(像 sbt -Dkey1=value) 的参数, 不同一个 JVM 啊.

那是不把 Play 2 的 fork in Test 和 parallelExecution in Test 分别改回成 false 和 true 就可以让测试用例并发执行了呢? 答案是 Yes. 但我们得相信 Play 2 把它们预设为 true 和 false 是有它的用意的, 比如集成测试的每个用例都会开启本地的 3333 端口, 如果让两个集成测试同时执行将会造成端口冲突. 细致说来, Play 2 其实是懒政, 只管一刀切而让所有测试按序执行而影响了效率, 如能利用好 sbt 的测试分组机制是可以达到测试的并发执行的.

这里引出 sbt 执行测试的几个机制:

1) sbt 总是对测试进行分组, 默认时所有的测试都包含在 组中, 可用 show testGrouping 查看, 如

fork in Test := false 时

> show test:fork
[info] false
> show testGrouping
[info] List(Group(,List(Test model.BookSpecTest : subclass(true, org.specs2.specification.core.SpecificationStructure), Test model.BookTest : annotation(false, org.junit.Test), Test service.BookServiceSpecTest : subclass(true, org.specs2.specification.core.SpecificationStructure), Test service.BookServiceTest : annotation(false, org.junit.Test)),InProcess))

fork in Test := true 时

> show test:fork
[info] true
> show testGrouping
[info] List(Group(,List(Test model.BookSpecTest : subclass(true, org.specs2.specification.core.SpecificationStructure), Test model.BookTest : annotation(false, org.junit.Test), Test service.BookServiceSpecTest : subclass(true, org.specs2.specification.core.SpecificationStructure), Test service.BookServiceTest : annotation(false, org.junit.Test)),SubProcess(ForkOptions(None,None,List(),Some(/Users/uqiu/Workspaces/test_in_parallel),List(-Dfrom.sbt.javaOptions=valueFromSbtJavaOptions),false,Map()))))

从上面看出 fork in Test 的取值直接影响了默认分组的运行策略, 是 InProcess 还是 SubProcess. 我们可以自定义分组, 如果是自定义分组里指定了 InProcess 或 SubProcess, 那么 fork in Test 的值将被忽略. javaOptions in Test 用来向 fork in Test := true 时默认分组传递 JVM 参数.

sbt 测试说到底还是分组执行, 组内自定义测试的运行策略. 现在我们懂得了 fork in Test 的功用, 如果是自定义了测试组它将一无所是. 再看另外两个属性的功能

1) parallelExecution in Test 是并发的一个总开关, 值为 true 时可让 InProcess 组内测试并发执行, 或让多个 SubProcess 组间并发, 组内按序执行.

2) 另一个实验中的设置 testForkedParallel in Test 相当的猛, 如果设置为 true 时可以让 InProcess 和 SubProcess 类型的组内并发执行当 for in Test 为 true 时, 但它受到 parallelExecution in Test 的约束, 前面说了它是个总开发.

我们来看一下 sbt 和 Play 2 的默认行为:

sbt 默认时 fork in Test := false 和 parallelExecution in Test := true, 所有测试分配在一个 InProcess 组中, 所以该组内的测试是并发执行的. 该分组在 sbt 所在 JVM 中执行, 所以能读取到启动 sbt 时的 JVM 参数.

Play 2 把这两个值分别设置为 for in Test := true 和 parallelExecution in Test := false, 所有测试分组在一个 SubProcess 的组中. 组内的测试将在 fork 的新 JVM 中执行, 该分组直接用 javaOptions in Test 定义作为 SubProcess 的 JVM 参数. 我们可以设置 parallelExecution in Test := true 来达到组间并发执行, 由于此时默认只有一个分组, 所以即使它为 true 所能看到的也是所有测试按序执行.

因此为达到 Play 2 的测试并发执行, 我们必须对所有测试实施自定义分组. 若为 SubProcess 类型的分组指定了 ForkOption 的话, javaOptions in Test 将失效. 当然我们可以通过简单的把 fork in Test, parallelExecution in Test, 和 testForkedParallel in Test 同时设为 true, 实现了分组内的测试也能并发执行, 但这种过高的并发会带来不可控性, 比如对唯一资源的抢占冲突, 因而暂不推荐使用.

如此一来, 最好以两回合来分解, 一为 fork 为 false 时, 二为 fork 和 parallelExecution 同为 true 时, 现在暂作一处, 有空再理. 接着当然是前面所述的两部份:

注: 本文采用 sbt 版本是 0.13.11, Scala 版本是 2.11.7.

一: sbt 默认 fork in Test := false 时测试用例在 sbt 所在 JVM 中并发执行

通过一个简单的 sbt 项目来体验, 该示例中包含有 JUnit 和 Scala Spec 两种类型的测试. 本例的目录及文件如下

├── build.sbt
└── test
├── model
│ ├── BookSpecTest.scala
│ └── BookTest.scala
└── service
├── BookServiceSpecTest.scala
└── BookServiceTest.java

由于只涉及到单元测试, 所以只有 test 目录, 文件内容依次如下:

build.sbt

name := "test-in-parallel"
scalaVersion := "2.11.7"

libraryDependencies ++= Seq(
"org.specs2" %% "specs2-core" % "3.7.2" % "test",
"com.novocode" % "junit-interface" % "0.11" % "test"
)

javaSource in Compile := new File(baseDirectory.value, "app")
scalaSource in Compile := new File(baseDirectory.value, "app")

javaSource in Test := new File(baseDirectory.value, "test")
scalaSource in Test := new File(baseDirectory.value, "test")

testOptions += Tests.Argument(TestFrameworks.JUnit, "-q", "-v")

javaOptions in Test += "-Dfrom.sbt.javaOptions=valueFromSbtJavaOptions"
引入了 Spec 和 JUnit 测试依赖, 把产品和测试代码分别指定到 app 和 test 目录, 学的 Play 2 的样, javaOptions 行是用来验证它在 fork 为 false 时是否有效.

test/model/BookTest.scala

package model

import org.junit.Test

class BookTest {

@Test
def testCreateBook1: Unit = {
BookTest.print("BookTest#testCreateBook1")
}

@Test
def testCreateBook2: Unit = {
BookTest.print("BookTest#testCreateBook2")
}
}

object BookTest {
def print(name: String): Unit = {
(1 to 3).foreach { n =>
System.err.println("from " + name + " " + Thread.currentThread() + " " + n)
Thread.sleep(500)
}
}
}
这里声明了一个辅助方法 BookTest.print(name) 来循环加延时输出当前线程名, 会被各个测试调用, 由此来观察用例是如何被执行.

本文原始链接 http://unmi.cc/sbt-tests-execute-in-parallel/, 来自 隔叶黄莺 Unmi Blog
test/model/BookSpecTest.scala

package model

import org.specs2.mutable.Specification

object BookSpecTest extends Specification {

"BookSpecTest" should {
"looks like this demo" in {
BookTest.print(getClass.getName)
"a" === "a"
}
}
}
test/service/BookServiceTest.java

package service;

import org.junit.Test;

import model.BookTest;

public class BookServiceTest {

@Test
public void testGetBookById() {
System.err.println("Property from.sbt.cmd " + System.getProperty("from.sbt.cmd"));
System.err.println("Property from.sbt.javaOptions " + System.getProperty("from.sbt.javaOptions"));
System.err.println("Property from.sbt.testGroup " + System.getProperty("from.sbt.testGroup"));

BookTest.print(getClass().getName());
}
}
这里试图从 sbt 启动命令行或 sbt 的 javaOptions 配置中读取系统属性, 从而验证测试用例是在新的 JVM 中还是 sbt 所在的 JVM 中执行

test/service/BookServiceSpecTest.scala

package service

import model.BookTest
import org.specs2.mutable.Specification

object BookServiceSpecTest extends Specification {

"BookServiceSpecTest" should {
"looks like this demo" in {
BookTest.print(getClass.getName)
"a" === "a"
}
}
}
好了, 我们现在用下面的命令来启来 sbt

sbt -Dfrom.sbt.cmd=ValueFromSbtCmd

再执行几个 sbt 任务观察它的输出为

[I] ➜ test_in_parallel git:(master) ✗ sbt -Dfrom.sbt.cmd=ValueFromSbtCmd
[info] Set current project to test-in-parallel (in build file:/Users/uqiu/Workspaces/bitbucket/sbt-in-action/chapter4/test_in_parallel/)
> show test:fork
[info] false
> show test:parallelExecution
[info] true
> show test:javaOptions
[info] List(-Dfrom.sbt.javaOptions=valueFromSbtJavaOptions)
[success] Total time: 0 s, completed Apr 12, 2016 11:47:35 PM
> test
[warn] javaOptions will be ignored, fork is set to false
Property from.sbt.cmd ValueFromSbtCmd
Property from.sbt.javaOptions null
Property from.sbt.testGroup null
from BookTest#testCreateBook1 Thread[pool-3-thread-5,5,main] 1
from service.BookServiceTest Thread[pool-3-thread-9,5,main] 1
from BookTest#testCreateBook1 Thread[pool-3-thread-5,5,main] 2
from service.BookServiceTest Thread[pool-3-thread-9,5,main] 2
from model.BookSpecTest$ Thread[specs2.fixed.env-300074353-7,5,main] 1
from service.BookServiceSpecTest$ Thread[specs2.fixed.env-842134031-7,5,main] 1
from service.BookServiceTest Thread[pool-3-thread-9,5,main] 3
from BookTest#testCreateBook1 Thread[pool-3-thread-5,5,main] 3
from model.BookSpecTest$ Thread[specs2.fixed.env-300074353-7,5,main] 2
from service.BookServiceSpecTest$ Thread[specs2.fixed.env-842134031-7,5,main] 2
from BookTest#testCreateBook2 Thread[pool-3-thread-5,5,main] 1
[info] Test run started
[info] Test service.BookServiceTest.testGetBookById started
[info] Test run finished: 0 failed, 0 ignored, 1 total, 1.53s
from model.BookSpecTest$ Thread[specs2.fixed.env-300074353-7,5,main] 3
from service.BookServiceSpecTest$ Thread[specs2.fixed.env-842134031-7,5,main] 3
from BookTest#testCreateBook2 Thread[pool-3-thread-5,5,main] 2
from BookTest#testCreateBook2 Thread[pool-3-thread-5,5,main] 3
[info] BookServiceSpecTest
[info]
[info] BookServiceSpecTest should
[info] + looks like this demo
[info]
[info]
[info] Total for specification BookServiceSpecTest
[info] Finished in 1 second, 540 ms
[info] 1 example, 0 failure, 0 error
[info]
[info] BookSpecTest
[info]
[info] BookSpecTest should
[info] + looks like this demo
[info]
[info]
[info] Total for specification BookSpecTest
[info] Finished in 1 second, 559 ms
[info] 1 example, 0 failure, 0 error
[info]
[info] Test run started
[info] Test model.BookTest.testCreateBook1 started
[info] Test model.BookTest.testCreateBook2 started
[info] Test run finished: 0 failed, 0 ignored, 2 total, 3.036s
[info] Passed: Total 5, Failed 0, Errors 0, Passed 5
[success] Total time: 4 s, completed Apr 12, 2016 11:47:42 PM

从前面的输出可以发觉所有的测试都是并发执行的, 测试中可以取得 sbt 的系统属性, 基本上证明测试是在 sbt 所在的 JVM 中执行. 如果设置 parallelExecution in Test := false 的话将禁上这种 fork 为 false 时的默认为测试并发执行的行为. 也就是说 fork 为 false, parallelExecution 为 false 时测试类将按序执行.

那如果我们在 build.sbt 中加上

fork in Test := true

会怎么样呢? 在上一个 sbt 控制台中 reload 后, 再执行 test 看看

> reload
[info] Set current project to test-in-parallel (in build file:/Users/uqiu/Workspaces/bitbucket/sbt-in-action/chapter4/test_in_parallel/)
> show test:fork
[info] true
> test
[info] Updating {file:/Users/uqiu/Workspaces/bitbucket/sbt-in-action/chapter4/test_in_parallel/}test_in_parallel...
[info] Resolving jline#jline;2.12.1 ...
[info] Done updating.
from model.BookSpecTest$ Thread[specs2.fixed.env994707824-7,5,main] 1
from model.BookSpecTest$ Thread[specs2.fixed.env994707824-7,5,main] 2
from model.BookSpecTest$ Thread[specs2.fixed.env994707824-7,5,main] 3
[info] BookSpecTest
[info]
[info] BookSpecTest should
[info] + looks like this demo
[info]
[info]
[info] Total for specification BookSpecTest
[info] Finished in 1 second, 543 ms
[info] 1 example, 0 failure, 0 error
[info]
from service.BookServiceSpecTest$ Thread[specs2.fixed.env-1284062022-7,5,main] 1
from service.BookServiceSpecTest$ Thread[specs2.fixed.env-1284062022-7,5,main] 2
from service.BookServiceSpecTest$ Thread[specs2.fixed.env-1284062022-7,5,main] 3
[info] BookServiceSpecTest
[info]
[info] BookServiceSpecTest should
[info] + looks like this demo
[info]
[info]
[info] Total for specification BookServiceSpecTest
[info] Finished in 1 second, 509 ms
[info] 1 example, 0 failure, 0 error
[info]
[info] Test run started
[info] Test model.BookTest.testCreateBook1 started
from BookTest#testCreateBook1 Thread[pool-1-thread-1,5,main] 1
from BookTest#testCreateBook1 Thread[pool-1-thread-1,5,main] 2
from BookTest#testCreateBook1 Thread[pool-1-thread-1,5,main] 3
[info] Test model.BookTest.testCreateBook2 started
from BookTest#testCreateBook2 Thread[pool-1-thread-1,5,main] 1
from BookTest#testCreateBook2 Thread[pool-1-thread-1,5,main] 2
from BookTest#testCreateBook2 Thread[pool-1-thread-1,5,main] 3
[info] Test run finished: 0 failed, 0 ignored, 2 total, 3.026s
[info] Test run started
[info] Test service.BookServiceTest.testGetBookById started
Property from.sbt.cmd null
Property from.sbt.javaOptions valueFromSbtJavaOptions
Property from.sbt.testGroup null
from service.BookServiceTest Thread[pool-1-thread-1,5,main] 1
from service.BookServiceTest Thread[pool-1-thread-1,5,main] 2
from service.BookServiceTest Thread[pool-1-thread-1,5,main] 3
[info] Test run finished: 0 failed, 0 ignored, 1 total, 1.51s
[info] Passed: Total 5, Failed 0, Errors 0, Passed 5
[success] Total time: 10 s, completed Apr 12, 2016 11:56:15 PM

不难发现当 fork in Test := true 之后, 所有的测试类将按序同步执行, 耗费更多的时间. 测试中获取不到 sbt 的系统属性, 可以得到通过 javaOptions 设置的系统属性, 即测试用例跑在一个新的 JVM 中.

我们该如何让测试用例在 fork 的 JVM 中并发执行了, 由此引出我们第二个议题

二. fork in Test := true 时用例分组并发执行

设置了 fork 为 true 之后所有的测试类将在一个 fork 的 JVM 中按序同步执行. 我们说了 fork 的 JVM 中并发执行的粒度是用例组 -- 组内按序, 组间并发. 如果没有自定义分组的话所有测试在一个默认分组中, 也就是说全部用例按序执行, 可以去掉下面的的 testGrouping in Test <<= definedTests in Test map groupByModule 这行验证一下. 本例中我们按所在不同的包分成 model 和 service 两个组.

因此, 新的 build.sbt 文件内容将如下:

import sbt.Tests.{Group, SubProcess}

name := "test-in-parallel"
scalaVersion := "2.11.7"

libraryDependencies ++= Seq(
"org.specs2" %% "specs2-core" % "3.7.2" % "test",
"com.novocode" % "junit-interface" % "0.11" % "test"
)

javaSource in Compile := new File(baseDirectory.value, "app")
scalaSource in Compile := new File(baseDirectory.value, "app")

javaSource in Test := new File(baseDirectory.value, "test")
scalaSource in Test := new File(baseDirectory.value, "test")

testOptions += Tests.Argument(TestFrameworks.JUnit, "-q", "-v")

parallelExecution in Test := true
fork in Test := true

javaOptions in Test += "-Dfrom.sbt.javaOptions=valueFromSbtJavaOptions"

def groupByModule(tests: Seq[TestDefinition]) = {
tests groupBy {test =>
test.name.split("\.")(0) //grouped by top package name
} map {
case (name, tests) => Group(name, tests,
SubProcess(ForkOptions(
runJVMOptions = Seq(
"-Dfrom.sbt.testGroup=valueFromSbtTestGroup " + name
)
)))
} toSeq
}

testGrouping in Test <<= definedTests in Test map groupByModule
再回到 sbt 控制台下 reload, 然后 test 一下

> reload
[info] Set current project to test-in-parallel (in build file:/Users/uqiu/Workspaces/bitbucket/sbt-in-action/chapter4/test_in_parallel/)
> show test:fork
[info] true
> show test:javaOptions
[info] List(-Dfrom.sbt.javaOptions=valueFromSbtJavaOptions)
[success] Total time: 0 s, completed Apr 13, 2016 12:30:22 AM
> show testGrouping
[info] List(Group(model,List(Test model.BookSpecTest : subclass(true, org.specs2.specification.core.SpecificationStructure), Test model.BookTest : annotation(false, org.junit.Test)),SubProcess(ForkOptions(None,None,List(),None,List(-Dfrom.sbt.testGroup=valueFromSbtTestGroupmodel),false,Map()))), Group(service,List(Test service.BookServiceSpecTest : subclass(true, org.specs2.specification.core.SpecificationStructure), Test service.BookServiceTest : annotation(false, org.junit.Test)),SubProcess(ForkOptions(None,None,List(),None,List(-Dfrom.sbt.testGroup=valueFromSbtTestGroupservice),false,Map()))))
[success] Total time: 0 s, completed Apr 13, 2016 12:30:27 AM
> test
from model.BookSpecTest$ Thread[specs2.fixed.env-1444496425-7,5,main] 1
from service.BookServiceSpecTest$ Thread[specs2.fixed.env994707824-7,5,main] 1
from model.BookSpecTest$ Thread[specs2.fixed.env-1444496425-7,5,main] 2
from service.BookServiceSpecTest$ Thread[specs2.fixed.env994707824-7,5,main] 2
from service.BookServiceSpecTest$ Thread[specs2.fixed.env994707824-7,5,main] 3
from model.BookSpecTest$ Thread[specs2.fixed.env-1444496425-7,5,main] 3
[info] BookSpecTest
[info] BookServiceSpecTest
[info]
[info]
[info] BookSpecTest should
[info] BookServiceSpecTest should
[info] + looks like this demo
[info] + looks like this demo
[info]
[info]
[info] Total for specification BookSpecTest
[info] Finished in 1 second, 550 ms
[info] 1 example, 0 failure, 0 error
[info]
[info]
[info]
[info] Total for specification BookServiceSpecTest
[info] Finished in 1 second, 536 ms
[info] 1 example, 0 failure, 0 error
[info]
[info] Test run started
[info] Test model.BookTest.testCreateBook1 started
from BookTest#testCreateBook1 Thread[pool-1-thread-1,5,main] 1
[info] Test run started
[info] Test service.BookServiceTest.testGetBookById started
Property from.sbt.cmd null
Property from.sbt.javaOptions null
Property from.sbt.testGroup valueFromSbtTestGroup service
from service.BookServiceTest Thread[pool-1-thread-1,5,main] 1
from BookTest#testCreateBook1 Thread[pool-1-thread-1,5,main] 2
from service.BookServiceTest Thread[pool-1-thread-1,5,main] 2
from BookTest#testCreateBook1 Thread[pool-1-thread-1,5,main] 3
from service.BookServiceTest Thread[pool-1-thread-1,5,main] 3
[info] Test model.BookTest.testCreateBook2 started
from BookTest#testCreateBook2 Thread[pool-1-thread-1,5,main] 1
[info] Test run finished: 0 failed, 0 ignored, 1 total, 1.52s
from BookTest#testCreateBook2 Thread[pool-1-thread-1,5,main] 2
from BookTest#testCreateBook2 Thread[pool-1-thread-1,5,main] 3
[info] Test run finished: 0 failed, 0 ignored, 2 total, 3.032s
[info] Passed: Total 5, Failed 0, Errors 0, Passed 5
[success] Total time: 6 s, completed Apr 13, 2016 12:30:43 AM

这时候测试类是并发执行是没问题的, 要细心点去才能从结果中看出组内是按序执行, 组与组是并发执行的. 测试类自定义分组后用 javaOptions in Test 定义的 JVM 参数都不可用, 而必须为每个分组单独定义 JVM 参数, 这说明了 sbt 为每一个测试分组启动了单独的 JVM. 原理上我们可以为每一个测试类建立一个单独的组, 那将要求 fork 很多的 JVM, 需实际考虑效果. 上面结果也能看出测试类分组后运行时间也有减少.

视情况设置 Tags.ForkedTestGroup, 默认为 1, 但好像也能同时处理 2 个分组的. 它的设置方法如下

concurrentRestrictions in Test := Seq(
Tags.limit(Tags.ForkedTestGroup, 8)
// Tags.limit(Tags.Test, 8),
// Tags.limit(Tags.All, 8)
)