« Changelog History


Kotest v3.0.x Release Notes

Release Date: 2018-03-29 // about 6 years ago
    • Module split out

    👕 KotlinTest has been split into multiple modules. These include core, assertions, the junit runner, and extensions such as spring, allure and junit-xml.

    🚀 The idea is that in a future release, further runners could be added (TestNG) or for JS support (once multi-platform Kotlin is out of beta). ⬆️ When upgrading you will typically want to add the kotlintest-core, kotlintest-assertions and kotlintest-runner-junit5 to your build ⚡️ rather than the old kotlintest module which is now defunct. When upgrading, you might find that you need to update imports to some matchers.

    testCompile 'io.kotlintest:kotlintest-core:3.0.0'
testCompile 'io.kotlintest:kotlintest-assertions:3.0.0'
testCompile 'io.kotlintest:kotlintest-runner-junit5:3.0.0'

    Gradle Users:

    Also you must include apply plugin: 'org.junit.platform.gradle.plugin' in your project and 🏗 classpath "org.junit.platform:junit-platform-gradle-plugin:1.1.0" to the dependencies section of your buildscript ✅ or tests will not run (or worse, will hang). This allows gradle to execute jUnit-platform-5 based tests (which KotlinTest builds upon). Note: Gradle says that this is not required as of 4.6 but even 👀 with 4.6 it seems to be required.

    Maven users:

    🔌 You need to include the following in your plugins:

    <plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>2.19.1</version>
    <dependencies>
        <dependency>
            <groupId>org.junit.platform</groupId>
            <artifactId>junit-platform-surefire-provider</artifactId>
            <version>1.1.0</version>
        </dependency>
    </dependencies>
</plugin>

    And you must include 

            <dependency>
            <groupId>io.kotlintest</groupId>
            <artifactId>kotlintest-runner-junit5</artifactId>
            <version>${kotlintest.version}</version>
            <scope>test</scope>
        </dependency>

    as a regular dependency.

    • 💥 Breaking: ProjectConfig

    👕 Project wide config in KotlinTest is controlled by implementing a subclass of AbstractProjectConfig. In previous versions you could 👕 call this what you wanted, and place it where you wanted, and KotlinTest would attempt to find it and use it. This was the cause of 👕 many bug reports about project start up times and reflection errors. So in version 3.0.x onwards, KotlinTest will no longer attempt to scan the classpath.

    👕 Instead you must call this class ProjectConfig and place it in a package io.kotlintest.provided. It must still be a subclass of 👕 AbstractProjectConfig This means kotlintest can do a simple Class.forName to find it, and so there is no startup penalty nor reflection issues.

    Project config now allows you to register multiple types of extensions and listeners, as well as setting parallelism.

    • 💥 Breaking: Interceptors have been deprecated and replaced with Listeners

    ✅ The previous inteceptors were sometimes confusing. You had to invoke the continuation function or the spec/test ✅ would not execute. Not invoking the function didn't mean the spec/test was skipped, but that it would hang.

    🚚 So interceptors are deprecated, and in some places removed. Those are not removed are now located in classes called 👀 SpecExtension and TestCaseExtension and those interfaces should be used rather than functions directly.

    Here is an example of a migrated interceptor.

    val mySpecExtension = object : SpecExtension {
    override fun intercept(spec: Spec, process: () -> Unit) {
      println("Before spec!")
      process()
      println("After spec!")
    }
}

    💅 As a replacement, in 3.0.0 we've added the TestListener interface which is the more traditional before/after style callbacks. ✅ In addition, these methods include the result of the test (success, fail, error, skipped) which gives you more ✅ context in writing plugins. The TestListener interface offers everything the old interceptors could do, and more.

    Here is an example of a simple listener.

    object TimeTracker : TestListener {

  var started = 0L

  override fun beforeTest(description: Description) {
    TimeTrackerTest.started = System.currentTimeMillis()
  }

  override fun afterTest(description: Description, result: TestResult) {
    val duration = System.currentTimeMillis() - TimeTrackerTest.started
    println("Test ${description.fullName()} took ${duration}ms")
  }
}

    If you want to use these methods in a Spec itself, then you can just override the functions directly because a Spec is already a TestListener.

    object TimeTracker : WordSpec() {

  var started = 0L

  override fun beforeTest(description: Description) {
    started = System.currentTimeMillis()
  }

  override fun afterTest(description: Description, result: TestResult) {
    val duration = System.currentTimeMillis() - started
    println("Test ${description.fullName()} took ${duration}ms")
  }

  init {
    "some test" should {
      "be timed" {
        // test here
      }
    }
  }
}

    Listeners can be added project wide by overriding listeners() in the ProjectConfig.

    🚀 Note: In the next release, new Extension functions will be added which will be similar to the old interceptors, but with complete control over the lifecycle. For instance, a future intercept method will enforce that the user skip, run or abort a test in the around advice. They will be more complex, and so suited to more advanced use cases. The new TestListener interface will remain of course, and is the preferred option.

    • Parallelism

    If you want to run more than one spec class in parallel, you can by overriding parallelism inside your projects 👕 ProjectConfig or by supplying the system property kotlintest.parallelism.

    Note the system property always takes precedence over the config.

    • 👍 Futures Support

    ✅ Test cases now support waiting on futures in a neat way. If you have a value in a CompletableFuture that you want ✅ to test against once it completes, then you can do this like this:

    val stringFuture: CompletableFuture<String> = ...

"My future test" should {
  "support CompletableFuture<T>" {
    whenReady(stringFuture) {
      it shouldBe "wibble"
    }
  }
}
    • 💥 Breaking: Exception Matcher Changes

    ✅ The shouldThrow<T> method has been changed to also test for subclasses. For example, shouldThrow<IOException> will also match 👕 exceptions of type FileNotFoundException. This is different to the behavior in all previous KotlinTest versions. If you wish to ✅ have functionality as before - testing exactly for that type - then you can use the newly added shouldThrowExactly<T>.

    • JUnit XML Module

    👌 Support for writing out reports in junit-format XML has added via the kotlintest-extensions-junitxml module which you will need to add to your build. This module ✅ provides a JUnitXmlListener which you can register with your project to autowire your tests. You can register this by overriding listeners() in ProjectConfig.

    class ProjectConfig : AbstractProjectConfig() {
    override fun listeners() = listOf(JUnitXmlListener)
}
    • Spring Module

    👕 Spring support has been added via the kotlintest-extensions-spring module which you will need to add to your build. This module ✅ provides a SpringListener which you can register with your project to autowire your tests. You can register this for just some classes by overriding the listeners() function inside your spec, for example:

    class MySpec : ParentSpec() {
    override fun listeners() = listOf(SpringListener)
}

    Or you can register this for all classes by adding it to the ProjectConfig. See the section on ProjectConfig for how to do this.

    • 💥 Breaking: Tag System Property Rename

    👕 The system property used to include/exclude tags has been renamed to kotlintest.tags.include and kotlintest.tags.exclude. Make ⚡️ sure you update your build jobs to set the right properties as the old ones no longer have any effect. If the old tags are detected ⚠ then a warning message will be emitted on startup.

    • 🆕 New Matchers

    ✅ beInstanceOf<T> has been added to easily test that a class is an instance of T. This is in addition to the more verbose beInstanceOf(SomeType::class).

    The following matchers have been added for maps: containAll, haveKeys, haveValues. These will output helpful error messages showing you which keys/values or entries were missing.

    🆕 New matchers added for Strings: haveSameLengthAs(other), beEmpty(), beBlank(), containOnlyDigits(), containADigit(), containIgnoringCase(substring), lowerCase(), upperCase().

    🆕 New matchers for URIs: haveHost(hostname), havePort(port), haveScheme(scheme).

    🆕 New matchers for collections: containNoNulls(), containOnlyNulls()

    • 💥 Breaking: One instance per test changes

    One instance per test is no longer supported for specs which offer nested scopes. For example, WordSpec. This is because of the tricky ✅ nature of having nested closures work across fresh instances of the spec. When using one instance per test, a fresh spec class is required ✅ for each test, but that means selectively executing some closures and not others in order to ensure the correct state. This has proved the largest source of bugs in previous versions.

    👕 KotlinTest 3.0.x takes a simplified approach. If you want the flexibilty to lay out your tests with nested scopes, then all tests will ✅ execute in the same instance (like Spek and ScalaTest). If you want each test to have it's own instance (like jUnit) then you can either ✅ split up your tests into multiple files, or use a "flat" spec like FunSpec or StringSpec.

    This keeps the implementation an order of magnitude simplier (and therefore less likely to lead to bugs) while offering a pragmatic approach to keeping both sets of fans happy.

    • 🆕 New Specs

    Multiple new specs have been added. These are: AnnotationSpec, DescribeSpec and ExpectSpec. Expect spec allows you to use the context ✅ and expect keywords in your tests, like so:

    class ExpectSpecExample : ExpectSpec() {
  init {
    context("some context") {
      expect("some test") {
        // test here
      }
      context("nested context even") {
        expect("some test") {
          // test here
        }
      }
    }
  }
}

    👕 The AnnotationSpec offers functionality to mimic jUnit, in that tests are simply functions annotated with @io.kotlintest.specs.Test. For example:

    class AnnotationSpecExample : AnnotationSpec() {

  @Test
  fun test1() {

  }

  @Test
  fun test2() {

  }
}

    And finally, the DescribeSpec is similar to SpekFramework, using describe, and, and it. This makes it very useful for those people who are looking 👕 to migrate to KotlinTest from SpekFramework.

    class DescribeSpecExample : DescribeSpec() {
  init {
    describe("some context") {
      it("test name") {
        // test here
      }
      describe("nested contexts") {
        and("another context") {
          it("test name") {
            // test here
          }
        }
      }
    }
  }
}
    • ✅ Property Testing with Matchers

    ✅ The ability to use matchers in property testing has been added. Previously property testing worked only with functions that returned a Boolean, like:

    "startsWith" {
  forAll(Gen.string(), Gen.string(), { a, b ->
    (a + b).startsWith(a)
  })
}

    But now you can use assertAll and assertNone and then use regular matchers inside the block. For example:

    "startsWith" {
  assertAll(Gen.string(), Gen.string(), { a, b ->
    a + b should startWith(a)
  })
}

    This gives you the ability to use multiple matchers inside the same block, and not have to worry about combining all possible errors into a single boolean result.

    • Generator Edge Cases

    Staying with property testing - the Generator interface has been changed to now provide two types of data.

    The first are values that should always be included - those edge cases values which are common sources of bugs. For example, a generator for Ints should always include values like zero, minus 1, positive 1, Integer.MAX_VALUE and Integer.MIN_VALUE. Another example would be for a generator for enums. That should include all the values of the enum to ensure ✅ each value is tested.

    ✅ The second set of values are random values, which are used to give us a greater breadth of values tested. The Int generator should return random ints from across the entire integer range.

    👀 Previously generators used by property testing would only include random values, which meant you were very unlikely to see the edge cases that usually cause issues - like the aforementioned Integer MAX / MIN. Now you are guaranteed to get the edge cases first and the random values afterwards.

    • 💥 Breaking: MockitoSugar removed

    🤡 This interface added a couple of helpers for Mockito, and was used primarily before Kotlin specific mocking libraries appeared. 🏗 Now there is little value in this mini-wrapper so it was removed. Simply add whatever mocking library you like to your build and use it as normal.

    • CsvDataSource

    ✅ This class has been added for loading data for table testing. A simple example:

    class CsvDataSourceTest : WordSpec() {
  init {

    "CsvDataSource" should {
      "read data from csv file" {

        val source = CsvDataSource(javaClass.getResourceAsStream("/user_data.csv"), CsvFormat())

        val table = source.createTable<Long, String, String>(
            { it: Record -> Row3(it.getLong("id"), it.getString("name"), it.getString("location")) },
            { it: Array<String> -> Headers3(it[0], it[1], it[2]) }
        )

        forAll(table) { a, b, c ->
          a shouldBe gt(0)
          b shouldNotBe null
          c shouldNotBe null
        }
      }
    }
  }
}
    • Matcher Negation Errors

    👍 All matchers now have the ability to report a better error when used with shouldNot and shouldNotBe. Previously a generic error was generated - which was usually the normal error but with a prefix like "NOT:" but now each built in matcher will provide a full message, for example: Collection should not contain element 'foo'