Spock maintainer Rob Fletcher demonstrates the current state of Kotlin testing frameworks by comparing it to Spek, JetBrains’s twist on a specification framework. As an experienced tester, Rob highlights the pros and cons of both libraries, with his wishes for improvements for Spek as well as what it excels at. Leverage Kotlin to vastly simplify some iterative testing and mocking.

Testing with Kotlin (0:00)

Hi, my name’s Rob Fletcher and we’re here to compare some interesting use cases that we see between different test frameworks, and how we can achieve similar things in Spek.

I’m a developer at Netflix, I’m not using Kotlin day-to-day, although I’d love to be. Currently I’m actually writing a book on Spock, the testing framework for Groovy. Some of the stuff I’m going to be showing you are some really cool features of Spock, and how we can or can’t achieve similar things, or can do things even better in Spek.

Let’s dive in. We will cover three primary areas: iterative tests, mocks, and TCKs.

Iterative Tests (1:05)

Iteration - Spock (1:23)

First we’re going to look at iterative tests. There are nice nested BDD structures that you can use with Spock there. Let’s look at how we can deal with tests that are parameterized, like this:

def "the diamond function rejects illegal characters"() {  
    when: 
    diamond.of(c) 
 
    then: 
    thrown IllegalArgumentException 
    
    where: 
    c << generator.chars() 
        .findAll { !('A'..'Z').contains(it) } 
        .take(100) 
}

Here’s a fairly typical Spock where we have something called a where block that has a parameter in it. It is driven using this funky left-shift syntax from an iterable source of data.

What’ll happen is this test will execute once per value that gets fed into this variable here, and that value can be used throughout the test. This is great for doing this kind of property-based testing, or anything where you’ve got a specific set of values that you need to cover testing edge cases with.

Iteration - Spek (2:05)

Now for a quick example of how we can handle something similar in Spek:

describe("handling invalid characters") { 
    
  chars().filter { it !in 'A'..'Z' }  
         .take(100) 
         .forEach { c ->
    
    it("throws an exception") 
      assertFailsWith(IllegalArgumentException::class) { 
        diamond.of(c)  
      }
    }
}

I love the simplicity of this because it reminds me a lot of Jasmine. I’ve done a bunch of front-end work in my time, and I always liked Jasmine and never found anything on the JVM that had a similar structure. Spek can now provide that.

Inside of our describe block, we can have just a for-each iteration. We can have a describe or a context or any level of that going on inside of that iteration. Therefore this test, defined by this it block, will be ran once per value that’s coming through there, and if you look at it in the IDE or on the command-line runner, you’ll see one separate test output per value.

Tokenized Test Names – Spock (3:07)

One of the nice things, obviously, when you do that you’ll get the same it every time, which is not ideal. One of the really nice things Spock lets you do is add this Unroll annotation onto your test and then you can refer in the name of the test to those values, and then when your report gets generated or your IDE test runner executes, you can see which, if one of those individual ones has failed you can see which one it was.

@Unroll
def "the diamond function rejects #c"() {  
  when: 
  diamond.of(c) 

  then: 
  thrown IllegalArgumentException 
  
  where: 
  c << generator.chars() 
                .findAll { !('A'..'Z').contains(it) } 
                .take(100) 
}

Tokenized Test Names & Nesting – Spek (3:27)

In Spek this is even easier, because you can just use string interpolation in your describe or your it block. Fantastically easy:

describe("handling invalid characters") { 
    
  chars().filter { it !in 'A'..'Z' }  
         .take(100) 
         .forEach { c ->
    
    it("throws an exception for '$c'") 
      assertFailsWith(IllegalArgumentException::class) { 
        diamond.of(c)  
      }
    }
}

One of the nice things you can do with Spek that you really can’t do with Spock, and this is something I’ve bashed my head off trying to do, is this dumb little diamond cutter. If anyone’s familiar with that puzzle, having this nested iteration where you’ve got one set of tests that need to run at a certain level of iteration and then some others inside of that.

('B'..'Z').forEach { c ->
  describe("the diamond of '$c'") { 
    val result = diamond.of(c) 
    
    it("is square") { 
      assert(result.all { it.length == rows.size }) 
    }
 
    ('A'..c).forEach { rowChar -> 
      val row = rowChar - 'A'
      val col = c - rowChar 

      it("has a '$rowChar' at row $row column $col") {  
        assertEquals(rowChar, result[row][col])
      }
    }
  }
}

Here we want to test generating for these different characters, but inside of that we want to verify that certain other characters appear in a pattern. So you can nest the iteration which is something you just cannot do with that Spock structure that I showed you there, where you’re really just got one level of parameterization.

Tabular Iteration - Spock (4:25)

One thing Spock does really nicely that Spek doesn’t do yet is doing tabular data. Instead of that left-shift operator here that we saw, Spock can let you define what’s called a data table where you can define multiple variables in these column headings, and they provide tables of data underneath.

@Unroll
def "the diamond of #c has #height rows"() {
  expect:
  diamond.of(c).size == height

  where:
  c | height
  'A' | 1
  'B' | 3
  'C' | 5 
}

The really cool thing is that IntelliJ IDEA will align those tables properly for you as well. Then you can have multiple parameters going into your tests. Right now Spek doesn’t have an awesome way to do that I don’t think, but I know it’s on the to-do list.

The best you can do right now is do something like this:

for ((c, height) in mapOf('A' to 1, 'B' to 2, 'C' to 3)) {  
  describe("the diamond of '$c") { 
    val result = diamond(c)
    it("has $height rows") { 
      assertEquals(height, result.size)
    }  
  }
}

We get away with it here because we’ve only got two dimensions of data, we can iterate over a map, and compose our describe and it blocks inside of that. If you’ve got three or more dimensions, it starts to get a lot trickier, although you can certainly imagine having some kind of data class, and iterating over a collection of those. It’s doable but it’s not as neat as the way you can do it in Spock.

Mocks - Mockito Meets Kotlin (5:45)

Mocks is an interesting subject with Kotlin generally and Spek particularly, for a couple of reasons. So let’s look at an attempt to make a Spek class use Mockito.

describe("publishing events") { 
  val eventBus = EventBus() 
  val subscriber = mock(Subscriber::class.java) as Subscriber<ExampleEvent> // can't infer generic type
  beforeEach { 
    eventBus.post(ExampleEvent("test running"))
  } 
 
  it("should send the event to the subscriber") { 
    verify(subscriber)  
      .onEvent(argThat {  // returns null so fails at runtime
        (it as ExampleEvent).description == "test running" // can't infer generic matcher argument
      }) 
  }
}

We have a couple of awkward things here. Firstly we have to do this really clunky, Java-like cast, because we have generic types on the type we’re trying to mock, and Kotlin is not aware of that at runtime because the types get erased. So we have to do the cast again inside of our assertion, our verify on the Mockito mock object.

We don’t have type inference supported so we lose all our generic type information and it’s kind of unpleasant.

To top it all off it just doesn’t work anyway because Mockito’s matches return null, and those will fail at run-time because Kotlin strictly null-checks lots of things.

There’s no Mock framework I’m aware of for Kotlin, but there certainly is some Mockito support distinctly for Kotlin so if you add this library to your Gradle build, you get a Kotlin version of those same Mockito structures, which is now a whole lot more concise.

repositories { 
  jcenter()  
} 
 
dependencies { 
  compile "com.nhaarman:mockito-kotlin:0.4.1"  
}

describe("publishing events") { 
  val eventBus = EventBus() 
  val subscriber: Subscriber<Event> = mock()  // reified generics allow type inference
 
  beforeEach { 
    eventBus.post(Event("test running"))  
  } 
 
  it("should send the event to the subscriber") { 
    verify(subscriber).onEvent(argThat { // returns dummy value
      description == "test running" // type inference on receiver
    })
  }
}

We’ve got type inference going on on the mocks so the fact that the variable subscriber has got a declared type with a generic type, Kotlin’s reified generics allow us to figure out what that type is inside of the mock factory there; you don’t need to specify the class, you don’t lose any of the generic type information, you don’t need to cast anything.

Inside of the verification block we’ve got a receiver defined so we don’t need to say it. on the arguments that go into that matcher, and we have type inference again so we don’t have to cast this argument.

This is much much nicer, and the matcher itself returns a dummy value, so we don’t have that nullability problem anymore. This is a very small library, and it eases the path to using Mockito in Kotlin and Spek. I would highly recommend you look at that if you’re using any kind of test doubles.

TCKs - One Test For All (8:13)

TCKs are Technology Compatibility Kits. The main idea behind them is that in a case of multiple implementations of something that all need to conform to a specific set of rules.

A simple example would be Java’s List. There’s ArrayList, there’s LinkedList, there’s a whole bunch of different implementations. But they all have to conform to a certain set of rules, like they return things in insertion order, some of them allow nulls, some of them don’t, but their equals and hash codes are consistent. There’re a whole bunch of consistent rules that have to apply.

As such, it’s really nice to be able to write tests for those things once, and then test all of your own implementations against the same set of tests without having to rewrite them.

TCK - Spock (9:01)

If we look at how Spock would handle this, you would typically declare an abstract test class, have some kind of abstract factory method that produces your class under test, write a bunch of tests that access that object, and then you will extend them with concrete classes that simply override that one factory method.

abstract class PubSubSpec extends Specification {
  @Subject abstract EventBus factory() 
  // ... tests appear here
}

class SyncPubSubSpec extends PubSubSpec {  
  @Override EventBus factory() { 
    new EventBus()  
  } 
}   

class AsyncPubSubSpec extends PubSubSpec {  
  @Override EventBus factory() { 
    new AsyncEventBus(newSingleThreadExecutor())  
  } 
}

They can also add their own tests if there is extended functionality for that particular implementation. Nevertheless, the idea is that when you execute your test suite, all of the tests defined in this abstract base class run for this subclass and for this subclass and for any others you’ve got.

Very quickly you build up a suite of acceptance tests for implementation of an interface.

TCK - Spek (10:19)

Now one of the things that’s interesting with Spek is that the tests themselves are defined in something of a static initializer in a Java class. Now you don’t have methods you can override to do this type of thing.

abstract class PubSubTck(val eventBus: EventBus) : Spek ({ 
  describe("publishing events") { 
    // ... tests appear here  
  } 
}) 
 
class SyncPubSubSpec  
: PubSubTck(EventBus()) 
 
class AsyncPubSubSpec  
: PubSubTck(AsyncEventBus(newSingleThreadExecutor()))

As such you can define class-level methods in a Spek class but the tests can’t see them, it can only see things that are in the companion object. Obviously companion objects like Java statics can’t have inheritance.

One solution to this is again to use an abstract class that extends the Spek base class, but define a property for your eventBus.

You may now define a property that implementations need to provide to the constructor, then write all your test DSL as normal. This way you can have concrete extensions of that that simply pass a value to that constructor, which is then accessible from within the test.

That’s pretty good, as long as you can express the constructing of your class under test in a sufficiently concise form that it goes in a constructor like that. If you can’t you probably want to use a factory method of some kind, and a lambda is an obvious solution so you can instead say, have again an abstract Spek that takes a factory function that goes from an empty parameter set to your class under test, then just have a value in there that for each individual test could create a new instance if you prefer, but have a value that calls on that factory function, then the extension classes for testing specific implementations just provide that factory function.

abstract class PubSubTck(val factory: () -> EventBus) : Spek ({ 
  val eventBus = factory() 
  describe("publishing events") { 
    // ... tests appear here  
  } 
}) 
 
class SyncPubSubFactorySpec  
: PubSubTck(::EventBus) 
 
class AsyncPubSubFactorySpec  
: PubSubTck({ AsyncEventBus(newSingleThreadExecutor()) })

So here we’ve got one that uses Kotlin’s funky new method reference syntax to reference the constructor of the standard EventBus. You can do that if it’s a zero arg constructor.

Here we’re actually using a lambda closure because it has a more complex constructor and some more complex setup going on. This works great. This gets you around the fact that you can’t have a hierarchy of virtual methods in a Spek class as such, but you can pass factories and properties to the constructor.

Diagrammed Assertions (12:27)

Up next is big Spek wishlist item for me: diagrammed assertions. This is a very cool feature that was introduced by Spock and later adopted by the Groovy language as a whole using its assert keyword.

Inside of an expect block in Spock, anything that can be evaluated as boolean is treated as an assertion, so this is an assertion.

expect:
order.user.firstName == "Rob"

If that assertion fails, Spock gives us some really fantastic output that looks like this:

order.user.firstName == "Rob" 
|     |    |         |
|     |    rob       false
|     |              1 difference (66% similarity)
|     |              (r)ob
|     |              (R)ob
|     [firstName:rob, lastName:fletcher]
Order<#3243627>

Every step in that expression on both sides of the binary operator is broken down and you can see each individual object. You can thus figure out what was going on. This way, if you’re scratching your head looking at this thinking “Well why is the name wrong? Was it the wrong user, was it the wrong order?”

It can be hard to tell until you see this awesome diagram breakdown where it can really help clarify things, as long as you have good toString() implementations on stuff generally.

Spock’s implementation of this is even so neat that if something toString()s the same, but doesn’t evaluate as equals it will point that out. If you’ve got a string with the number one in it compared to an integer one, and they don’t compare equals although they print the same, it’ll point out to you that actually the reason it’s failing is the types are different.

This power assert feature was adopted by the Groovy language as a whole so Groovy’s assert keyword does this by default. ScalaTest also has a trait you can implement that gives you a similar set of functionality, a similar feature.

This is the number one thing I would love to see added to Spek, because I’ve written Spock tests for many years now and living without this would be one of the harder things I would have to do. So, much as I love Spek, that’s the number one thing that’s missing right now for me.

This covers the interesting, useful testing cases of both frameworks that I wanted to highlight. Thanks.

Next Up: Kotlin in Depth#1: The Train to Kotlinville is Leaving! Are you on Board?