Acceptance Testing

The Offline-first Approach to Mobile App Development

If you have tried building offline-first capabilities into your apps, you have experienced the limitations of common methods like caching and manual replication. In this video Adam Fish, Director of Product at Realm, and Sergey Bondari, Mobile Architect at Concur, discuss the pros and cons of common offline-first development methods while proposing a new approach.

Introduction (0:00)

I’ll talk about acceptance testing, along with some of its approaches and pitfalls.

Two Kinds of Tests (0:33)

There are two types of tests:

  • One that verifies your code is correct - unit tests.
  • One that validates that you wrote the correct code - acceptance tests.

Testing is important. Moreover, defect avoidance in code is more powerful than defect removal, and less costly.

Attempts at Acceptance Testing (1:53)

There have been many technologies describing themselves as acceptance testing tools, but the approaches may not have been correct.

One issue with the testing tools available is that they don’t scale. Most software projects start out small while testing, but as time goes on, developers end up no longer running the tests because they take too long.

Another is that the tests break and we make no effort to fix them because there’s more pressing work to do.

Invariably, attempts at acceptance testing seems to devolve this way. Is it the tools that are inherently the problem, or is it the way we’re using the tools?

Behavior-Driven Development (4:52)

Another name for acceptance testing is behavior-driven development. Get your customers to write the requirements, execute them against the code, and come back with results that confirm the code works as expected.

This is a great way to do software development, because finding issues early saves time, without having to complete new work based on the misunderstanding of the requirements in the process.

The difference between merely using an acceptance test framework and doing acceptance test driven development

Aslak Hellesay, who is the inventor of Cucumber, posited this difference to people. He concluded that if you’re using a framework to only automate UI tests, you’re just using the tools, and not doing behavior-driven development.

The business logic, behaviors, along with the requirements are where we should verify that the software built is correct. Instead, many developers focus on other areas, including UI tests. These are not service or contract-level tests. If you’re excuting UI tests/full stack tests, they may become too slow and fragile, then they break.

FitNesse + Demo (7:58)

There’s a tool that’s not built around UI automation, and it’s called FitNesse. It has been used extensively across the industry for at least 15 years, but just not on iOS.

It’s a stand-alone wiki and acceptance testing framework.

Because it’s a wiki you can deploy it on a web server, and anybody can edit and read it. This is useful for collaborating with your customer, defining requirements of your software, in a written natural language (not code).

FitNesse gives you is the ability to plan, read, write, and execute requirements. It works across platforms so the same exact requirements can be used for Mac, iOS, Android, etc.

To demonstrate this tool, I can do some testing against a naughts and crosses game (tic-tac-toe).

Suppose I implemented testing as a couple of acceptance tests using XCUITest. I run the unit tests, then run the UI tests. The test succeeded, but they’re too slow: they took 28 seconds and will not scale.

FitNesse allows for acceptance tests that does not require doing things in the UI. The tooling does not exist for iOS, but we turned it into a single pod, which can be integrated into a project.

Add a target for the acceptance tests

This acceptance test target is a special version of your application. It’s actually a Mac application, which means it’ll run faster, and has no dependencies, and it has a clean architecture.

Running FitNesse

After setting up, there’s a file called LaunchFitNesse, which is just a shell script bundled as one of the resources that comes with the project. Running the script downloads FitNesse for you as well. With the exception of Java, this should be all you need to run FitNesse.

Requirements / Truth Tables

Tests for FitNesse revolves around the concept of tables. This is similiar to truth tables you studied in computer science.

Before writing any code, the goal is to write the requirements: that the inputs are a certain number of turns you should take, and given those turns, the message that should be coming out from the game should be the player’s won or the player’s lost, or it’s a stalemate.

We are trying to get the truth of what our application is describing without anything else.

The tests

In transfering the tests we had in the original testing framework. We want to ensure we have the cases, such as a diagonal line wins or a horizontal line wins.

Suppose we need to test for a scneario of a stalemate. I have a spreadsheet that covers this on FitNesse. This can be converted into a table and tested.

In testing human versus the computer, I’m going to edit the spreadsheet that came from the game designers by formatting it.

To implement these tests in code, I switch to acceptance test driven development mode, and start writing the code to make this tests pass.

In doing it this way, I’m developing and aiming at the right set of behaviors.

Conclusion (24:14)

When you architect code correctly in testing, you are able to drop out of the view controller layer, and replace it with FitNesse so it can talk directly to your system code.

A good, clean, architecture is essential to doing good acceptance testing. The reason why you should do this is beacuse it will afford you the ability to write tests that independently validates the behavior of your software without other implementation details - of databases, networks, or UI.

It’s important to get the customers of your software in the practice of writing tests. You may need to help them do that, but that is what BDD is. BDD is not automating UI automation tests.

By understanding the requirements early, it saves you time, because you’re not wasting time building the wrong things or getting confused about requirements.

Acceptance Testing Resources


Paul Stringer

Paul Stringer