Contract Testing 101
Contract testing has been gaining a lot of attention these past few years, especially with the rise of our microservices architecture. It’s one of the testing and quality standards that we are aiming for here at Zoopla for new projects to implement, especially when they are working with data. In this blog post, I’ll talk about why traditional API testing will not scale for microservices, the problems we’ve faced with relying solely on unit testing and API testing, what contract testing is, and how it can help you deliver changes and features to your services safely.
Why Traditional API Testing doesn’t Scale
To start, let's imagine that we have a simple REST service here.
We have a web client as the data consumer which makes a call to the REST API for some data. The API here is our data provider and will grab data from the database and give it back to the consumer. From a testing perspective, we can write API integration tests and then run them against a test environment to ensure that we’re getting the correct data from the provider.
If we then look at microservices, where large software projects are broken down into smaller modules or components and are developed independently by different teams, this can bring in a different challenge to testing.
The challenge becomes trying to test all of the different microservices together in a dedicated testing environment as traditional API integration testing will not scale out well. Different teams can deploy their changes to different services at the same time resulting in data that can constantly change. In addition, some of the services might be down due to environment issues and so therefore relying on API integration tests will be cumbersome.
While deploying a lot of small, independent services has benefits, integration testing becomes complex as the integration points between different services increases. Imagine as well that you have hundreds (or thousands!) of different services communicating with each other like Amazon or Netflix; how are you going to ensure that every change you make doesn’t impact the other services?
Here at Zoopla, we have implemented a microservices architecture where our User Interface interacts with a GraphQL layer, which in turn interacts with additional API services. We need to have high confidence that whatever data our GraphQL layer needs, that it’s matching what the data provider gives.
Problems with Unit Testing
Having tests that are isolated are very valuable. They give us a fast feedback loop to determine if different inputs will yield expected outputs. Teams can fail fast and find issues early on. The problem with this, however, is that it doesn't give us the confidence to release.
You might implement a mock provider and this might not be the true representation of the real provider and likewise, the simulated consumer might not be the true representation of the actual consumer.
It also doesn't prevent broken changes to be deployed on a dedicated environment. The provider can deploy broken changes to production especially if they’re on a different pipeline and doesn't trigger the consumer’s tests on a regular basis.
Problems with Integration Testing
With API integration testing, we test the actual endpoints and assert that the provider returns the data that the consumer is expecting. While these are very valuable, having too many of these tests becomes quite complex in a microservices architecture.
It's slow and brittle since they are subjected to a lot of data changes. It also requires dedicated environments where the services are integrated with one another, thereby requiring lots of maintenance.
Similarly, it also won’t prevent broken changes being deployed on a dedicated environment. So, what else can we do to test our services more efficiently?
If we look at how we can test a fire alarm, we don't set our house on fire. That would be a disastrous thing to do. Instead, we test the contract it holds with our ears by using the fire alarm testing button.
This is where Contract Testing comes in.
Contract Testing
Contract testing is a form of testing where you test the contract between the services communicating to each other. It consists of the following:
- Consumer - a client or another service that consumes data from a provider.
- Provider - a service that provides data to a consumer.
- Contract/Pact - a document which serves as the contract between consumer and provider. This is normally in a JSON format and captures the consumer needs from the provider, including the data types, status codes, and responses the provider will return.
- Pact Broker - a hosted service which stores all the contracts. This serves as the communication channel between consumers and providers.
One of the popular contract tools is Pact. Pact is a tool that is predominantly consumer driven and dictates that the consumer defines the contract.
With consumer driven contract testing, the provider is free to change their behaviour without affecting the consumer tests if the consumer doesn’t use it. This also means that we can have more confidence in making changes on both the provider and consumer side.
The strategy that we’ve adopted here in Zoopla is that the consumer is our GraphQL layer and the providers are the different data sources maintained by the different teams.
How does Contract Testing work?
So how does Contract testing work?
Contract testing works in two parts from a consumer and provider perspective.
The consumer writes their test from what they expect the provider to respond to. Instead of communicating to an actual data provider, Pact simulates a mock provider. The expectations are then recorded on a contract which the consumer then uploads to a Pact Broker.
The provider then grabs the contract from the broker, replays the requests that the consumer specified on the contract and Pact will then verify if the expectations are similar to what the provider provides.
By utilising Contract Testing, you can get the following benefits:
- Fast feedback (as the tests are very quick to run)
- Run the tests independently on both the consumer and provider pipeline.
- Less maintenance is required (unless there's been a change in the contract). Pact allows you to match data types so you can ignore data changes, but if there are changes on the data type this can be flagged.
- Confidence to release. Pact has a CLI tool which lets you check if you can safely deploy changes to a specific environment via their can i deploy command.
- No need for integrated environments (as the contract will be uploaded to the Pact Broker).
- The contract acts as the source of truth and provides additional documentation to your team.
- Increased confidence that you can evolve codebases continuously knowing that Pact will ensure that the contract is satisfied.
- Stops over-reliance on slow API integration tests or UI tests (which can be challenging in micro services).
- Ensures that only parts of the API that will actually be used by consumers will be developed.
When NOT to use Contract Testing?
On the other hand, there are scenarios where using Pact or contract testing might not be ideal:
- A lack of buy-in from teams across your org
- Public APIs where your consumers are unknown
- Third Party APIs where it's difficult or impossible to convince them to have provider tests in their pipeline. However, Pactflow has recently released bi-directional contract testing which lets you do this!
Wrapping Up
Contract testing, while very valuable, is not a replacement for functional testing or performance testing. It's an additional layer of tests that you can utilise to have better confidence that changes in your APIs or data services do not have any adverse effects. We have a lot of teams here at Zoopla implementing contract testing now, and we aim to push it as standard going forward. To learn more about contract testing, the following resources can come in handy:
In a future blog post, we’ll look at how we have implemented contract testing in our Gitlab pipelines on both the consumer and provider side.
Image sources
- Post image by Van Tay Media on Unsplash
- Post image by DataRobot
- Post image by John Cafazza on Unsplash
- Post image by Pact