Skip to content

How we Improved our Microservices Integration Tests Using Pytest

Gilad Hoze
By Gilad Hoze
7 min read
Testing
How we Improved our Microservices Integration Tests Using Pytest

At Seekret, when we started to write integration tests (in pytest) for our solution, we only had a couple of services. In the beginning, it worked well, but as time went by, more services were written (let’s go microservices architecture!), and our old test infrastructure couldn’t handle it anymore.

To solve it, we had to refactor our infrastructure. Before we started the refactor, though, we did some thinking about the big problems in our integration tests, and what’s keeping them from being maintainable, cost-effective, and of course, reliable.

Our biggest problems

Due to our old test infrastructure, our team spent hours developing tests or handling issues in the current test infrastructure. We’re going to cover in detail the main three problems that we had.

Maintainability

We had massive code duplication, especially in the services' initialization, including environment variables, dependencies, and more.

Every time we wanted to change the way a service is initialized, we had to change it in each test file, and of course, with time, we had more and more services and the time we invested in changing all the relevant places grew exponentially.

Furthermore, we weren’t fully utilizing pytest, which made our tests long, messy, and hard to read. For example, we rarely used parameterized tests, and that could’ve made our tests much cleaner, readable, and overall better.

Lastly, we had haphazard dependencies which resulted in unnecessary test configuration that had to be made before each test.

Readability

As I mentioned, we had a lot of code duplication, which, in the end, resulted in boilerplate code. Even if we had test setups that were practically the same, they looked a bit different, and that made the rationale behind the code much harder to see.

In addition, there were a lot of complex configurations for the tests, and even the simplest tests were difficult to define and set up.

Learning Curve

It took a lot of time for our new programmers to successfully write integration tests; they had no examples to learn from. There weren’t tests that demonstrated our “good practices” and because those similar tests were written differently, there’s no way of knowing what was the “correct way” to write a certain test.

To understand the described problems better, take a look at the following test:

Let’s assume we have two services - service1 and service2 and service2 depends on service1.

SERVICE_1_ADDR = "localhost:8080"
SERVICE_2_ADDR = "localhost:8081"
@dataclasses.dataclass
class Service1:
addr: str
@dataclasses.dataclass
class Service2:
addr: str
service1_addr: str = ""
@pytest.fixture
def service1() -> Service1:
return Service1(SERVICE_1_ADDR)
@pytest.fixture
def service2() -> Service2:
return Service2(SERVICE_2_ADDR, SERVICE_1_ADDR)
def test_service2(service2, service1):
service2.service1_addr != ""

As you can see, the test defines unnecessary fixtures, and the service dependencies aren’t configured properly. The above example is a simple one with only two services, but of course, when the number of services grows, this problem also grows.

Possible solutions

We understood that if we wanted to improve our integration tests, we had to solve the problems above. I’m going to focus on the major change that resulted in a significant contribution to solving these problems.

We wanted to pinpoint the major cause of the problems above, and it was the services' initialization. Eventually, we needed to centralize it and make it easy and intuitive to use.

We had 2 main approaches that we investigated to solve the above problems:

Docker compose

docker-compose is a tool we were familiar with and it seemed like a good tool to use to set up our services. We started to look into it and we found a great pytest plugin - pytest-docker-compose.

This plugin allows you to write docker-compose files for your tests and use them in the “pytest way” - fixtures. I know you want to know how this can help you in your tests, so let’s dive into examples:

For this example, we’ll have two services - service1 and service2 which depends on service1.

Let’s define the docker-compose file for these services:

version: '3'
services:
service1:
image: ${CONTAINER_REGISTRY}/service1:latest
ports:
- $SERVICE1_EXTERNAL_PORT:$INTERNAL_PORT
environment:
- BIND_ADDRESS=0.0.0.0:$RPC_PORT
networks:
internal:
aliases:
- service1
service2:
image: ${CONTAINER_REGISTRY}/service2:latest
ports:
- $SERVICE2_EXTERNAL_PORT:$INTERNAL_PORT
environment:
- BIND_ADDRESS=0.0.0.0:$RPC_PORT
depends_on:
- service1
networks:
internal:
aliases:
- service2
networks:
internal:
driver: bridge

Now that we have our simple docker-compose file with the definition of the two services (including their dependencies), let’s continue to the pytest part:

First of all, we wanted to utilize the pytest custom markers feature to define what services our test requires, so let’s look at the next example:

@pytest.mark.service2
def test_service2(service2):
pass

As you can see, I wanted to test service2 so I marked my test with the service2 marker.

Now, let’s write the required fixtures to compose and access our services:

@pytest.fixture
def container_getter(request, docker_project) -> ContainerGetter:
"""
This fixture reads the test markers and composes the services
according to them.
"""
# "usefixtures", "parametrize" are builtin markers, hence we want to ignore them
services = [m.name for m in request.node.own_markers if
m.name not in ["usefixtures", "parametrize"]]
containers = docker_project.up(services)
if not containers:
raise ValueError("`docker-compose` didn't launch any containers!")
yield ContainerGetter(docker_project)
docker_project.down(ImageType.none, request.config.getoption(
"--docker-compose-remove-volumes"))

We defined a fixture that only composes the services that were marked in the test, which returns an object that enables container access (to test them afterward).

@dataclasses.dataclass
class Service2:
container_name = "service2"
host: str = ""
port: int = 0
@pytest.fixture
def service2(container_getter) -> Service2:
c = container_getter.get(Service2.container_name)
try:
network_info = c.network_info[0]
return Service2(network_info.hostname, network_info.host_port)
except IndexError:
raise DockerConfigurationError(f"Invalid port bindings in {c.name}")

We defined a basic data class that represents service2 and another fixture that uses the previous one and returns an instance of service2 with its network details.

Now, let’s write our simple test:

@pytest.mark.service2
def test_service2(service2):
assert service2.host != "" and service2.port != 0

As you can see, the test is defined the same as before but we added an assertion that validates our service got its network configuration properly.

Now, after we set everything up, our test configuration is rather simple and concise, and all the service's setup is done in the docker-compose file (including the service's dependencies).

All we needed to do in the test is to define the service we wanted to test. By now, I think you can say that this test infrastructure is maintainable (even easier to maintain if you are already familiar with docker-compose), simple, readable, and easy to learn.

Service wrappers and pytest fixtures

This approach is mainly focused on wrapping the service's execution and letting the pytest fixtures mechanism handle dependencies between services. The implementation is rather simple and straight-forward and it doesn’t require any additional plugins, so let’s dive into it:

We’re going to define 1 abstract class that wraps our services and controls their execution:

class Service:
"""
This class wraps executables and implement setup and cleanup logic for them.
"""
def __init__(self, exeutable_path):
self._exeutable_path = exeutable_path
self.exit_stack = ExitStack()
self._process = None
def __enter__(self):
self._process = self.exit_stack.enter_context(subprocess.Popen(self._exeutable_path))
return self
@property
def process():
return self._process
def __exit__(self, exc_type, exc_val, exc_tb):
# Validates that the process exists before attempting to kill it
if self._process is not None:
self.kill()
self.exit_stack.__exit__(exc_type, exc_val, exc_tb)
self.exit_stack = ExitStack()
self._process = None
def kill(self, timeout=None):
"""Kills the process and wait for it to exit"""
try:
self._process.kill()
self._process.wait(timeout)
except ProcessLookupError:
logger.warning(f'Got lookup error while trying to kill '
f'process {self._process.pid}')

The class above enables control of the service execution by using context managers.

Now let’s define our two services:

class Service1(Service):
executable_path = "/path/to/service1"
def __init__(self):
super().__init__(self.executable_path)
class Service2(Service):
executable_path = "/path/to/service2"
def __init__(self, service1: Service1):
self.service1 = service1
super().__init__(self.executable_path)

After we’ve defined our two services, we'll need to write fixtures that utilize the yield keyword (like we did before) to set up the services, provide their instances to our test, and, in the end, tear them down:

@pytest.fixture
def service1() -> Service1:
with Service1() as svc:
yield svc
@pytest.fixture
def service2(service1) -> Service2:
with Service2(service1) as svc:
yield svc

Notice that service2 depends on service1 and we enforced this dependency by using the service1 fixture in the definition of the service2 fixture.

Also, we can see that the fixture is entering the service’s context, yields its instance, and, thanks to the pytest yield fixtures, the test’s teardown will exit the service’s context.

Now, all is set up and we can write our simple test:

def test_service2(service2):
assert hasattr(service2, "service1") and service2.service1.process is not None

The test checks that service2 has the service1 and validates that service1 is been executed successfully (remember that we didn’t need to call it explicitly in our test).

Conclusion

Tests are a crucial stage during the lifecycle of software development, and in my opinion, integration tests play an important role in this stage.

In the end, we’ve chosen the “Service wrappers and pytest fixtures” approach because it was more straightforward and less time to implement (for us). Nevertheless, the “docker-compose” approach has its advantages and might be a better fit to solve your problems.

Eventually, these changes took us three weeks to implement and integrate (we had 20+ services and 50+ integration tests) but we can say for sure that it was cost-effective and it improved the code quality as well as the quality of life of our R&D team. As the old saying goes “the happier the software developer, the better the code he writes”. I’m kidding; there isn't an old phrase that says that but now there is!