Split-machine testing using GitHub Actions

Services like GitLab and GitHub Actions for test pipelines and Continuous Integration are great. As someone who has spent years wrestling with Jenkins, moving to these minimalist yaml-syntax, container-based pipelines has been a breath of fresh air. As these platforms are based on top of docker, you can easily make use of ‘typical’ network services such as PostgreSQL, Redis and RabbitMQ. However this capability can be used beyond generic network services that support your application, and instead be used to test your application that has been deployed across multiple containers.

Why is this important?

“Well it works on my machine…” is a phrase I am used to hearing all too often. I am also guilty of belting out this phrase occasionally 😳. Testing your project/product running entirely locally is often done for convenience whilst developing. Having a test pipeline that provides isolated testing, on a clean environment, is capable of catching some environment-based issues. However, even with good test coverage, there are still issues that may not manifest if your test pipeline is testing everything ‘locally’, i.e. all of your application’s components are running locally on the test environment. Some issues I’ve personally experienced include a service binding to 127.0.0.1 instead of 0.0.0.0, and a service that is dependent on files that do not form part of the deployment. Deploying your application into a realistic test environment, across multiple machines/containers can provide another dimension of testing and catch such issues.

In this article I’m going to present a number of methods for deploying a multi-component project to individual containers in your test pipeline, providing a more realistic environment and allowing you to achieve ‘split-machine testing’. My hope is that I can save someone, somewhere some time.

In this article I will focus on GitHub Actions, but this pattern can also be replicated in GitLab.

Getting Started

To demonstrate this I’m going to use one of my open-source projects: robotframework-remoterunner. This is a Python package that allows you to execute robotframework test suites remotely. This comes in the form of two scripts: rfagent that runs on the remote machine listening for execution requests, and rfremoterun that initiates the remote run. Naturally these are designed to run on entirely separate machines, otherwise they serve no purpose. As such, running integration tests that spin up instances on the local machine are not realistic.

Implementation 1: Using Services

So the plan is to stretch the model and re-purpose a ‘service’ to run my rfagent script and run rfremoterun script natively on the runner, effectively achieving ‘split-box’ testing. Services are containers, so I’ve created an image that runs a bash script to checkout my repo, install the package, and launch the rfagent script. I’ve chosen for this image to be generic enough to support any Python Package. The image is designed in a way that the container expects environment variables to be present which instruct it what repository & commit to checkout, and what command to run to launch the app. I arrived at this:

This is a relatively simple image, inheriting from the Python 3.8-slim-buster base image, but git is also installed and the entrypoint is the setup_app.sh shell script that gets copied into /workspace.

The setup_app.sh script is composed of:

Here you can see that a git repository is cloned and a specific branch or commit is checked out. The package is then installed using pip, and the app is launched by running the app launch script. These are all environment variables that the container expects to be present.

Once this image has been published to DockerHub, we can then make use of it in our GitHub Workflow to create a service to run the rfagent.

Here we have a service that runs in a fresh container, and we pass in the Git URL, commit hash and command line in order to run the rfagent. We also map port 1471 so that the rfremoterun script can connect in.

In the steps section of the workflow, the following actions are performed:

• Checkout the repository
• Install Python 3.8
• Install the package
• Poll on a curl to http://0.0.0.0:1471 to wait until the rfagent service is setup and running (it takes a little time to checkout and install)
• Execute rfremoterun

We now have the two components running on separate machines (or containers rather), meaning there is complete isolation apart from the mapped port. This achieves our goal but there are some inefficiencies. For one we’re checking out the repository twice which slows down the build, but there’s also a docker image that now needs to be maintained.

Implementation 2: Mounting Volumes

We can address these inefficiencies by making use of volumes. This feature means we can mount a directory directly into the container, allowing us to pass files in. This avoids having to do a separate git clone within the rfagent container. This also eliminates the need for git to be installed, meaning we can simplify the workflow even further by just using the Python base image rather than maintaining our own. On the Runner services are launched before the steps are executed, so to make use of these changes we also have to move the launching of the rfagent container to be done manually in the steps section. This is necessary because otherwise the host directory being mounted will not exist at the point the service is being launched. Our complete workflow now looks like this:

This fixes the issues raised with the first implementation, but the workflow file is beginning to look quite messy, especially with a long docker run command.

Implementation 3: A Minimalist Approach

Now that we’ve seen that we can make full use of docker on the GitHub Runner, we can go for a more minimalist approach using docker-compose. This move means we have dedicated containers for both the rfagent and rfremoterun scripts, and the docker configuration is abstracted away into a docker-compose.yml file keeping the workflow file tidy. Note that on the rf-remoterunner container we mount two volumes, one containing the .whl file that will be installed, and the other containing the robotframework test suite to be executed.

The workflow file is much simpler and hands the container management and configuration to docker-compose. Using the exit-code-from flag ensures that the exit code is returned (which is implicitly checked by the Runner for failure), and the rfagent container is torn down once rf-remoterunner has finished executing.

Conclusion

Here I’ve presented three examples of test pipelines that make use of multiple containers which should fulfill a variety of use cases. In any case, the takeaway point is to use docker on the Runner to its full potential in order to create richer test environments. By doing this you can catch issues associated with only ever testing your application locally.