The Travel Corporation: Review Apps

We did a lot of work at the Travel Corporation, this describes some of the more technical aspects of our engagement. We also used A Monorepo, Feature Toggles, and Designed APIs for Real Users and Wrote Automated Tests for Every Feature.

Also, see the details of our implementation of Review Apps at Using EKS to Deploy All Your Pull Requests.

Deploying All the Pull Requests

Automated testing can only hope to execute scenarios that you can think of ahead of time. To compliment that, manual exploratory testing gives the team a chance to find new and interesting scenarios and conditions - especially when integrating with complex external systems. Scenarios that do cause bugs are of course are included as automated tests, so we continue to have good automated regression coverage.

However we found that we had a problem integrating manual testing into our release cycle, and I suspect we weren’t the only team to experience this. Here’s how our release cycle originally worked:

We’d make a new feature branch. The feature branch would have automated tests running on a continuous integration server. When these tests were all good, and reviews were positive, we’d merge. Only then would we be able to do our manual exploratory and acceptance testing. At this point it’s likely that this testing would reveal issues that meant we weren’t able to release the code, resulting in our master branch being polluted with features that weren’t releasable.

Now compound this issue with multiple in-flight features and bug fixes from developers on our own team, and accepting changes from developers on several other teams, this quickly became a major bottleneck for release. We essentially have a master that is full of features already released, features unreleased but untested (🤷‍♀️🤷‍♂️), features tested and passed (✅) and features tested and failed (❌). The result was that we spent a lot of time trying to figure out if we were ok to release… and there were always new features being merged… a situation that was becoming very difficult to manage (🤯)!

Following Heroku’s lead on what they called Review Apps, we decided to build a new, isolated, fully working environment for each and every new pull-request submitted to the repository. This meant that we could perform manual exploratory testing on each feature before it was merged into the master branch. We could also perform integration testing with external partners, or demo it to other stakeholders and integrate valuable feedback before merging. The process looks a bit like this:

Following this, our master branch was now virtually always releasable. We still liked to do a few manual smoke tests on things before we went for release, but our release frequency went from (at the time) once every week or two down to once or twice per day. The effect was pretty obvious to everybody involved in the project.

All new environments were listed in the pull requests on GitHub using the Deployment API so our developers and testers could easily access them.

The tech behind this was originally a Flynn cluster for simplicity, but which we eventually dropped in favour of Kubernetes for better support and reliability, you can read more on how we did that in our blog on the subject, Using EKS to Deploy All Your Pull Requests.