On the Hunt for the Fastest CI Service

A speed- and usability-focused comparison of the Travis CI, CircleCI, and Semaphore CI Continuous Integration services.
jet fleet

Imagine that you’re a developer making changes to a shared codebase. Following git etiquette, you create a new branch for the changes to live in and make your changes there. But, instead of merging those changes immediately, you keep your code in your local branch until you have finished a larger set of related changes. Unfortunately, when you go to merge your branch into the master branch, another developer has already merged into the master branch, creating merge conflicts. Now, you are spending hours making sure you don’t break anything while resolving the conflicts: welcome to integration hell.

Until relatively recently, every developer had experienced integration hell. Then, in 1991, a man named Grady Booch proposed Continuous Integration (CI) – the process of integrating code into a shared repository frequently so as to minimize the pain of dealing with conflicting changes. As an added bonus, continuously integrated development allows us to verify each integration with an automated build and automated tests, and when using either Continuous Deployment or Continuous Delivery, the changes can be deployed to production as soon as they are integrated.

CI platforms exist to automate these bonuses. A CI platform can test, build, and deploy the code without any user intervention (though many still prefer to do that last step manually). When using a good CI platform, developers can deploy bug fixes quickly and introduce new features seamlessly. When using a bad CI platform, developers may spend hours waiting for their changes to be processed, only to receive an error in a test that should have been thrown immediately.

Here at Babbel, we have used Travis as our CI platform for years. With changing climates (see #travisAlums on Twitter), we decided to research other options. Below, I’ve ranked Travis against two other CI platforms: CircleCI – a popular alternative used in a number of large projects including the open source Vue.js (available on GitHub) – and Semaphore – the self-labeled “fastest CI/CD platform”; we will soon see if the latter claim is true.

Obviously, speed will be a key metric in our comparison. However, we will also note the usability of each platform. To measure build speed, I configured all three services on both an internal Babbel codebase and Vue.js. Please note that both of these codebases are JavaScript-based; you may find different results using different languages. We executed each app’s production build process thirty times on each codebase using each CI service and recorded the self-reported times. Note that the data is self-reported, and while it appears that Circle and Semaphore report the “start” time as the time a commit is received, Travis reports when the task exits its initial queue, which seems to be roughly thirty seconds after the other two start.

CI Services on Babbel Magazine
Measurement of Build Speed of CI Services (see Appendix for raw data)
CI Services on Vue
Measurement of Build Speed of CI Services (see Appendix for raw data)

In terms of speed, Travis ranked third in our tests. Despite not reporting the time a build spends in its queue, Travis’ reported build times lagged behind both Circle and Semaphore, and Travis’ fastest average build speed took 60 percent more time than the next slowest! Specifically, there is only one data point in which Travis outperforms any other service.

The difference in speed between Circle and Semaphore was less glaring. Circle took only 13 and 43 percent more time than Semaphore on our code and the Vue codebase respectively. A two-sample t-test, which measures the difference between any two numerical distributions (calculated from evanmiller.org), confirms Semaphore as the faster alternative in both cases. For statistics nerds: using 99% confidence intervals and the null hypothesis of equality between distributions, we achieved p values of 0.0015 and 0.0001 on our code and Vue, respectively.

When considering usability, Travis lacked features found in both of the other services. With both Circle and Semaphore, starting a new project is as simple as saving an autogenerated file into your GitHub repository, but getting started with Travis involves wrapping your head around the specifics of the configuration files and weeding through the unintuitive documentation. Similarly, both Circle and Semaphore support parallelization naturally using their workflows and pipelines, but enabling the same functionality with Travis involves manually using their build matrices, in which you manually configure a number of virtual machines.

Screenshot from CircleCI UI
Insights from Circle’s Dashboard

Usability is clearly improved for Circle and Semaphore. They both have incredibly simple tools to create an initial configuration file, and both offer a fully functional CLI for those so inclined. Semaphore offers a fairly minimalist dashboard, but still supports core functionality (though before their recent redesign, functionality as simple as deleting projects wasn’t possible from the dashboard). Circle, in contrast, has a beautiful and intuitive online dashboard as well as a similarly powerful CLI, and provides a variety of views as well as insights into build statistics, shown above. Additionally, Circle has by far the most readable and thorough documentation. They even have a whole section of documentation dedicated to migrating from Travis, which provides better explanations of Travis’ commands than their own documentation.

Median Speed on Babbel Magazine (h:mm:ss)0:02:330:01:310:01:19
Median Speed on Vue (h:mm:ss)0:03:550:02:020:01:29
Executes in
MacOS VMYesYesYes
Linux VMYes – Ubuntu OnlyYes – any from Amazon Machine ImagesYes – Ubuntu Only
Windows VMYes – Very Early SupportYesNo
Docker ContainersNoYesNo
Deploying with Docker
Docker CommandsYesYesYes
Docker ComposeYesYes – Only on VMsYes
Layer CachingYesYes – On all but WindowsYes
Terraform Support
Official Terraform ProviderNoNoNo

The choice between Semaphore and Circle is one of personal preference: would you sacrifice a small increase in build time for a better developer experience? Personally, I use Circle for my own projects–I find the dashboard intuitive and I enjoy looking at Circle’s “insights”. Based on the results of my research, the fastest choice appears to be Semaphore, and most usable appears to be Circle. But between the easy-to-use Circle and the speedy Semaphore, it’s up to you.


Babbel Magazine
Trial 00:02:290:01:370:01:25
Trial 10:02:320:01:340:01:21
Trial 20:02:330:01:400:01:18
Trial 30:02:390:01:420:01:20
Trial 40:02:330:01:430:01:15
Trial 50:02:340:01:360:01:25
Trial 60:02:340:01:520:01:30
Trial 70:02:300:01:210:01:25
Trial 80:02:330:01:380:01:29
Trial 90:02:340:01:360:01:15
Trial 100:02:330:01:210:01:56
Trial 110:02:320:01:190:01:10
Trial 120:02:310:01:210:01:14
Trial 130:02:330:01:320:01:12
Trial 140:02:340:01:200:01:16
Trial 150:02:360:01:420:01:15
Trial 160:02:310:01:190:01:08
Trial 170:02:340:01:260:01:20
Trial 180:02:350:01:310:01:15
Trial 190:02:300:01:240:01:09
Trial 200:02:300:01:220:01:10
Trial 210:02:360:01:300:01:24
Trial 220:02:350:01:210:01:31
Trial 230:02:430:01:510:01:29
Trial 240:02:320:01:200:01:16
Trial 250:02:360:01:290:01:25
Trial 260:02:370:01:570:02:12
Trial 270:02:320:01:290:01:19
Trial 280:02:290:01:260:01:09
Trial 290:02:340:01:440:01:12
Arithmetic Mean0:02:330:01:320:01:22
Geometric Mean0:02:330:01:320:01:21
Trial 00:03:510:01:470:01:27
Trial 10:03:570:02:130:01:17
Trial 20:03:550:01:560:01:19
Trial 30:04:010:02:090:01:20
Trial 40:04:120:01:570:01:32
Trial 50:04:020:01:440:01:27
Trial 60:03:570:01:580:01:24
Trial 70:04:110:02:100:01:32
Trial 80:03:550:01:520:01:21
Trial 90:04:020:01:580:01:31
Trial 100:03:480:01:560:01:34
Trial 110:03:440:05:280:01:31
Trial 120:03:500:01:450:01:20
Trial 130:04:110:02:120:01:23
Trial 140:03:540:01:490:01:31
Trial 150:03:540:02:070:01:35
Trial 160:04:060:02:110:01:54
Trial 170:04:000:02:000:01:18
Trial 180:03:470:02:010:01:18
Trial 190:04:090:02:050:01:21
Trial 200:03:480:02:180:01:31
Trial 210:03:550:02:150:01:24
Trial 220:03:490:02:040:01:35
Trial 230:03:520:01:510:01:32
Trial 240:03:490:02:090:01:26
Trial 250:03:550:01:470:01:40
Trial 260:03:480:02:450:02:19
Trial 270:03:590:01:540:01:18
Trial 280:04:020:02:080:01:36
Trial 290:04:010:02:020:01:35
Arithmetic Mean0:03:570:02:090:01:30
Geometric Mean0:03:570:02:060:01:29

Photo by Jeremy Straub on Unsplash

Want to join our Engineering team?
Apply today!