Free hosted 'CI as Code' services for private repos for hobby (focus on Scala/SBT)
When I was in university Jenkins (Hudson) was the thing for automated testing. Another famous one is Bamboo.
Please note that the timings between builds differs (a lot) for the same service. I didn't do a lot of builds and average the timings, but it gives an indication of the build time with and without cache. Note that Gitlab CI only offers cache for the workspace, and since the sbt and ivy caches are in the user home, I didn't cache them.
Lately I'm convinced that these (self-hosted) tools are a 'bad thing'. You configure them compeletely via an UI outside your repo.
I'm proponent of the CI definition as code you check into your repository. That way every commit and every branch can build according to the right CI definition. Other pro's: it's versioned and backed up, repeatable, and changes are tracable.
Not all my hobby projects make the daylight, and also because I'm Dutch, I was looking into CI services that offer a free tier and allow you to build private repos and support Scala (and SBT). And it should work with all major Git hosting providers: Github, GitLab and BitBucket. I had a preference for a CI provider that also supports using custom Docker images and also allows using attached services, for example a database, to run the tests, without the need to bake them in your build-image yourself.
Not all tested providers (completely) satisfy the Git providers criterium, but I think they are all noteworthy.
I tried out the following hosted CI as a Service providers, in alphabetical order:
- BitBucket Pipelines - https://bitbucket.org/product/features/pipelines
- Circle CI 2.0 - https://circleci.com/
- GitLab CI - https://about.gitlab.com/features/gitlab-ci-cd/
- Shippable - https://www.shippable.com
- Travis CI - https://travis-ci.org/
Comparison
Feature | BitBucket Pipelines | Circle CI 2.0 | Gitlab CI | Shippable | Travis CI |
Github | No | Yes | No | Yes | Yes |
GitLab | No | No | Yes | Yes | No |
BitBucket | Yes | Yes | No | Yes | No |
Git Project Hosting | Yes | No | Yes | No | No |
CI Projects | Every hosted Git repo | Infinite | Every hosted Git repo | Infinite | Infinite |
Users | 5 | Infinite | Infinite | Login via Oauth, can be linked to one Github+one GitLab+one BitBucket account | Infinite |
Free tier | 50 min / month | 1500 min / month | 2000 min / month | 150 builds / month | 100 builds |
Parallel tests | - | 1 | - | 1 | Yes |
Concurrent builds | Infinite | 1 | Infinite | 1 | Yes |
Build custom Docker images | No | Yes | Yes | Yes | Yes |
Use custom Docker images | Yes | Yes | Yes | Yes | Yes |
Access to Docker command | No | Yes | Yes | No | Yes |
Cache | Yes | Yes | Yes (inside workspace only) | Yes | Yes |
Services (attach to other containers during build) | Yes | Yes | Yes | No (include services and build custom image) | Yes |
Environment vars / secrets | Yes | Yes | Yes | Yes | Yes |
Free support | Forum + Ticket system | Forum | Public ticket system | Forum | Docs only |
Debugging | Local docker | SSH Access to build container | Local docker | No | No |
Multi-project workflows | No | Yes | Premium | Yes | Beta |
API | Yes | Yes | Limited | Yes | |
Limits | 5GB disk, 4GB RAM, 120 min / build | 4GB RAM | 4GB RAM | AWS c4.large (3.75GB RAM) | 120 min max / build |
Cross builds / build matrix | No | No | No | Yes | Yes |
Artifact storage | No (integrations) | Yes | Yes | No (integrations) | No (integrations) |
Config format | Yaml | Yaml | Yaml | Yaml | Yaml |
Tools (validator, CLI) | Online validator | - | Online validator | - | Online validator + CLI |
Open source extras | Website | four free linux containers ($2400 annual value) | - | Infitine builds | Infinite builds |
Students | Website | - | - | - | Website |
Smallest upgrade | $2/month: 500 min | $50/month: 2 containers, infinite min/month | $19/month: 10,000 min/month | $25/month: c4 large node - infinite builds, full API | $69/month: 1 concurrent job, unlimited everything |
Setup experience | Fast and simple | Ok | Advanced - rough product | Ok | Fast and simple |
Demo project for all systems
I created a super simple small repository with a 'calculator' that can store a value in Redis. You can override the redis connection with a Typesafe config / environment variable, in case that's needed for any of the CI systems (or: between test and production!?). I provided links to all the repositories below and the build time without cache and with cache.
BitBucket Pipelines | Circle CI 2.0 | Gitlab CI | Shippable | Travis CI | |
Demo project repo | Repo | Repo | Repo | Repo | Repo |
Initial build | 2 min 58 sec | 1 min 1 sec | 2 min 00 sec | 2 min 38 sec | 2 min 29 sec |
Cached build | 2 min 0 sec 1 min 30 sec |
0 min 34 sec | No caching, 2nd build: 3 min 44 sec | 0 min 54 sec | 1 min 59 sec |
Please note that the timings between builds differs (a lot) for the same service. I didn't do a lot of builds and average the timings, but it gives an indication of the build time with and without cache. Note that Gitlab CI only offers cache for the workspace, and since the sbt and ivy caches are in the user home, I didn't cache them.
My choice for CI
For a simple project as the demo project, I would decide on where my code is hosted + the most free builds/minutes per month. That being said, my code is scattered all over the Big Three hosting companies. In that regard, Shippable is the only one that provides integration with all three. I didn't like the experience where I have to bake the (database) services in the build image (because you cannot link another container) if you want something they do not provide.
I liked the Circle CI 2.0 experience the best. The cached build speed was also amazing. Pricing if you want more than the free plan is a bit steep though.
Bitbucket Pipelines was easy to setup and get working for this project, as was Travis CI. They are linked to BitBucket and Github respectively and Travis doesn't really have a private free plan (a trial of 100 builds 'once' only).
Gitlab CI is a rough product, the yaml can be make really really complicated, I believe, with special features. The build time was not the best.
CD, other languages, workflows, notifications and integrations
I only tested the push and get feedback loop of the beforementioned products. Of course there is a lot more to it, like automatic deployments and pushing artifacts to storages. I only tested Scala/SBT and Redis integrations. Workflows where projects are linked together and trigger builds on each other, fan-in/fan-out, scheduled builds, notifications, a lot of things are not covered here. I encourage you to look around at all the possible features and integrations and share your experiences as well!
Questions on my experience? Ask!
Nice comparison, Jasper! I agree with you to a certain degree that config management with self hosted tools can be a bit of a pain. But I do think the picture your sketching of managing Jenkins is a bit too negative. Nowadays it's quite easy to run Jenkins in a containerized way on platforms like Kubernetes. There's several Helm Charts available (e.g. https://github.com/kubernetes/charts/tree/master/stable/jenkins) to get a Jenkins server up and running without managing the XML config yourself. For agents / slaves you can specify your own pod templates or Docker images, which of course you can keep under source control. With Jenkins pipelines (https://jenkins.io/doc/book/pipeline/), you also keep the CI definition in the source code of your projects. That way the UI config which is not under source control as you describe it stays minimal.
ReplyDeleteConclusion: I think the ultimate decision really depends on how much flexibility you need. With Jenkins you basically have total freedom. It's easy to extend. There's a gazillion plugins available. If you don't need all this flexibility though, I'd definitely go for a managed solution. If managed solutions are "slowing you down" it's not such a hassle anymore to host your own CI as I tried to describe.
Hi, thanks for your comment! While I totally forgot about the new Jenkins Blue Ocean and Pipelines plugins (with the bit more complex Groovy DSL for the definition file), you still need to run the service yourself. Not all hobby tinkerers have a spare computer or want to pay for a 24/7 turned-on EC2 instance. In those cases, these 'free tiers' are a wonderful solution and don't forget, they save you on maintenance/ops work so you can focus on your project!
ReplyDelete