Free hosted 'CI as Code' services for private repos for hobby (focus on Scala/SBT)

When I was in university Jenkins (Hudson) was the thing for automated testing. Another famous one is Bamboo.

Lately I'm convinced that these (self-hosted) tools are a 'bad thing'. You configure them compeletely via an UI outside your repo.

I'm proponent of the CI definition as code you check into your repository. That way every commit and every branch can build according to the right CI definition. Other pro's: it's versioned and backed up, repeatable, and changes are tracable.

Not all my hobby projects make the daylight, and also because I'm Dutch, I was looking into CI services that offer a free tier and allow you to build private repos and support Scala (and SBT). And it should work with all major Git hosting providers: Github, GitLab and BitBucket. I had a preference for a CI provider that also supports using custom Docker images and also allows using attached services, for example a database, to run the tests, without the need to bake them in your build-image yourself.

Not all tested providers (completely) satisfy the Git providers criterium, but I think they are all noteworthy.

I tried out the following hosted CI as a Service providers, in alphabetical order:

Comparison

Feature BitBucket Pipelines Circle CI 2.0 Gitlab CI Shippable Travis CI
Github No Yes No Yes Yes
GitLab No No Yes Yes No
BitBucket Yes Yes No Yes No
Git Project Hosting Yes No Yes No No
CI Projects Every hosted Git repo Infinite Every hosted Git repo Infinite Infinite
Users 5 Infinite Infinite Login via Oauth, can be linked to one Github+one GitLab+one BitBucket account Infinite
Free tier 50 min / month 1500 min / month 2000 min / month 150 builds / month 100 builds
Parallel tests - 1 - 1 Yes
Concurrent builds Infinite 1 Infinite 1 Yes
Build custom Docker images No Yes Yes Yes Yes
Use custom Docker images Yes Yes Yes Yes Yes
Access to Docker command No Yes Yes No Yes
Cache Yes Yes Yes (inside workspace only) Yes Yes
Services (attach to other containers during build) Yes Yes Yes No (include services and build custom image) Yes
Environment vars / secrets Yes Yes Yes Yes Yes
Free support Forum + Ticket system Forum Public ticket system Forum Docs only
Debugging Local docker SSH Access to build container Local docker No No
Multi-project workflows No Yes Premium Yes Beta
API Yes Yes Limited Yes
Limits 5GB disk, 4GB RAM, 120 min / build 4GB RAM 4GB RAM AWS c4.large (3.75GB RAM) 120 min max / build
Cross builds / build matrix No No No Yes Yes
Artifact storage No (integrations) Yes Yes No (integrations) No (integrations)
Config format Yaml Yaml Yaml Yaml Yaml
Tools (validator, CLI) Online validator - Online validator - Online validator + CLI
Open source extras Website four free linux containers ($2400 annual value) - Infitine builds Infinite builds
Students Website - - - Website
Smallest upgrade $2/month: 500 min $50/month: 2 containers, infinite min/month $19/month: 10,000 min/month $25/month: c4 large node - infinite builds, full API $69/month: 1 concurrent job, unlimited everything
Setup experience Fast and simple Ok Advanced - rough product Ok Fast and simple

Demo project for all systems

I created a super simple small repository with a 'calculator' that can store a value in Redis. You can override the redis connection with a Typesafe config / environment variable, in case that's needed for any of the CI systems (or: between test and production!?). I provided links to all the repositories below and the build time without cache and with cache.


 BitBucket Pipelines Circle CI 2.0 Gitlab CI Shippable Travis CI
Demo project repo Repo Repo Repo Repo Repo
Initial build 2 min 58 sec 1 min 1 sec 2 min 00 sec 2 min 38 sec 2 min 29 sec
Cached build 2 min 0 sec
1 min 30 sec		 0 min 34 sec No caching, 2nd build:
3 min 44 sec		0 min 54 sec 1 min 59 sec

Please note that the timings between builds differs (a lot) for the same service. I didn't do a lot of builds and average the timings, but it gives an indication of the build time with and without cache. Note that Gitlab CI only offers cache for the workspace, and since the sbt and ivy caches are in the user home, I didn't cache them.

My choice for CI

For a simple project as the demo project, I would decide on where my code is hosted + the most free builds/minutes per month. That being said, my code is scattered all over the Big Three hosting companies. In that regard, Shippable is the only one that provides integration with all three. I didn't like the experience where I have to bake the (database) services in the build image (because you cannot link another container) if you want something they do not provide.

I liked the Circle CI 2.0 experience the best. The cached build speed was also amazing. Pricing if you want more than the free plan is a bit steep though.

Bitbucket Pipelines was easy to setup and get working for this project, as was Travis CI. They are linked to BitBucket and Github respectively and Travis doesn't really have a private free plan (a trial of 100 builds 'once' only).

Gitlab CI is a rough product, the yaml can be make really really complicated, I believe, with special features. The build time was not the best.

CD, other languages, workflows, notifications and integrations

I only tested the push and get feedback loop of the beforementioned products. Of course there is a lot more to it, like automatic deployments and pushing artifacts to storages. I only tested Scala/SBT and Redis integrations. Workflows where projects are linked together and trigger builds on each other, fan-in/fan-out, scheduled builds, notifications, a lot of things are not covered here. I encourage you to look around at all the possible features and integrations and share your experiences as well!

Questions on my experience? Ask!

Comments

  1. Unknown17/8/17 10:04

    Nice comparison, Jasper! I agree with you to a certain degree that config management with self hosted tools can be a bit of a pain. But I do think the picture your sketching of managing Jenkins is a bit too negative. Nowadays it's quite easy to run Jenkins in a containerized way on platforms like Kubernetes. There's several Helm Charts available (e.g. https://github.com/kubernetes/charts/tree/master/stable/jenkins) to get a Jenkins server up and running without managing the XML config yourself. For agents / slaves you can specify your own pod templates or Docker images, which of course you can keep under source control. With Jenkins pipelines (https://jenkins.io/doc/book/pipeline/), you also keep the CI definition in the source code of your projects. That way the UI config which is not under source control as you describe it stays minimal.

    Conclusion: I think the ultimate decision really depends on how much flexibility you need. With Jenkins you basically have total freedom. It's easy to extend. There's a gazillion plugins available. If you don't need all this flexibility though, I'd definitely go for a managed solution. If managed solutions are "slowing you down" it's not such a hassle anymore to host your own CI as I tried to describe.

    ReplyDelete
  2. Jasper Timmer17/8/17 10:17

    Hi, thanks for your comment! While I totally forgot about the new Jenkins Blue Ocean and Pipelines plugins (with the bit more complex Groovy DSL for the definition file), you still need to run the service yourself. Not all hobby tinkerers have a spare computer or want to pay for a 24/7 turned-on EC2 instance. In those cases, these 'free tiers' are a wonderful solution and don't forget, they save you on maintenance/ops work so you can focus on your project!

    ReplyDelete

Post a Comment

Popular posts from this blog

Microk8s cluster on a homelab (Proxmox) local network with a Fritzbox and Metallb

Homelab/server with Proxmox connected to my Fritzbox 7590 (XS4ALL NL) configured on home network 192.168.178.0/24  Inside Proxmox I have 3 vms (ubuntu server 20.04 LTS) that I named k8s-{1,2,3} During setup of the Ubuntu server vms I selected install openSSH and microk8s In the fritzbox I selected the option to give proxmox host and k8s-1,2,3 always the same IP After boot of servers: On k8s-1 I did sudo microk8s add-node and ran the output on k8s-2, then I repeated that for k8s-3. I enabled addons on k8s-1:  sudo microk8s enable dns metallb It will ask for an IP range, I opted for 192.168.100.0/24 Then in the Fritzbox, I went to network, network settings, more network settings, static routing tables I added the route 192.168.100.0, 255.255.255.0, 192.168.178.x (ip of k8s-1), active, and I saved it. Now I could test the Metallb: wget https://raw.githubusercontent.com/charmed-kubernetes/metallb-operator/master/docs/example-microbot-lb.yaml kubectl apply -f example-microbot-lb.yaml kubec
Read more

ABN Amro bank statement export to .OFX

I recently started to use YNAB 4 (You Need A Budget) software to manage/monitor my spendings. You can find it here  (referral + $6 discount alert). Great and easy to use software to budget your expenses. But for Dutch users I found a big disadvantage: you cannot import files from ABN Amro. YNAB only imports .OFX (Money) or .QFX (Quicken) and to some extend (they say they do not prefer it) .QIV or .CSV files. On the other end, ABN Amro only exports PDF, Excel (.xls), MT904 (.STA), TXT (.TAB, tab delimited) and CAMT. 053 formats. To get your transactions into YNAB, you need to download a MT904 converter. You can find it here:  http://sourceforge.net/projects/mtconverter/  You need to have Java installed for it to work ( www.java.com ). To use the converter, I created a batch file with the following lines: for /r %%i in (in\*.sta) do ( echo %%~ni java -jar mtconverter-0.01.jar "%%i" out\%%~ni.ofx del %%i ) Save it as convert.bat in the same directory as the .
Read more

Using Flying Saucer and iText in Java to convert XHTML to PDF

The situation Where I work, we were generating reports in XHTML, for printing. The styling should be easily configurable. The problem with this approach is the cross-browser look of your report. IE and Chrome print two completely different reports, so to speak. To prevent the difference between browsers, and to re-use the existing report generation, I decided I needed to render the print at the serverside. So I had html, and I wanted it to look the same on every computer. Pdf is a good medium for this purpose, so I needed a html to pdf library, for our Java system. I first tried iText by itself, but this did not apply the css. Browsing the web a bit further I found the combo Flying Saucer and iText , and this was a winning combination for us. Flying Saucer is a Java library that renders XHTML/XML + CSS to screen/image/PDF. For PDF there is a dependency on iText, a library to create pdf files. The code In this blogpost I provide some codesnippets, not a full working example.
Read more