Travis-CI Review

A couple of days ago I stumbled across Travis-CI, a tool for automated build tetsts of github projects. It sounded like a great idea to me, since I were very lax lately with testing builds of PCP on other platforms, so I gave it a try.

Basically Travis-CI works very simple: you create a .travis.yml file, which contains instructions to configure, build and test your github project, and you create an account on Travis-CI and activate a github project of yours. Then every time you make a commit, Travis-CI creates a fresh VM instance, checks out your latest code and runs the instructions in your .travis.yml file.

However, I had lots of problems making it work. Admittedly some of them were caused by code which either didn't compile or run on linux. This is a fat point on the plus side, because I was totally unaware of those issues. But I had much more trouble getting all the dependencies to work.

According to Travis' documentation each VM contains everything needed like gcc, perl, node.js and whatnot. But in reality you have to denote your project to a specific environment, which is C in my case. The unittests in the PCP source are driven by a perl script which needs a couple of additional modules. Their documentation stated, they use perlbrew, which I know and like a lot since I use it almost every day. Unfortunately perlbrew is not installed in a C VM. And so I had to install the perl modules "manually" (by wgetting them, untar, perl Makefile.PL, make and sudo install).

The good thing is you can do almost anything on the Travis-CI VM including root installs with sudo.  Thanks to this ability I were able to get the unittests to work. But PCP also has a python binding. This was mush more troublesome. Python itself were installed, but no headers, no python-pip, no cffi and so on. First I tried to install a package "python-cffi", which does exist according to google, but not on the Travis-CI VM. Instead I had to install the python headers, python-pip and libffi by downloading and compiling it. There's also an libffi ubuntu package available but it could not be installed because of some unspecified and unresolvable conflict. Such conflicts of binary package management tools are the very reason I left linux behind many years ago.

Well, now, 60 commits later, I've got it working:

Would I recommend Travis-CI? Definitly yes. However, there are some pros and cons I like to point out:

Pros:

1. Each commit leads to a test build and if that fails you get notified.
2. Pull requests lead to test builds as well, so you'll see if a patch breaks something or not.
3. It's a free service, and a recource hungry at that. so - wow!
4. It just works and every problem can be fixed by yourself, given you know how to do it.
5. The Travis-CI documentation is outstanding with lots of examples.

Cons:

1. There's only one platform available for tests: Linux (and only one distribution: Ubuntu). It maybe possible to test on MacOSX, but as of this writing only upon request.
2. While you can choose between clang and gcc, you cannot test with different versions of the compilers.
3. Those language specific environments make it difficult to use with a project with multiple language needs.
4. There's no way to live test your .travis.yml file, like on a VM where you can login temporarily and initiate the test manually multiple times and tune the .travis.yml file til it works. Instead you've got to commit everything you like to try out, wait until Travis-CI tests are done, look on the site for the results and repeat the process. This iterative process comsumes a lot of time.
5. Travis-CI is a german startup in berlin and as always with german companies they do just not respond to emails. I mailed them because of the mixed language issue outlined above, but noting came back and I had to figure it out myself like a blind one in a foreign country (at least sometimes I felt like this during the process detailed in 3.).
6. Although they provide a Pro account for which you've to pay, I doubt the business model. Maybe the company will vanish overnight (or, worse, get sold and closed).