python-dbus and Continuous Integration

Recently in my Gnome GSoC 2015 project, I had to setup continuous integration for dbus in Travis, Scrutinizer and Cricle-CI. All three are pretty well known, and yet I had quite some difficulties in setting it up correctly.

The main issue with this arises due to the fact that they use virtualenv to handle python versions inside the Linux box they provide. Let me explain what that means. Automatic tests need to be run on the same python version that the program advertises that it supports. So, if your program’s documentation says that it supports python 3.4 it makes sense to run tests on the continuous integration platform with python 3.4 and not python 3.2

But the CI platform may or may not natively support that version – for example Travis uses Ubuntu 12.04 which doesn’t use the latest python 3.4 – the answer to this is using a virtual environment which has python 3.4 installed in it. So, essentially virtual environments are meant to give you an isolated workbench where you can run tests.

Now comes the difficult part with dbus. Dbus does not use python’s setup.py method of installation – it uses Autotools. There is an old bug report (#55439) about this which has a very old patch which doesn’t work anymore and was never accepted into the codebase. So I had to find an alternative.

PYTHON-DBUS

So, after lots of trials, it seems the best way to install python-dbus was to install it from source. And I wrote a shell script to do just that for circle-ci and travis:

echo Trying to find python version...

python_version=`python -c 'import sys; print(".".join(map(str, sys.version_info[:3])))'`

case "$python_version" in
  "3.4"*)
    system_python=python3.4
    ;;
  "3.3"*)
    system_python=python3.3
    ;;
  "3.2"*)
    system_python=python3.2
    ;;
  *)
    echo Python version was not understood. It was detected as - $python_version
    ;;
esac

sudo apt-get install ${system_python}-dev

sh .install.dbus.sh

echo Downloading python-dbus...
wget http://dbus.freedesktop.org/releases/dbus-python/dbus-python-1.2.0.tar.gz -O dbus-python.tar.gz -q
echo Unpacking python-dbus...
tar -zxvf dbus-python.tar.gz > /dev/null
rm dbus-python.tar.gz

mkdir -p python-tmpenv

python_virtualenv=$VIRTUAL_ENV
python_tmpenv=`pwd`/python-tmpenv

cd dbus-python-1.2.0

PYTHON=`sudo which ${system_python}` ./configure --prefix=$python_tmpenv
make
make install

cd ..

echo Copying files from the temporary python env to the virtualenv
cp -r $python_tmpenv/* $python_virtualenv

So, as you see – the first thing it does it finds the python version that the current virtualenv uses – and has a switch case to find the corresponding binary for that version. Then it installs the -dev package for it which installs that python version’s headers into the system (NOT the virtualenv). Then after downloading and extracting the python-dbus source code it uses

sudo which ${system_python}

To find the system’s python binary (not the python binary inside the virtualenv) as this binary has the headers along with it and compiles python-dbus using the python in the system. Now it copies these compiled files into the virtualenv.

Scrutinizer was a little different in it’s configurations and did not have the $VIRTUALENV variable set, so there’s an extra if condition to handle that. Also, scrutinizer and travis needed the deadsnakes repository for the pythonXX-dev packages. In Scrutinizer add-apt-repository can be used, but travis needs the following to be added to travis.yml :

addons:
  apt:
    sources:
    - deadsnakes

Another issue I did face with this was that it did not have the dbus headers to compile python-dbus. In scrutinizer and circle-ci this was easily fixed by

sudo apt-get install libdbus-glib-1-dev

But in Travis, this installed dbus-1.4 (while python-dbus needs dbus-1.6). So, I wrote a similar script to install dbus from source too.

echo Downloading dbus...
wget http://dbus.freedesktop.org/releases/dbus/dbus-1.6.30.tar.gz -O dbus.tar.gz -q
echo Unpacking dbus...
tar -zxvf dbus.tar.gz > /dev/null
rm dbus.tar.gz

cd dbus-1.6.30

./configure
make
sudo make install

cd ..

But now, we need to switch on dbus for travis to use this new dbus. This needs to be done on the same line where you call your tests, because travis runs each line on a new shell (like make). So, I had to change the original

python3 ./run_tests.py --cover --disallow-test-skipping

to

dbus-launch python3 ./run_tests.py --cover --disallow-test-skipping

And that got it up and running 🙂

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s