Black&White Project/Compilation Farm/Setup

From Kiwix
Jump to navigation Jump to search

Organization

The Compilation Farm is installed on the download.kiwix.org server. It is organized as:

  • A buildbot master running as user buildbot on the server.
  • A set of VirtualBox VMs running buildbot slaves.
    • ubuntu32: i686 deb, i686 static
    • ubuntu64: x86_64 deb, x86_64 static
    • macosx: universal (i386+x86_64) dmg with app.
    • win32: i386 windows static
    • win64: x64 windows static
    • fedora: i686 sugar static
    • arm: armel kiwix-serve static

Master Setup

  • Install required packages
# apt-get install build-essential subversion zip unzip python-pip vim python-dev
pip install virtualenv virtualenvwrapper
  • Create user buildbot
  • Log-in as user buildbot
  • Edit .bashrc and add at the end
export EDITOR="vim -v"

export WORKON_HOME=~/envs
source /usr/local/bin/virtualenvwrapper.sh
  • Log-out then log-in to enable virtualenvwrapper
  • Create virtualenv
mkvirtualenv --no-site-packages buildbot
mkdir -p buildbot && cd buildbot
  • Install builbot and create master
easy_install buildbot
buildbot create-master master
  • Deploy configuration
svn co https://kiwix.svn.sourceforge.net/svnroot/kiwix/buildbot/ code
cd master && ln -sf ../code/master.cfg master.cfg && cd -
cp code/secret_pass.py master/secret_pass.py
  • Edit the password file in master/secret_pass.py
  • Start the master
buildbot start master
  • Create an nginx proxy at /etc/nginx/sites-available/buildbot.kiwix.org
server {
  listen localhost:81;
  listen 88.190.13.50:80;
  server_name buildbot.kiwix.org;

  access_log off;

  proxy_buffering off;
  proxy_max_temp_file_size 0;

  set_real_ip_from 127.0.0.1;
  set_real_ip_from 88.190.13.50;
  real_ip_header X-Forwarded-For;

  location / {
    proxy_set_header X-Real-IP  $remote_addr;
    proxy_set_header X-Forwarded-For $remote_addr;
    proxy_pass http://localhost:8010;
    proxy_set_header Host $host;
  }
}

Automatic Startup

  • crontab -e on builbot user
@reboot /home/buildbot/envs/buildbot/bin/buildbot start /home/buildbot/buildbot/master

Slave Setup

buildbot slave does not require much: a working python/setuptools environnment.

Linux Slave Setup

  • Create a VirtualBox VM (20GB of storage - dynamic)
  • Install Ubuntu 10.04.4 (Newer versions of Ubuntu uses a recent glibc)
  • Select OpenSSH server during install.
  • Create a regular user (dev)
  • Use whole disk partitioning
  • Install packages:
# apt-get install --fix-missing build-essential autoconf automake libtool pkg-config fastjar libssl-dev libexpat1-dev libxml2-dev libgnutls-dev libgpg-error-dev libgcrypt-dev uuid-dev libuuid1 subversion zip unzip python-pip vim python-dev
  • Install python tools
# pip install virtualenv virtualenvwrapper

From now, you won't need root privileges anymore.

  • Log in as dev user
  • Edit ~.bashrc and add at the end
export EDITOR="vim -v"

export WORKON_HOME=~/envs
source /usr/local/bin/virtualenvwrapper.sh
  • Log out then log-in again (to activate virtualenvwrapper)
  • Create virtualenv
mkvirtualenv --no-site-packages buildbot
  • Install buildbot
easy_install buildbot-slave
  • Create the slave
mkdir -p buildbot && cd buildbot
buildslave create-slave slave SERVER_IP:9989 BUILDER_ID BUILDER_PASS
  • Edit files in slave/info/ and put host description and maintainer
  • Start the slave
cd slave/
buildslave start


VirtualBox Setup

Install the non-free version of Virtual Box:

wget -q http://download.virtualbox.org/virtualbox/debian/oracle_vbox.asc -O- | sudo apt-key add -
echo "deb http://download.virtualbox.org/virtualbox/debian lucid contrib non-free" >> /etc/apt/sources.list
apt-get update
apt-get install virtualbox-4.1