Difference between revisions of "Black&White Project/Compilation Farm/Setup"

From Kiwix
Jump to navigation Jump to search
 
(34 intermediate revisions by 3 users not shown)
Line 6: Line 6:
* A buildbot master running as user buildbot on the server.
* A buildbot master running as user buildbot on the server.
* A set of VirtualBox VMs running buildbot slaves.
* A set of VirtualBox VMs running buildbot slaves.
** '''ubuntu32''': i686 deb, i686 static
** '''ubuntu32''' (192.168.5.50): i686 static
** '''ubuntu64''': x86_64 deb, x86_64 static
** '''ubuntu64''' (192.168.5.51): x86_64 static
** '''macosx''': universal (i386+x86_64) dmg with app.
** '''macosx''' (192.168.5.52): universal (i386+x86_64) dmg with app.
** '''win32''': i386 windows static
** '''win32''' (192.168.5.53): i386 windows static
** '''win64''': x64 windows static
** '''win64''' (192.168.5.54): x64 windows static
** '''fedora''': i686 sugar static
** '''fedora''' (192.168.5.55): i686 sugar static (''kiwixdev'')
** '''arm''': armv5 kiwix-serve static
** '''arm''' (no direct IP. ssh on localhost through port 5022 ''10.0.2.2'' ''192.168.5.56''): armel kiwix-serve static
** '''debian32''' (192.168.5.57): i686 wheezy deb
** '''debian64''' (192.168.5.58): x86_64 wheezy deb
 
=== Usage ===
 
A utility script, ''kbox'' is present to facilitate start and stop of the VMs.
 
<code><pre>(buildbot)buildbot@www:~/buildbot$ kbox --help
Usage: kbox [options]
 
Handy wrapper around virtualbox for kiwix
 
Options:
  -h, --help  show this help message and exit
  --listlocal  list names of VMs. Edit script to change.
  --listall    list all VMs with their status.
  --usb        list all USB devices on the host.
  --startall  start all VMs.
  --stopall    sends shutdown to all VMs.
  --start=VM  starts the VM named [VM].
  --stop=VM    sends shutdown to the VM named [VM].
  --info=VM    displays info for the VM named [VM].
  --ssh=VM    connects via SSH to the VM named [VM].</pre></code>
 
* Example connection to ''ubuntu32'' VM:
<code><pre>kbox --ssh ubuntu32</pre></code>


== Master Setup ==
== Master Setup ==


* Install required packages
* Install required packages
<code><pre># apt-get install build-essential subversion zip unzip python-pip vim python-dev
<code><pre># apt-get install build-essential git zip unzip python-pip vim python-dev
pip install virtualenv virtualenvwrapper</pre></code>
pip install virtualenv virtualenvwrapper</pre></code>
* Create user ''buildbot''
* Create user ''buildbot''
Line 34: Line 60:
buildbot create-master master</pre></code>
buildbot create-master master</pre></code>
* Deploy configuration
* Deploy configuration
<code><pre>svn co https://kiwix.svn.sourceforge.net/svnroot/kiwix/buildbot/ code
<code><pre>git archive --prefix=code/ --remote git://git.code.sf.net/p/kiwix/kiwix HEAD:buildbot | tar x
cd master && ln -sf ../code/master.cfg master.cfg && cd -
cd master && ln -sf ../code/master.cfg master.cfg && cd -
cp code/secret_pass.py master/secret_pass.py</pre></code>
cp code/secret_pass.py master/secret_pass.py</pre></code>
Line 40: Line 66:
* Start the master
* Start the master
<code><pre>buildbot start master</pre></code>
<code><pre>buildbot start master</pre></code>
* Create an nginx proxy at '''/etc/nginx/sites-available/buildbot.kiwix.org'''
<code><pre>server {
  listen localhost:81;
  listen 88.190.13.50:80;
  server_name buildbot.kiwix.org;
  access_log off;
  proxy_buffering off;
  proxy_max_temp_file_size 0;
  set_real_ip_from 127.0.0.1;
  set_real_ip_from 88.190.13.50;
  real_ip_header X-Forwarded-For;
  location / {
    proxy_set_header X-Real-IP  $remote_addr;
    proxy_set_header X-Forwarded-For $remote_addr;
    proxy_pass http://localhost:8010;
    proxy_set_header Host $host;
  }
}</pre></code>
=== Automatic Startup ===
* ''crontab -e'' on ''builbot'' user
<code><pre>@reboot /home/buildbot/envs/buildbot/bin/buildbot start /home/buildbot/buildbot/master</pre></code>


== Slave Setup ==
== Slave Setup ==
Line 45: Line 97:


=== Linux Slave Setup ===
=== Linux Slave Setup ===
* Create a VirtualBox VM
* Create a VirtualBox VM (20GB of storage - dynamic)
* Install Ubuntu 10.04.4 (Newer versions of Ubuntu uses a recent glibc)
* Install Ubuntu 10.04.4 (Newer versions of Ubuntu uses a recent glibc)
* Select OpenSSH server during install.
* Select OpenSSH server during install.
Line 51: Line 103:
* Use whole disk partitioning
* Use whole disk partitioning
* Install packages:
* Install packages:
<code><pre># apt-get install --fix-missing build-essential autoconf automake libtool pkg-config fastjar libssl-dev libexpat1-dev libxml2-dev libgnutls-dev libgpg-error-dev libgcrypt-dev uuid-dev libuuid1 subversion zip unzip python-pip vim python-dev</pre></code>
<code lang="bash"><pre># apt-get install --fix-missing acpi acpid build-essential autoconf automake libtool pkg-config fastjar libssl-dev  
libexpat1-dev libxml2-dev libgnutls-dev libgpg-error-dev libgcrypt-dev uuid-dev libuuid1 git zip unzip python-pip vim python-dev</pre></code>
* Install python tools
* Install python tools
<code><pre># pip install virtualenv virtualenvwrapper</pre></code>
<code><pre># pip install virtualenv virtualenvwrapper</pre></code>
Line 62: Line 115:


export WORKON_HOME=~/envs
export WORKON_HOME=~/envs
source /usr/local/bin/virtualenvwrapper.sh</pre></code>
source /usr/local/bin/virtualenvwrapper.sh
source ${WORKON_HOME}/buildbot/bin/activate</pre></code>
* Log out then log-in again (to activate virtualenvwrapper)
* Log out then log-in again (to activate virtualenvwrapper)
* Create virtualenv
* Create virtualenv
Line 70: Line 124:
* Create the slave
* Create the slave
<code><pre>mkdir -p buildbot && cd buildbot
<code><pre>mkdir -p buildbot && cd buildbot
buildslave create-slave slave SERVER_IP:9989 BUILDER_ID BUILDER_PASS</pre></code>
buildslave create-slave slave 192.168.5.1:9989 ubuntu32 BUILDER_PASS</pre></code>
* Edit files in ''slave/info/'' and put host description and maintainer
* Edit files in ''slave/info/'' and put host description and maintainer
* Start the slave
* Start the slave
<code><pre>cd slave/
<code><pre>cd slave/
buildslave start</pre></code>
buildslave start</pre></code>
=== Automatic Startup ===
* ''crontab -e'' on ''dev'' user
<code><pre>@reboot /home/dev/envs/buildbot/bin/buildslave start /home/dev/buildbot/slave</pre></code>
== VirtualBox Setup ==
* Install the non-free version of Virtual Box:
<code><pre>wget -q http://download.virtualbox.org/virtualbox/debian/oracle_vbox.asc -O- | sudo apt-key add -
echo "deb http://download.virtualbox.org/virtualbox/debian lucid contrib non-free" >> /etc/apt/sources.list
apt-get update
apt-get install virtualbox-4.1
apt-get install linux-headers
/etc/init.d/vboxdrv setup
wget -c http://download.virtualbox.org/virtualbox/4.1.14/Oracle_VM_VirtualBox_Extension_Pack-4.1.14-77440.vbox-extpack
VBoxManage extpack install /home/buildbot/vms/Oracle_VM_VirtualBox_Extension_Pack-4.1.14-77440.vbox-extpack</pre></code>
* VMs will be stored in /home/buildbot/vms
* Configure Network
<code><pre>VBoxManage hostonlyif create
VBoxManage hostonlyif ipconfig vboxnet0 --ip 192.168.5.1 --netmask 255.255.255.0
VBoxManage dhcpserver add --ifname vboxnet0 --ip 192.168.5.1 --netmask 255.255.255.0 --lowerip 192.168.5.10 --upperip 192.168.5.50 --enable</pre></code>
* Create a HDD Image
<code><pre>VBoxManage createhd --filename ~/vms/ubuntu32/ubuntu32.vdi --size 20480 --variant Standard</pre></code>
* Create VM (VBoxManage list ostypes for types list)
<code><pre>VBoxManage createvm --register --name ubuntu32 --ostype Ubuntu --basefolder ~/vms/
VBoxManage modifyvm ubuntu32 --memory 512 --audio none --clipboard disabled --usb off --usbehci off --nic1 nat --nictype1 82543GC --nic2 hostonly --nictype2 82543GC --hostonlyadapter2 vboxnet0 --vrde on --boot1 dvd --boot2 disk
VBoxManage storagectl ubuntu32 --name hdd --add sata --controller IntelAhci --bootable on
VBoxManage storagectl ubuntu32 --name cd --add ide --controller PIIX4 --bootable on
VBoxManage storageattach ubuntu32 --storagectl hdd --type hdd --port 1 --medium ~/vms/ubuntu32/ubuntu32.vdi
VBoxManage storageattach ubuntu32 --storagectl cd --type dvddrive --port 1 --device 1 --medium ~/vms/ubuntu-10.04.4-server-i386.iso
</pre></code>
* Start VM & Install System using rdesktop
<code><pre>VBoxManage startvm ubuntu32 --type headless</pre></code>
On your computer, connect with:
<code><pre>rdesktop download.kiwix.org:3389</pre></code>
* Configure Network (eth1) with a static IP according to mapping above in ''/etc/network/interfaces''
<code><pre>iface eth1 inet static
    address    192.168.5.50
    netmask    255.255.255.0
    gateway    192.168.5.1
auto eth1
</pre></code>
* Once installed, tweak the VM
<code><pre>VBoxManage storageattach ubuntu32 --storagectl cd --type dvddrive --port 1 --device 1 --medium none
VBoxManage modifyvm ubuntu32  --vrde off</pre></code>
From now, connection to the VM will be SSH only.
== Post B&W todo ==
* Start VM automaticaly/transparently if a compilation is required - otherwise VM is offline.
* Start chained compilations : only the first one is planned, and all other ones occur one by one afterwards.
* Bump code with bumpVersion.sh to have a version like svn-[date]
* Rename bin packages
== Tips ==
* win32 upload issue: VBoxManage modifyvm win32 --natsettings1 1500,64,64,64,1024
* [https://people.debian.org/~aurel32/qemu/armel/ qemu arm VM]

Latest revision as of 15:31, 24 February 2015

Organization

The Compilation Farm is installed on the download.kiwix.org server. It is organized as:

  • A buildbot master running as user buildbot on the server.
  • A set of VirtualBox VMs running buildbot slaves.
    • ubuntu32 (192.168.5.50): i686 static
    • ubuntu64 (192.168.5.51): x86_64 static
    • macosx (192.168.5.52): universal (i386+x86_64) dmg with app.
    • win32 (192.168.5.53): i386 windows static
    • win64 (192.168.5.54): x64 windows static
    • fedora (192.168.5.55): i686 sugar static (kiwixdev)
    • arm (no direct IP. ssh on localhost through port 5022 10.0.2.2 192.168.5.56): armel kiwix-serve static
    • debian32 (192.168.5.57): i686 wheezy deb
    • debian64 (192.168.5.58): x86_64 wheezy deb

Usage

A utility script, kbox is present to facilitate start and stop of the VMs.

(buildbot)buildbot@www:~/buildbot$ kbox --help
Usage: kbox [options]

Handy wrapper around virtualbox for kiwix

Options:
  -h, --help   show this help message and exit
  --listlocal  list names of VMs. Edit script to change.
  --listall    list all VMs with their status.
  --usb        list all USB devices on the host.
  --startall   start all VMs.
  --stopall    sends shutdown to all VMs.
  --start=VM   starts the VM named [VM].
  --stop=VM    sends shutdown to the VM named [VM].
  --info=VM    displays info for the VM named [VM].
  --ssh=VM     connects via SSH to the VM named [VM].
  • Example connection to ubuntu32 VM:
kbox --ssh ubuntu32

Master Setup

  • Install required packages
# apt-get install build-essential git zip unzip python-pip vim python-dev
pip install virtualenv virtualenvwrapper
  • Create user buildbot
  • Log-in as user buildbot
  • Edit .bashrc and add at the end
export EDITOR="vim -v"

export WORKON_HOME=~/envs
source /usr/local/bin/virtualenvwrapper.sh
  • Log-out then log-in to enable virtualenvwrapper
  • Create virtualenv
mkvirtualenv --no-site-packages buildbot
mkdir -p buildbot && cd buildbot
  • Install builbot and create master
easy_install buildbot
buildbot create-master master
  • Deploy configuration
git archive --prefix=code/ --remote git://git.code.sf.net/p/kiwix/kiwix HEAD:buildbot | tar x
cd master && ln -sf ../code/master.cfg master.cfg && cd -
cp code/secret_pass.py master/secret_pass.py
  • Edit the password file in master/secret_pass.py
  • Start the master
buildbot start master
  • Create an nginx proxy at /etc/nginx/sites-available/buildbot.kiwix.org
server {
  listen localhost:81;
  listen 88.190.13.50:80;
  server_name buildbot.kiwix.org;

  access_log off;

  proxy_buffering off;
  proxy_max_temp_file_size 0;

  set_real_ip_from 127.0.0.1;
  set_real_ip_from 88.190.13.50;
  real_ip_header X-Forwarded-For;

  location / {
    proxy_set_header X-Real-IP  $remote_addr;
    proxy_set_header X-Forwarded-For $remote_addr;
    proxy_pass http://localhost:8010;
    proxy_set_header Host $host;
  }
}

Automatic Startup

  • crontab -e on builbot user
@reboot /home/buildbot/envs/buildbot/bin/buildbot start /home/buildbot/buildbot/master

Slave Setup

buildbot slave does not require much: a working python/setuptools environnment.

Linux Slave Setup

  • Create a VirtualBox VM (20GB of storage - dynamic)
  • Install Ubuntu 10.04.4 (Newer versions of Ubuntu uses a recent glibc)
  • Select OpenSSH server during install.
  • Create a regular user (dev)
  • Use whole disk partitioning
  • Install packages:
# apt-get install --fix-missing acpi acpid build-essential autoconf automake libtool pkg-config fastjar libssl-dev 
libexpat1-dev libxml2-dev libgnutls-dev libgpg-error-dev libgcrypt-dev uuid-dev libuuid1 git zip unzip python-pip vim python-dev
  • Install python tools
# pip install virtualenv virtualenvwrapper

From now, you won't need root privileges anymore.

  • Log in as dev user
  • Edit ~.bashrc and add at the end
export EDITOR="vim -v"

export WORKON_HOME=~/envs
source /usr/local/bin/virtualenvwrapper.sh
source ${WORKON_HOME}/buildbot/bin/activate
  • Log out then log-in again (to activate virtualenvwrapper)
  • Create virtualenv
mkvirtualenv --no-site-packages buildbot
  • Install buildbot
easy_install buildbot-slave
  • Create the slave
mkdir -p buildbot && cd buildbot
buildslave create-slave slave 192.168.5.1:9989 ubuntu32 BUILDER_PASS
  • Edit files in slave/info/ and put host description and maintainer
  • Start the slave
cd slave/
buildslave start

Automatic Startup

  • crontab -e on dev user
@reboot /home/dev/envs/buildbot/bin/buildslave start /home/dev/buildbot/slave

VirtualBox Setup

  • Install the non-free version of Virtual Box:
wget -q http://download.virtualbox.org/virtualbox/debian/oracle_vbox.asc -O- | sudo apt-key add -
echo "deb http://download.virtualbox.org/virtualbox/debian lucid contrib non-free" >> /etc/apt/sources.list
apt-get update
apt-get install virtualbox-4.1
apt-get install linux-headers
/etc/init.d/vboxdrv setup
wget -c http://download.virtualbox.org/virtualbox/4.1.14/Oracle_VM_VirtualBox_Extension_Pack-4.1.14-77440.vbox-extpack
VBoxManage extpack install /home/buildbot/vms/Oracle_VM_VirtualBox_Extension_Pack-4.1.14-77440.vbox-extpack
  • VMs will be stored in /home/buildbot/vms
  • Configure Network
VBoxManage hostonlyif create
VBoxManage hostonlyif ipconfig vboxnet0 --ip 192.168.5.1 --netmask 255.255.255.0
VBoxManage dhcpserver add --ifname vboxnet0 --ip 192.168.5.1 --netmask 255.255.255.0 --lowerip 192.168.5.10 --upperip 192.168.5.50 --enable
  • Create a HDD Image
VBoxManage createhd --filename ~/vms/ubuntu32/ubuntu32.vdi --size 20480 --variant Standard
  • Create VM (VBoxManage list ostypes for types list)
VBoxManage createvm --register --name ubuntu32 --ostype Ubuntu --basefolder ~/vms/
VBoxManage modifyvm ubuntu32 --memory 512 --audio none --clipboard disabled --usb off --usbehci off --nic1 nat --nictype1 82543GC --nic2 hostonly --nictype2 82543GC --hostonlyadapter2 vboxnet0 --vrde on --boot1 dvd --boot2 disk
VBoxManage storagectl ubuntu32 --name hdd --add sata --controller IntelAhci --bootable on
VBoxManage storagectl ubuntu32 --name cd --add ide --controller PIIX4 --bootable on
VBoxManage storageattach ubuntu32 --storagectl hdd --type hdd --port 1 --medium ~/vms/ubuntu32/ubuntu32.vdi
VBoxManage storageattach ubuntu32 --storagectl cd --type dvddrive --port 1 --device 1 --medium ~/vms/ubuntu-10.04.4-server-i386.iso
  • Start VM & Install System using rdesktop
VBoxManage startvm ubuntu32 --type headless

On your computer, connect with:

rdesktop download.kiwix.org:3389
  • Configure Network (eth1) with a static IP according to mapping above in /etc/network/interfaces
iface eth1 inet static
    address    192.168.5.50
    netmask    255.255.255.0
    gateway    192.168.5.1
auto eth1
  • Once installed, tweak the VM
VBoxManage storageattach ubuntu32 --storagectl cd --type dvddrive --port 1 --device 1 --medium none
VBoxManage modifyvm ubuntu32  --vrde off

From now, connection to the VM will be SSH only.

Post B&W todo

  • Start VM automaticaly/transparently if a compilation is required - otherwise VM is offline.
  • Start chained compilations : only the first one is planned, and all other ones occur one by one afterwards.
  • Bump code with bumpVersion.sh to have a version like svn-[date]
  • Rename bin packages

Tips

  • win32 upload issue: VBoxManage modifyvm win32 --natsettings1 1500,64,64,64,1024
  • qemu arm VM