42
edits
Difference between revisions of "Blog/Kiwix Compile Farm"
Jump to navigation
Jump to search
m
no edit summary
(Created page with "''Published 14/06/2012 by ~~~'' Kiwix is a special software. Special because it's difficult to define: * A Desktop software for browsing offline content with ZIM on Mac, Wind...") |
m |
||
| Line 1: | Line 1: | ||
''Published 14/06/2012 by [[User:Kelson|Kelson]] ([[User talk:Kelson|talk]])'' | ''Published 14/06/2012 by [[User:Kelson|Kelson]] ([[User talk:Kelson|talk]])'' | ||
Kiwix is a special software | Kiwix is a special software because it is difficult to define: | ||
* A Desktop software for browsing offline content with ZIM on Mac, Windows and Linux. | * A Desktop software for browsing offline content with ZIM on Mac, Windows and Linux. | ||
* A Server allowing to serve ZIM content on those platforms + | * A Server allowing to serve ZIM content on those platforms + ARM Linux. | ||
* A library of ZIM files for popular content: | * A library of ZIM files for popular content: Wikipedia, Wikileaks, etc. | ||
* Very few developers (most of the time it's 2). | * Very few developers (most of the time it's 2). | ||
* Very large (and growing) number of users. | * Very large (and growing) number of users. | ||
| Line 31: | Line 31: | ||
* Linux 32/64b with dependencies (used to be a PPA for Ubuntu until they removed xulrunner). | * Linux 32/64b with dependencies (used to be a PPA for Ubuntu until they removed xulrunner). | ||
Knowing that only reg has a Mac, that the Windows setup for building Kiwix is complicated and that both Kelson and reg are using Linux, testing and distributing new versions of the code is very difficult. | Knowing that only reg has a Mac, that the Windows setup for building Kiwix is complicated, and that both Kelson and reg are using Linux, testing and distributing new versions of the code is very difficult. | ||
This is not a unique problem ; most large multi platform | This is not a unique problem; most large multi-platform software face the same issues and we did nothing but imitate them: we deployed a build farm. | ||
== The Solution == | == The Solution == | ||
| Line 45: | Line 45: | ||
== Buildbot == | == Buildbot == | ||
A [http://en.wikipedia.org/wiki/Build_farm compile farm] is a set of servers ; each building a platform or target of the software. | A [http://en.wikipedia.org/wiki/Build_farm compile farm] is a set of servers; each building a platform or target of the software. | ||
To manage those, [http://en.wikipedia.org/wiki/Comparison_of_continuous_integration_software a large number of software] exist. | To manage those, [http://en.wikipedia.org/wiki/Comparison_of_continuous_integration_software a large number of software] exist. | ||
After some research, we chose [http://trac.buildbot.net/ Buildbot] because: | After some research, we chose [http://trac.buildbot.net/ Buildbot] because: | ||
* | * Easy installion | ||
* It looked very powerful | * It looked very powerful | ||
* | * Clear documentation | ||
* It's written in Python (including the configuration file). | * It's written in Python (including the configuration file). | ||
| Line 62: | Line 62: | ||
* slaves which only needs to run the slave software (python). Those are logic-less. | * slaves which only needs to run the slave software (python). Those are logic-less. | ||
Kiwix already | Kiwix already rents a very powerful server in a data center for serving [http://download.kiwix.org/ downloads]. We used it to hold everything. | ||
== VirtualBox == | == VirtualBox == | ||
| Line 95: | Line 95: | ||
Take a look at ours as an example: [http://kiwix.svn.sourceforge.net/viewvc/kiwix/buildbot/master.cfg?revision=HEAD&view=markup master.cfg] | Take a look at ours as an example: [http://kiwix.svn.sourceforge.net/viewvc/kiwix/buildbot/master.cfg?revision=HEAD&view=markup master.cfg] | ||
We don't use any advanced | We don't use any advanced features so it's easy to understand. We chose: | ||
* fixed | * fixed daily times to run our builds (at night – server time) | ||
* builds (tarball, etc) are uploaded to the server's /var/www/ for direct web access. | * builds (tarball, etc) are uploaded to the server's /var/www/ for direct web access. | ||
Buildbot handles the transfer of files between master and slave. | Buildbot handles the transfer of files between master and slave. | ||
| Line 103: | Line 103: | ||
* Builds are announced and controllable by the IRC bot. | * Builds are announced and controllable by the IRC bot. | ||
Although it's simple, it | Although it's simple, it took a lot of tweaking and tests (fortunately it's easy) to write a good configuration file, you need to have a proper and documented build mechanism for all your targets otherwise you'll probably go crazy. We completed a complete rewrite of our autotools Makefiles for all platforms __before__ we setup buildbot. It sounds dumb but it's important. | ||
== Outcomes == | == Outcomes == | ||
| Line 114: | Line 114: | ||
If you intend to reproduce, here's a few things we've learned and want to share. | If you intend to reproduce, here's a few things we've learned and want to share. | ||
* Installing OSX on non-Mac hardware is tricky: you need a recent Intel CPU (support for VTx) but not too recent otherwise your OSX Install DVD won't know about it (and refuse to install). | * Installing OSX on non-Mac hardware is tricky: you need a recent Intel CPU (support for VTx) but not too recent otherwise your OSX Install DVD won't know about it (and refuse to install). | ||
* On OSX, Apple packages (MacOSX updates, XCode) have an expiration period. If you install an XCode version 2 years (that's an example) after it's been released, the installer will fail with no useful feedback. It's due to the package's signature being too old. You can still install it by unpacking/repacking the packages. | * On OSX, Apple packages (MacOSX updates, XCode) have an expiration period. If you install an XCode version 2 years (that's an example) after it's been released, the installer will fail with no useful feedback. It's due to the package's signature being too old. You can still install it by unpacking/repacking the packages. | ||
* SSH to your QEmu VM is done using a QEmu proxy so you ssh to localhost on a different port. | * SSH to your QEmu VM is done using a QEmu proxy so you ssh to localhost on a different port. | ||
* VRDP requires a good connexion if you intend to do a lot of configuration inside Windows (384k clearly is a pain!). | * VRDP requires a good connexion if you intend to do a lot of configuration inside Windows (384k clearly is a pain!). | ||
* Buildbot slaves freezes frequently. Not sure why but sometimes it fails to answer to build request and stays attached doing nothing. As a workaround, we delete and recreate the buildbot slave folder daily in a cron job. | * Buildbot slaves freezes frequently. Not sure why but sometimes it fails to answer to build request and stays attached doing nothing. As a workaround, we delete and recreate the buildbot slave folder daily in a cron job. | ||
* Buildbot slaves have network issues some times. We're not sure if it's related to buildbot, VBox or something else but it's frequent that the slave can't checkout the source tree or can't download our dependencies from the web. | * Buildbot slaves have network issues some times. We're not sure if it's related to buildbot, VBox or something else but it's frequent that the slave can't checkout the source tree or can't download our dependencies from the web. | ||
* Windows slave frequently loose connexion to the master. Might just be a Windows configuration issue. | * Windows slave frequently loose connexion to the master. Might just be a Windows configuration issue. | ||
== What's Next? == | == What's Next? == | ||
* Improve our wrapper script to handle VRDP access to VMs by controlling iptables. | * Improve our wrapper script to handle VRDP access to VMs by controlling iptables. | ||
* Add SSH to the Windows slave so we can do basic tests in console. | * Add SSH to the Windows slave so we can do basic tests in console. | ||
* Investigate the network/slaves problems so that it works 24/7. | * Investigate the network/slaves problems so that it works 24/7. | ||
* Automate & build a similar platform for the creation of ZIM files so we can focus only on code thereafter. | * Automate & build a similar platform for the creation of ZIM files so we can focus only on code thereafter. | ||