Talk:Library

Network challenges

The library deals with two types of online content:

• The catalog, containing the list of content (possibility) available online. The library is an XML file which can be downloaded per HTTP from a fix URL (configured in Kiwix), many URLs could be configured. For now the only one catalog is available at http://www.kiwix.org/library.xml
• The ZIM files themself. Their location is given in the online library. They are available using Metalink. Metalink files are XML files listing different locations/protocols to access the same content. Metalink is though to allow robust mirroring. On our server the metalink files are served using Mirrorbrain (example). In Kiwix the Metalink files are downloaded using HTTP and then given to aria2x which is embedded in Kiwix. Aria2c is a command line tool able to understand Metalink and download the content from different sources using HTTP/FTP/Bittorrent.

Long term

The vision for the library is that it works correctly in full decentralized mode (without central node). For now we fail because:

• Mirrobrain haa some small issues with Bittorrent
• aria2c is not usable in Android (we need somehow libaria2 - but the dev has started on this and a few things are already available to test)
• We have no solution to deal with a decentralized catalogue (I think that a custom Kademilia network could be a solution...)

So these are the things we need to fix on a long term perspective.

Short term

Android needs "urgently" an internal library. The main reason is that people get difficulties to download big ZIM files on FAT32 SD cards; but an internal library is also great to get an overview, upgrades, etc... But we won't be able to make something so great like on the PC version (aria2c is not available) and it doesn't make sense to develop too much stuff and then remove it in 12 months. The best approach is probably to write a minimal library.xml parser and a minimal metalink parser. The downloader should only use Android HTTP primitives... but it would be really good to implemented the automatic 2GB splitting to avoid problems with fat32. Security to avoid to high data transfer is mandatory.

Visualisation/Storage challenges

• We use for now XML as a storage backend. This is slow and not adapted to big databases. We should change this to something more efficient like SQLlite. This should be implemented in C++ and provided to all our versions of kiwix: PC, kiwix-serve, kiwix for Android.
• The visualization should be improved for almost all our version (especially for kiwix-serve). Kiwix-serve should allow to download the content directly from this overview.