KiwixKolibri

Introduction

This page is about hackathon projects that integrate Kiwix and Kolibri.

General context

Things that currently exist:

• KA lite content packs
• Content channels (Kolibri) and zim files (Kiwix)
• Content curation server + sushi chefs content pipeline
• ZimFiles listings (formats?)
• IIAB software
• Kiwix server and mobile apps
• sharable linkable URLs (content level links) example link
• Kolibri Search API

Questions:

• Does Kiwix have ability to do deep links?
  • Mobile app deep links
  • What is the kiwix-server content link format
• Can a zim file contain other zim files?
• What are good sample code examples that use python bindings to libzim for metadata/read/write

User scenarios

Backend content integration

This importing/exporting currently existing content.

Context:

• Kiwix content repository running in the cloud
• Scripts (zim chefs) for creating OpenZim file formats from various sources
• Kolibri Content curation server running in the cloud
• Chef scripts that package various sources as kolibri channels

USER STORY 1.1: As a Sushi Chef author I want to import content from ZimFiles

• read OPDS listing to know what OpenZIM files are available
• use libzim to extract contents
• use chef script to package contents as channel
• upload to content curation server

UPDATE: Not a clear use case? Why import from ZimFile and not original source? Maybe useful if you want to mix content from ZimFiles and other content sources (e.g. KA videos + wikipedia pages)

USER STORY 1.2: As a ZimFile author I want to import content from Kolibri channels

• read CCServer listing to find listed channels
• download content database + storage (need new code that simulates what kolibri does)
• use libzim to package content as OpenZIM file
• upload to Kiwix content repository

UPDATE: could be useful to create subject-specific channels to distribute as ZimFiles (for their small size).

USER STORY 1.3: As a subject specialist (e.g. medical wikipedia) I want to use the content curation server to remix an content from existing kolibri channel and zim files

• Prerequisite: USER STORY 1.1/1.2 upload of content
• use CC Server to remix content from multiple channels
• publish and view channel in Kolibri
• can also export as openzimfile (e.g. to include in a standalone Kiwix app)

Discussion points:

• Possible integration of available content via OPDS (ask IIAB/Kiwix/RACHEL ppl)
• Aligning standards of exporting meta data to OPDS, such that Kiwix Zim resources and Kolibri Channels can be searched and browsed through the same interfaces. Collaborate with Tim Moody on this.

Exporting Kolibri channels to Zim

Static contents like videos in Khan Academy or Sikana.tv can be exported as a static zim, given we use the meta data (topic trees) for navigation + some template parameters like title, logo and colors.

Rendering Zim files in Kolibri

Kolibri rendered access data from wherever Zim files are stored (maybe preferable to link to Kiwix deep-link / shallow links)

Importing from a Zim into a Kolibri channel

Importing a OpenZim file into a channel in the Kolibri Content Curation server: Build a "sushi chef" script (using the ricecooker) to create a topic tree by traversing the content within the Zim. Depending on the format of the content in the zim, this might be a topic tree with videos, or in the Kolibri "HTML5 app" format, a set of pages that will be iframed. May not be very useful for something like all of Wikipedia, but can be useful for smaller Zim projects (or ones with a lot of rich media files).

• Benefit: Someone can start creating exercises and re-organizing for a specific educational purpose
• Benefit: Other channels can include new learning materials from already existing Zim resources

School server integration

Context:

• Kolibri running on port 8080
• Kiwix running on 8000

Shallow-linking (channel/zimfile-level linking)

GOAL: implement working prototype during hackathon

Detect that Kiwix is installed, get an index of OpenZIM bundles and their metadata, and show links to the corresponding Kiwix URLs alongside the Kolibri channel list. This seems like a good first step. Detecting that the other is running isn't as simple as it might seem, as people run on various ports. An interesting challenge with a clear but simple outcome.

USER STORY 2.1.1: Search in Kiwix also surfaces Kolibri content channels

• User is Kiwix main channel search interface
• User sees a mix of OpenZim files available and Kolibri channels
• Clicking on a Kolibri channel in the listing, takes user to Kolibri channel import screen

USER STORY 2.1.2: Search in Kolibri also surfaces Kiwix content channels

(same as above but reversed)

USER STORY 2.2: As a student completing a Kolibri "research exercise" activity, I want to access Kiwix content for research.

• Open exercise in Kolibri interface, e.g. what is bleach made of?
• Use Kiwix to search for content/answers
• Switch back to Kolibri tab to enter answer

Deep-linking

GOAL: discuss but leave for future session since too big project for five days.

A more general version of cross-linking to resources between the platforms. Possibly based either on a GUID or URI approach (which will likely be implemented in some form for cross-linking within Kolibri channels anyway), or through shared metadata/tagging schemes.

From Kiwix: Say you export a zim of a Kolibri channel which has an exercise, then we can use a placeholder with a deep link back to Kolibri.

From Kolibri: Instead of trying to fit in a Wikipedia zim, we can find a mechanism to add links as related contents. Like "Want to learn more about atomic fusion, read about it on Wikipedia".

USER STORY 2.3.1: Kiwix user is watching a TED talk video on programming.

• Kiwix calls a "related content" Kolibri API endpoint to find related items in Kolibri
• Suggested/related items: "watch an intro to programming from KA"
• Link to Kolibri channel or content item
• Kolibri Content-level links are fairly consistent. They can also be constructed from searching this endpoint: https://kolibridemo.learningequality.org/api/content/197934f144305350b5820c7c4dd8e194/contentnode/?page=1&page_size=20&search=physics

USER STORY 2.3.2: Kolibri user is watching an introduction to programming KA video

• Kolibri calls “related content” on Kiwix API endpoint
• Suggested/related items: "watch an Video programming TED Talk"
• Link to Kiwix ZimFile or directly to content item

USER STORY 2.4: As a Kiwix user I want to access Perseus exercises.

• While browsing KA channel in Kiwix, I encounter a perseus exercise
• Link text "open in Kolibri" <a href=”Template:KOLIBRI URL/exercise_id”>
• At Kiwix boot time it check if Kolibri is also installed and sets KOLIBRI_URL appropriately
• Kolibri renders exercise form Kolibri content dir
• Student solves exercise in Kolibri
• Note: This idea doesn't make too much sense: presumably OpenZim data distribution for videos only, and no exercises. Maybe this user story is a special case of USER STORY 2.3.1? (once we have content-level linking)

USER STORY 2.5. As a student learning about "Topic X" in Kolibri, I can search for supporting material withing the Kiwix installation

• Working on exercise in Kolibri interface
• Click on "Search Library" function
• API call to Kiwix indexes to search within all OpenZim files installed in Kiwix
• Clicking on a result opens resource in Kiwix (in new tab)
• Switch back to Kolibri tab to finish exercise

Android integration

• How can we know if both apps are installed?
• Discuss future deep linking between apps
  • See APPENDIX A with custom scheme for ACTION_VIEW
  • `kiwix:/` and `kolibri:/` link format seems possible
  • TODO; look at how server links work see if we can use same structure
• Note: better pursue content backend integration and server integration instead of Android since the dominant use case is "small server through the web"

Dev setup for hackathon

1. Kolibri , docs , demo
2. Content Curation Server

TODO: before hackathon

• Docker compose file for content pipeline (fast install dev environment)
  • CCServer
  • Kolibri
  • KA-lite (KA-content packs?)
  • Kiwix
  • Example sushichef in dev mode project (install -e .)
  • Example openzim project in dev mode project (install -e .)
  • High level intro to Kolibri and Kolibri content pipeline. Explain using above setup how content starts from a local dir + youtube links into a running Kolibri instance.
• Update KiwixKolibri wiki page
• Link to KiwixKolibri page from main event page

(more notes to be added soon)

Trying Kolibri

Online

Please note that Kolibri is alpha software - we do not recommend deploying it before later this year. Also, please do not share/promote it externally yet, as we are still stabilizing it and streamlining the UX for public usability.

Using locally:

1. Download the latest .pex release: https://github.com/learningequality/kolibri/releases
2. Make the file executable: chmod +x kolibri-v0.4.3.pex
3. Run Kolibri ./kolibri-v0.4.3.pex start
4. Navigate with your browser to http://localhost:8080

Once installed, you can add content by logging in as an administrator and using one of the many available Channel IDs... these are just a few of them, we should be able to have more

Khan Academy small test:      8b4d3e6d3d4842ba8ea658335b5dd252
Khan Academy English Math:    1ceff53605e55bef987d88e0908658c5
Khan Academy Swahili Math:    ec164fee25ee526296e68f7c10b1e169
Khan Academy with exercises:  f6d2f857af2e304093648daf2d8cacec
PhET Interaction Simulations: 197934f144305350b5820c7c4dd8e194
CK-12:                        1d8f6d84618153c18c695d85074952a7
MIT Blossoms:                 913efe9f14c65cb1b23402f21f056e99
Sikana in Chinese (ZH):       3e9ffc29aa0b59c3bda8d8c7ed179685
Sikana in French (FR):        8ef625db6e86506c9a3bac891e413fff
Sikana in English (EN):       3e464ee12f6a50a781cddf59147b48b1

In case you find an issue, feel free to ask in the community forum or file an issue on Github.

More ideas

• Special purpose Zim files for Wikipedia content that we can link to from within Kolibri content.
  • USE CASE: a Kolibri channel on fractions fractions + associated zim file
  • channel is a topic tree + videos + exercises + new node type (KiwixRef)
  • e.g. KiwixRef link to Wikipedia page fractions opens Kiwix in a new tab
  • Kiwix renders a custom ZIMfile that conains fractions + 3 links deep

• Include content from ZIM files in Koibri
  • Example use case: in order to avoid duplication of supporting code,

Appendix A: Technical notes about deep-linking integration on Android =

Using Intents to cross-link between the Kolibri and Kiwix apps. Useful links:

1. Intent basics
2. Intent class
3. intent-filters
4. list of "common" intents, for example Camera
5. we're interested in the ACTION_VIEW intent: ACTION VIEW, the view's URI can be resolved with a custom schema
6. Article about ACTION_VIEWs using a custom scheme

We can define two custom `scheme`s such that:

• ACTION_VIEW intents that start with `kolibri://<path>` will be handled by the Kolibiri app (see below)
• ACTION_VIEW intents starting with `kiwix://<path>` will be handled by the Kiwix app,
• Suppose you're in the Kiwix app and you're browsing some videos from KA zim file.

The exercises don't work within the Kiwix appthough, but you can click on an perseus exercise and the Kiwix app will send you to the Kolibri app as follows

    Uri exercise = Uri.parse("kolibri:/learn/#/1ceff53605.../topics/c/845ce535...");
    Intent callIntent = new Intent(Intent.ACTION_VIEW, exercise);
    // do check here
    callIntent.startActivity()

We could something like this in the Kolibri android app manifest:

    <activity
        android:name=".activities.LinkDispatcherActivity"
        android:noHistory="true" // needed ???
        android:launchMode="singleInstance"
        android:theme="@android:style/Theme.NoDisplay" >
        
         <intent-filter>
             <action android:name="android.intent.action.VIEW"/>
             <category android:name="android.intent.category.DEFAULT"/>
             <category android:name="android.intent.category.BROWSABLE"/>
             <data android:scheme="kolibri"/>
             <data android:host="view_channel"/>
             <data android:host="view_node"/>
             <data android:pathPattern=".*"/>
         </intent-filter>
         
    </activity>

in the Kiwix android app

    <activity
        android:name=".activities.LinkDispatcherActivity"
        android:noHistory="true" // needed ???
        android:launchMode="singleInstance"
        android:theme="@android:style/Theme.NoDisplay" >
        
         <intent-filter>
             <action android:name="android.intent.action.VIEW"/>
             <category android:name="android.intent.category.DEFAULT"/>
             <category android:name="android.intent.category.BROWSABLE"/>
             <data android:scheme="kiwix"/>
             <data android:host="view_zimfile"/>
             <data android:host="view_page"/>
             <data android:pathPattern=".*"/>
         </intent-filter>
         
    </activity>

To verify there is an activity available that can respond to the intent, call queryIntentActivities() to get a list of activities capable of handling your Intent. If the returned List is not empty, you can safely use the intent. For example:

PackageManager packageManager = getPackageManager();

List = activities = packageManager.queryIntentActivities(intent, PackageManager.MATCH_DEFAULT_ONLY);
boolean isIntentSafe = activities.size() > 0;

If isIntentSafe is true, then we know that the intent-receiving app is installed and will respond to the intent.