Difference between revisions of "Lyon 2017"

From Kiwix
Jump to navigation Jump to search
(→‎UX & Onboarding: Adding link so I can create a separate page for this topic.)
 
(52 intermediate revisions by 12 users not shown)
Line 1: Line 1:
This page summarizes the action plan for the '''Kiwix Spring Hackathon 2017'''.
This page summarizes the action plan for the '''Kiwix Spring Hackathon 2017'''.


== Additional Hackathons ==
A larger [[Hackathon Wikimania 2017|http://OFF.NETWORK Hackathon]] will follow, in New York State right near Montreal (August 13-18, 2017) immediately after [http://wikimania2017.wikimedia.org/ Wikimania 2017].
[[UkHackathon2017]] A mini hackathon is planned for 14th and 15th December 2017 in England that will focus on improving the quality of the Kiwix Android app.
== Goals ==
== Goals ==


Line 12: Line 16:
* Consider possible ways to enable users to provide device-info such as storage locations to help us improve support for their devices.
* Consider possible ways to enable users to provide device-info such as storage locations to help us improve support for their devices.


; Scrappers (Python)
; Scrapers (Python)
* Fix Gutenberg scrapper and generate new ZIM files
* Fix Gutenberg scraper and generate new ZIM files
* Fix KALite ZIM export (for a first release)
* Fix KALite ZIM export (for a first release)
* Create a fully automatized solution for Stackexchange projects ZIM files
* Create a fully automatized solution for Stackexchange projects ZIM files


; Scrapper Mediawiki (Nodejs)
; Scraper Mediawiki (Nodejs)
* Implement Wikihow scrapper (non-parsoid Mediawiki scraper)
* Implement Wikihow scraper (non-parsoid Mediawiki scraper)
* Scrape categories
* Scrape categories
* Support videos (if Parsoid allows it)
* Support videos (if Parsoid allows it)
Line 24: Line 28:
; Phet
; Phet
* Release a new version of Phet Simulation with an improved UX
* Release a new version of Phet Simulation with an improved UX
* Add category filtering


; HTML5 (Javascript)
; HTML5 (Javascript)
Line 39: Line 44:
==== Apache ====
==== Apache ====
* Create a zim module for Apache (based on [http://stackoverflow.com/questions/42605138/how-can-i-write-an-apache-module-in-c http://stackoverflow.com/questions/42605138/how-can-i-write-an-apache-module-in-c) Julian]
* Create a zim module for Apache (based on [http://stackoverflow.com/questions/42605138/how-can-i-write-an-apache-module-in-c http://stackoverflow.com/questions/42605138/how-can-i-write-an-apache-module-in-c) Julian]
=== zip2zim (zimwriterfs as a service) ===
Create an online tool for converting zip files of HTML into deployable zim files


== Group Discussion Topics ==
== Group Discussion Topics ==
Line 51: Line 59:
When: 14:00 ish Tuesday 4th April 2017
When: 14:00 ish Tuesday 4th April 2017
==== Intro ====
==== Intro ====
We believe large numbers of users only use the app briefly before abandoning it. There are various ideas why this might be so. Perhaps they don't understand what the app can do for them, or they may find it hard to get started. There may be problems downloading content, etc. Also, with our growth in custom apps, including non-Wikipedia content such as PhET simulations, perhaps the user interface, menu options, and some of the functionality is a poor fit for the content and capabilities. We'd like to discuss the [[UX and Onboarding|UX]], including onboarding (getting users from first starting the app to using it productively), and consider ways we might usefully improve the UX.
We believe large numbers of users only use the app briefly before abandoning it. There are various ideas why this might be so. Perhaps they don't understand what the app can do for them, or they may find it hard to get started. There may be problems downloading content, etc. Also, with our growth in custom apps, including non-Wikipedia content such as PhET simulations, perhaps the user interface, menu options, and some of the functionality is a poor fit for the content and capabilities. We'd like to discuss the UX, including onboarding (getting users from first starting the app to using it productively), and consider ways we might usefully improve the UX.
 
Sum up of the meeting : [[UX and Onboarding]]


=== Automatic testing ===
=== Automatic testing ===
Line 59: Line 69:


We should discuss our experience/ideas on that topic, and see if we could do something in common.
We should discuss our experience/ideas on that topic, and see if we could do something in common.
Sum up of the meeting : [[Automatic testing meeting on 2017-04-05]]


== Achievements ==
== Achievements ==


=== Browser extensions ===
=== Browser extensions ===
A basic browser extension has been commited.
Basic browser extensions have been submitted to Mozilla and Google, to make them available on Firefox and Chrome. It's the same code on both of them.


Before it is available from Google and Mozilla extensions system, it can be installed manually :
Before they are available from their native extensions system, they can be installed manually :
* Checkout the kiwix-html5 github repo (master branch)
* For Firefox :
* For Firefox :
** open a new tab with URL "about:debugging"
** Open http://download.kiwix.org/nightly , choose the latest date available, and click on the latest kiwix-firefox-signed-extension-*.xpi
** click on "load a temporary module" and choose the "manifest.json" file from your local copy of the code
** Confirm the installation
** It's expected that only the jQuery mode will work under Firefox extension (Firefox limitation for now)
** This jQuery mode works well on wikipedia ZIM files, but has not been tested on other kind of ZIM files, where it should have big limitations
* For Chromium/Chrome :
* For Chromium/Chrome :
** open a new tab with URL "chrome://extensions/"
** Open http://download.kiwix.org/nightly , choose the latest date available, and download the latest kiwix-chrome-signed-extension-*.crx file
** click on "developer mode" then on "load unpackaged extension" and choose the directory of your local copy of the code (where the "manifest.json" fil is)
** open a new Chromium/Chrome tab with URL "chrome://extensions/"
* On both browsers, a Kiwix icon should appear on the right of the address bar
** drag and drop the crx file into this tab
* If you click on this icon, it should open a new tab with the Kiwix application. You must then choose your ZIM file(s), and you should be able to browse it
* On both browsers, a Kiwix icon appears on the right of the address bar
* If you click on this icon, it opens a new tab with the Kiwix application. You must then choose your ZIM file(s), and you should be able to browse it


=== Gutenberg scraper ===
=== Gutenberg scraper ===
* Updated dependencies and made it python3 compatible
* Updated dependencies and made it python3 compatible
* Parallelized long-running operations: download, parsing and export
* Parallelized long-running operations: download, parsing and export
* Fixed 3 bugs related to the scraping process
* Fixed all bugs and enhancements from github
* Add docker image and [https://hub.docker.com/r/openzim/gutenberg/ CI integration in Docker-hub]
* Publication of [https://pypi.python.org/pypi/gutenberg2zim Python package]
 
=== Sotoki ===
* Release first version
* Correct last small mobile bug
* Package to python package : https://pypi.python.org/pypi?:action=display&name=sotoki
* Add downloader for dump
* Add intern link between questions, answers and users
* Add Dockerfile and [https://hub.docker.com/r/openzim/sotoki/ CI integration in Docker-hub]
* Add option to specify zim path
 
=== youtube scraper ===
* update code and package to python package : https://pypi.python.org/pypi?:action=display&name=youtube2zim
* Add Dockerfile and [https://hub.docker.com/r/openzim/youtube/ CI integration in Docker-hub]
* add option to specify zim path
 
=== openedx/FUN scraper ===
* brainstorming about how to do it
* making a first basically (no special case, only for edx.org, some things aren't put offline like fonts) working prototype
 
=== Android ===
* Restructured project to allow for easier testing and readability
* Various bug fixes
* Introduced new developers to tooling and processes
* Travis CI
* Custom apps generation moved to gradle
* Very basic UI instrumentation tests added
* New version bookmark/Reading List feature
* Full automated Custom app generation
 
=== zimwriterfs ===
* Add Docker file and [https://hub.docker.com/r/openzim/zimwriterfs/ CI integration in Docker-hub]
* Fix small bug in usage()
 
=== Ideascube builder ===
* Communication between python script and QEMU process
** establish ssh connection
** enable ssh server via QEMU process input/output
* Setting up environment for [https://github.com/ideascube/ansiblecube/tree/oneUpdateFile ansiblecube] (as buildMyCube.sh does)
* Script configuring and compiling linux for QEMU armhf emulation
* Deployment of ideascube with ansiblecube in QEMU


=== sotoki ===
=== zip2zim ===
* correct last small mobile bug
* Created working prototype
* package to python package : https://pypi.python.org/pypi?:action=display&name=sotoki
* Add docker image and [https://hub.docker.com/r/openzim/zip2zim/ CI integration in Docker-hub]
 
=== PhET ===
* Category Filtering
* UI/UX improvements
 
=== wikihow scraper ===
* Stub
 
=== Apache module ===
* First version ready for Alpha testing on Debian-based servers see https://github.com/kiwix/kiwix-apache
 
=== Kalite ===
* First version working
 
=== Search engine ===
* Last bug fixes ft search
* Integration ft search zimlib
* More Travis integration (kiwix-lib, kiwix-tools)
 
== Code organization ==
* Setup the whole git/docker "openZIM" organization
 
== MWoffliner ==
* Add Dockerfile and [https://hub.docker.com/r/openzim/mwoffliner/ CI integration in Docker-hub]
* Add support local Parsoid


== Impact ==
== Impact ==
Line 126: Line 207:
== Costs ==
== Costs ==


* Accommodation: ~4.000 CHF
* Accommodation: 4'341.40 CHF
* Travels: ~2.000 CHF
* Travels: 3'051.83 CHF
* F&B: ~3000 CHF
* F&B: 2'127.82 CHF
-> ~9.000 CHF
* Goodies: 150 CHF
;Total: 9'671.05 CHF
 
[[Category:hackathon]]

Latest revision as of 13:19, 28 February 2023

This page summarizes the action plan for the Kiwix Spring Hackathon 2017.

Additional Hackathons

A larger http://OFF.NETWORK Hackathon will follow, in New York State right near Montreal (August 13-18, 2017) immediately after Wikimania 2017.

UkHackathon2017 A mini hackathon is planned for 14th and 15th December 2017 in England that will focus on improving the quality of the Kiwix Android app.

Goals

Android
  • Setup the Android automated testing up and make a proof of concept
  • Release a new version of Kiwix for Android (maybe a first version with multizim support?)
  • Implement Wikimed Feedback solution
  • Implement a solution to allow non-tech to launch/update Android custom apps
  • More sophisticated bookmark
  • Work on 2.3 release
  • Consider possible ways to enable users to provide device-info such as storage locations to help us improve support for their devices.
Scrapers (Python)
  • Fix Gutenberg scraper and generate new ZIM files
  • Fix KALite ZIM export (for a first release)
  • Create a fully automatized solution for Stackexchange projects ZIM files
Scraper Mediawiki (Nodejs)
  • Implement Wikihow scraper (non-parsoid Mediawiki scraper)
  • Scrape categories
  • Support videos (if Parsoid allows it)
Phet
  • Release a new version of Phet Simulation with an improved UX
  • Add category filtering
HTML5 (Javascript)
  • Launch the first pure HTML ZIM read browser extensions (FF+Chrome)
Search
  • Fix fulltext search/indexes problems
ZIM building
  • Setup a ZIM building scheduler (with Celery?)
Ideascube builder
  • Get a Qemu Raspbian image ready in Travis

Apache

zip2zim (zimwriterfs as a service)

Create an online tool for converting zip files of HTML into deployable zim files

Group Discussion Topics

During the hackathon we'd like to discuss various topics collectively. These include:

  • UX and particularly Onboarding
  • Desktop apps
  • Incorporating analytics

We aim to agree a topic with about a day's notice so those who wish can do some preparation, e.g. reading up, sharing ideas and notes, etc.

UX & Onboarding

When: 14:00 ish Tuesday 4th April 2017

Intro

We believe large numbers of users only use the app briefly before abandoning it. There are various ideas why this might be so. Perhaps they don't understand what the app can do for them, or they may find it hard to get started. There may be problems downloading content, etc. Also, with our growth in custom apps, including non-Wikipedia content such as PhET simulations, perhaps the user interface, menu options, and some of the functionality is a poor fit for the content and capabilities. We'd like to discuss the UX, including onboarding (getting users from first starting the app to using it productively), and consider ways we might usefully improve the UX.

Sum up of the meeting : UX and Onboarding

Automatic testing

When: Wednesday 5th April 2017

Intro

Some repos already have automatic testing (unit tests and/or UI tests). Some other ones plan to add it.

We should discuss our experience/ideas on that topic, and see if we could do something in common.

Sum up of the meeting : Automatic testing meeting on 2017-04-05

Achievements

Browser extensions

Basic browser extensions have been submitted to Mozilla and Google, to make them available on Firefox and Chrome. It's the same code on both of them.

Before they are available from their native extensions system, they can be installed manually :

  • For Firefox :
    • Open http://download.kiwix.org/nightly , choose the latest date available, and click on the latest kiwix-firefox-signed-extension-*.xpi
    • Confirm the installation
    • It's expected that only the jQuery mode will work under Firefox extension (Firefox limitation for now)
    • This jQuery mode works well on wikipedia ZIM files, but has not been tested on other kind of ZIM files, where it should have big limitations
  • For Chromium/Chrome :
    • Open http://download.kiwix.org/nightly , choose the latest date available, and download the latest kiwix-chrome-signed-extension-*.crx file
    • open a new Chromium/Chrome tab with URL "chrome://extensions/"
    • drag and drop the crx file into this tab
  • On both browsers, a Kiwix icon appears on the right of the address bar
  • If you click on this icon, it opens a new tab with the Kiwix application. You must then choose your ZIM file(s), and you should be able to browse it

Gutenberg scraper

  • Updated dependencies and made it python3 compatible
  • Parallelized long-running operations: download, parsing and export
  • Fixed all bugs and enhancements from github
  • Add docker image and CI integration in Docker-hub
  • Publication of Python package

Sotoki

youtube scraper

openedx/FUN scraper

  • brainstorming about how to do it
  • making a first basically (no special case, only for edx.org, some things aren't put offline like fonts) working prototype

Android

  • Restructured project to allow for easier testing and readability
  • Various bug fixes
  • Introduced new developers to tooling and processes
  • Travis CI
  • Custom apps generation moved to gradle
  • Very basic UI instrumentation tests added
  • New version bookmark/Reading List feature
  • Full automated Custom app generation

zimwriterfs

Ideascube builder

  • Communication between python script and QEMU process
    • establish ssh connection
    • enable ssh server via QEMU process input/output
  • Setting up environment for ansiblecube (as buildMyCube.sh does)
  • Script configuring and compiling linux for QEMU armhf emulation
  • Deployment of ideascube with ansiblecube in QEMU

zip2zim

PhET

  • Category Filtering
  • UI/UX improvements

wikihow scraper

  • Stub

Apache module

Kalite

  • First version working

Search engine

  • Last bug fixes ft search
  • Integration ft search zimlib
  • More Travis integration (kiwix-lib, kiwix-tools)

Code organization

  • Setup the whole git/docker "openZIM" organization

MWoffliner

Impact

This hackathon gathers different kind of tasks which all in one will help to implement the Wikimed Projet and reach our goals in term of new content for 2017.

Method

~ 10 developers meet during a week to make this steps forward.

Place

The hackathon will take place in Lyon, France

Agenda

  • April 2th afternoon: arrivals (suggested arrival time 5PM)
  • April 3th -> 9th: Kiwix hackathon
  • April 9th afternoon: departures (suggested departure time 3PM)

People

Travel + Accommodation
  • Adrian
  • Elad
  • Emmanuel
  • Joe
  • Julian
  • Rashiq
  • Renaud
Sponsored
  • Hayley
Only Travel
  • Isaac
Located in Lyon
  • Matthieu
  • Mossroy
  • Guillaume

Costs

  • Accommodation: 4'341.40 CHF
  • Travels: 3'051.83 CHF
  • F&B: 2'127.82 CHF
  • Goodies: 150 CHF
Total
9'671.05 CHF
Retrieved from "https://wiki.kiwix.org/w/index.php?title=Lyon_2017&oldid=39216"