Difference between revisions of "Lyon 2017"

Jump to navigation Jump to search
Tag: Reverted
(30 intermediate revisions by 10 users not shown)
Line 1: Line 1:
This page summarizes the action plan for the '''Kiwix Spring Hackathon 2017'''.
This page summarizes the action plan for the '''Kiwix Spring Hackathon 2017'''.


== Additional Hackathons ==
A larger [[Hackathon Wikimania 2017|http://OFF.NETWORK Hackathon]] will follow, in New York State right near Montreal (August 13-18, 2017) immediately after [http://wikimania2017.wikimedia.org/ Wikimania 2017].
[[UkHackathon2017]] A mini hackathon is planned for 14th and 15th December 2017 in England that will focus on improving the quality of the Kiwix Android app.
== Goals ==
== Goals ==


Line 12: Line 16:
* Consider possible ways to enable users to provide device-info such as storage locations to help us improve support for their devices.
* Consider possible ways to enable users to provide device-info such as storage locations to help us improve support for their devices.


; Scrappers (Python)
; Scrapers (Python)
* Fix Gutenberg scrapper and generate new ZIM files
* Fix Gutenberg scraper and generate new ZIM files
* Fix KALite ZIM export (for a first release)
* Fix KALite ZIM export (for a first release)
* Create a fully automatized solution for Stackexchange projects ZIM files
* Create a fully automatized solution for Stackexchange projects ZIM files


; Scrapper Mediawiki (Nodejs)
; Scraper Mediawiki (Nodejs)
* Implement Wikihow scrapper (non-parsoid Mediawiki scraper)
* Implement Wikihow scraper (non-parsoid Mediawiki scraper)
* Scrape categories
* Scrape categories
* Support videos (if Parsoid allows it)
* Support videos (if Parsoid allows it)
Line 71: Line 75:


=== Browser extensions ===
=== Browser extensions ===
A basic browser extension has been commited.
Basic browser extensions have been submitted to Mozilla and Google, to make them available on Firefox and Chrome. It's the same code on both of them.


Before it is available from Google and Mozilla extensions system, it can be installed manually :
Before they are available from their native extensions system, they can be installed manually :
* For Firefox :
* For Firefox :
** download the .xpi file from https://framadrop.org/r/Sq5oZm0Uv0#pcLwtwvzsSuFzCeSFsqNDgXf8tzo4riA56teTIX7YVE=
** Open http://download.kiwix.org/nightly , choose the latest date available, and click on the latest kiwix-firefox-signed-extension-*.xpi
** open a new tab with URL "about:debugging"
** Confirm the installation
** click on "load a temporary module" and choose the xpi file
** It's expected that only the jQuery mode will work under Firefox extension (Firefox limitation for now)
** It's expected that only the jQuery mode will work under Firefox extension (Firefox limitation for now)
** This jQuery mode works well on wikipedia ZIM files, but has not been tested on other kind of ZIM files, where it should have big limitations
** This jQuery mode works well on wikipedia ZIM files, but has not been tested on other kind of ZIM files, where it should have big limitations
* For Chromium/Chrome :
* For Chromium/Chrome :
** download the .crx file from https://framadrop.org/r/HOhudv1lr8#+ITMyqfwcCcLzMz3UG0Kw+YfOeksy2zl1J28wTDV1Ak=
** Open http://download.kiwix.org/nightly , choose the latest date available, and download the latest kiwix-chrome-signed-extension-*.crx file
** open a new tab with URL "chrome://extensions/"
** open a new Chromium/Chrome tab with URL "chrome://extensions/"
** drag and drop the crxf file into this tab
** drag and drop the crx file into this tab
* On both browsers, a Kiwix icon should appear on the right of the address bar
* On both browsers, a Kiwix icon appears on the right of the address bar
* If you click on this icon, it should open a new tab with the Kiwix application. You must then choose your ZIM file(s), and you should be able to browse it
* If you click on this icon, it opens a new tab with the Kiwix application. You must then choose your ZIM file(s), and you should be able to browse it


=== Gutenberg scraper ===
=== Gutenberg scraper ===
* Updated dependencies and made it python3 compatible
* Updated dependencies and made it python3 compatible
* Parallelized long-running operations: download, parsing and export
* Parallelized long-running operations: download, parsing and export
* Fixed 3 bugs related to the scraping process
* Fixed all bugs and enhancements from github
* Add docker image and [https://hub.docker.com/r/openzim/gutenberg/ CI integration in Docker-hub]
* Publication of [https://pypi.python.org/pypi/gutenberg2zim Python package]


=== Sotoki ===
=== Sotoki ===
Line 98: Line 103:
* Add downloader for dump
* Add downloader for dump
* Add intern link between questions, answers and users
* Add intern link between questions, answers and users
* Add docker image and [https://hub.docker.com/r/openzim/sotoki/ CI integration in Docker-hub]
* Add Dockerfile and [https://hub.docker.com/r/openzim/sotoki/ CI integration in Docker-hub]
* Add option to specify zim path


=== youtube scrapper ===
=== youtube scraper ===
* update code and package to python package : https://pypi.python.org/pypi?:action=display&name=youtube2zim
* update code and package to python package : https://pypi.python.org/pypi?:action=display&name=youtube2zim
* Add docker file and [https://hub.docker.com/r/openzim/youtube/ CI integration in Docker-hub]
* Add Dockerfile and [https://hub.docker.com/r/openzim/youtube/ CI integration in Docker-hub]
* add option to specify zim path
 
=== openedx/FUN scraper ===
* brainstorming about how to do it
* making a first basically (no special case, only for edx.org, some things aren't put offline like fonts) working prototype


=== Android ===
=== Android ===
* Restructured project to allow for easier testing and readability.
* Restructured project to allow for easier testing and readability
* Various bug fixes
* Various bug fixes
* Introduced new developers to tooling and processes
* Introduced new developers to tooling and processes
* Travis CI
* Travis CI
* Custom apps generation moved to gradle
* Very basic UI instrumentation tests added
* New version bookmark/Reading List feature
* Full automated Custom app generation


=== zimwriterfs ===
=== zimwriterfs ===
Line 119: Line 134:
** enable ssh server via QEMU process input/output
** enable ssh server via QEMU process input/output
* Setting up environment for [https://github.com/ideascube/ansiblecube/tree/oneUpdateFile ansiblecube] (as buildMyCube.sh does)
* Setting up environment for [https://github.com/ideascube/ansiblecube/tree/oneUpdateFile ansiblecube] (as buildMyCube.sh does)
* Script configuring and compiling linux for QEMU armhf emulation (does not have options for ansiblecube captive portal yet)
* Script configuring and compiling linux for QEMU armhf emulation
* Deployment of ideascube with ansiblecube in QEMU


=== zip2zim ===
=== zip2zim ===
Line 128: Line 144:
* Category Filtering
* Category Filtering
* UI/UX improvements
* UI/UX improvements
=== wikihow scraper ===
* Stub
=== Apache module ===
* First version ready for Alpha testing on Debian-based servers see https://github.com/kiwix/kiwix-apache
=== Kalite ===
* First version working
=== Search engine ===
* Last bug fixes ft search
* Integration ft search zimlib
* More Travis integration (kiwix-lib, kiwix-tools)
== Code organization ==
* Setup the whole git/docker "openZIM" organization
== MWoffliner ==
* Add Dockerfile and [https://hub.docker.com/r/openzim/mwoffliner/ CI integration in Docker-hub]
* Add support local Parsoid


== Impact ==
== Impact ==
Line 174: Line 211:
* F&B: ~3000 CHF
* F&B: ~3000 CHF
-> ~9.000 CHF
-> ~9.000 CHF
[[Category:hackathon|L]]

Navigation menu