Difference between revisions of "Esino Lario 2016"

From Kiwix
Jump to navigation Jump to search
(→‎People: Added myself to kiwix as well as the RACHEL project since I'm involved in both and care about improving both.)
Tag: Manual revert
 
(58 intermediate revisions by 7 users not shown)
Line 1: Line 1:
''DRAFT''
[[File:164-BGrigna giugno08 001.jpg|thumb|Esino Lario from Grigna]]
[[File:B2014 03 19 Esino 1937 Comune.jpg|thumb|Esino Lario Town Hall]]
 
This page summarizes the action plan for the '''Kiwix Wikimania 2016 hackathon'''. This hackathon has for main motto to improve the developement of Kiwix serve and ZIM content and bring Kiwix integrators together.
This page summarizes the action plan for the '''Kiwix Wikimania 2016 hackathon'''. This hackathon has for main motto to improve the developement of Kiwix serve and ZIM content and bring Kiwix integrators together.


Line 5: Line 7:


;MWoffliner
;MWoffliner
* Split MWoffliner is a command line script and a nodejs module
* Build MWoffliner scheduler (server/client)
* Split MWoffliner in a command line script and a nodejs module
* Modify MWofliner to apply "readibility" extension
* Modify MWofliner to apply "readibility" extension
* Add categories to MWoffliner


;OCG
;OCG
* Get OCG able to deal correctly with js/css resoures
* Get OCG able to deal correctly with js/css resources
* Reuse MWoffliner when necessary
* Get OCG able to generate ZIM files
* Get OCG able to generate ZIM files


;ZIMit
;ZIMit
 
* Release 1.0 ([https://github.com/almet/zimit/issues tasks])
** Add an user interface
** Add logging to the workers
** Test it and make it work on a selection of sites
** ...


; kiwix-serve
; kiwix-serve
Line 18: Line 29:
* Add ability to search in many ZIM files at the seame time
* Add ability to search in many ZIM files at the seame time
* Add UI translation
* Add UI translation
* Create kiwix-serve for Android, and perhaps integrate it into an Android app, or integrate it into the Kiwix Android app
'''webserver-zim-plugin'''


Currently, kiwix-serve runs as a separate, self-contained web server. If practical (I'm sure it's technically possible) I'd like a way to enable existing web servers to interact with zim file contents. [Julian Harty]
; Web server (nginx/Apache) ZIM module
* kiwix-serve runs as a separate, self-contained web server. Having a way to enable existing web servers to interact directly with ZIM @Julian
 
* Create a plugin that can be used by other web servers e.g. nginx, Apache, to interact with zim files.
* Create a plugin that can be used by other web servers e.g. nginx, Apache, to interact with zim files.
; New catalog
; New catalog
* Create OPDS backend
* Create OPDS backend
* Integrate OPDS in Kiwix for Android & iOS
* Integrate OPDS in Kiwix for Android & iOS
'''Mobile apps'''
 
; Android (& iOS)
* Allow the app to act as a web server (integrate kiwix-serve) @Julian
* Release new version of Kiwix for Android
* Release new version of Kiwix for Android
* Release new version of Kiwix for iOS
* Add in-app analytics [provided the team agrees, etc] to help the development and project team learn more about how the app is used so we can help improve the UX and the quality of the mobile apps. @Julian
* Add in-app analytics [provided the team agrees, etc] to help the development and project team learn more about how the app is used so we can help improve the UX and the quality of the mobile apps. [Julian Harty]
 
; Gutenberg
* Fix a [https://github.com/kiwix/gutenberg/issues?q=is%3Aopen+is%3Aissue+milestone%3A2.0 few issues] and release version 2.0]
 
; Sotoki
* Release version 1.0
 
; XSCE/Internet in a Box
* Participate in New Catalog Design
* Modify Admin Console to use New OPDS Catalog for ZIM Downloads
* Generate Navigation html snippets for downloaded ZIMs and integrate into menus
* Integrate ZIMit into XSCE Admin Console
* Modify impacted Help Files
* Adapt XSCE to new search strategy
* Integrate proxied kiwix server into apache in XSCE
* Test on three platforms: XO, RPi, NUC
* Stretch Goal: Generate Navigation snippets for downloaded ZIMs for use by WordPress
 
; Kiwix for Android
* Fix 3 bugs related to localisation @Emmanuel


== Achievements ==
== Achievements ==
; youtube2zim
* [https://github.com/kiwix/youtube/issues/5#issuecomment-226708678 Remove useless --dl-playlist] @Dattaz
* [https://github.com/kiwix/youtube/issues/6 if no playlist in channel, playlist select box should be invisible] @Dattaz
* [https://github.com/kiwix/youtube/issues/8 Take care of the "presentation" video] @Dattaz
* [https://github.com/kiwix/youtube/issues/9 Use  youtube-dl re-encoding feature (instead of custom avconv syscall] @Dattaz
* Relase of [https://github.com/kiwix/youtube/releases/tag/v1.0 version 1.0] @Dattaz
; zimwriterfs
* Clean-up of README and compilation scripts @Scott
* Fix many small bugs discovered by hackathon attendees @Emmanuel
* Split the whole code in multiple files @Matthieu
* Allow to write Xapian index directly in the ZIM file @Matthieu
* Speed up seriously the overall speed @Matthieu
; PhET
* Many big improvements to the PhET scrapper @Joe
* Move of the PhET scrapper source code to Github Kiwix project @Joe
* Publication and automatisation of monthly update of PhET ZIM files @Emmanuel
* Publication on Android store of the first PhET app @Emmanuel
; Kiwix Desktop
* Migration to new Xapian 1.4 (port for Windows/OSX still to do) @Emmanuel
* Add support for new glass single-file Xapian fulltext indexes @Emmanuel
; Zimbalaka
* Fix [http://zimbalaka.openzim.org our live instance of Zimbalaka] @Alexis
; Libzim
* Simplification of the ZIM creation process @Scott
* Upgrade libzim/refdoc to allow to store fulltext index (get fileoffset of a content) @Matthieu
* Release libzim 1.3 @Emmanuel
; Node-libzim
* [https://github.com/cscott/node-libzim Nodejs binding for zimlib] @Scott
* Release of [https://github.com/cscott/node-libzim/releases/tag/0.0.2 0.0.2] @Scott
; Catalog
* Common agreement between attendees of [[Catalog|what information should belong to the catalog]] @Tim @Adam @Matthieu @Alexis @Emmanuel
* Common agreement to use OPDS
* First version of OPDS export has been done for Ideascube @Alexis
* First reader of OPDS import @Tim
; Sotoki (Stack Exchange projects ZIM generator)
* Revamping of the source code for better performanaces @Dattaz
* [https://github.com/kiwix/sotoki/pull/32 Big PR that correct issue] #30 #27 #25 #23 #22 #21 #20 #19 #17 #9 @Dattaz
* Almost ready for 1.0 release @Dattaz
; Kiwix for iOS
* 1.5.1 testing and release @Chris & @Emmanuel
* 1.6 Testflight release with few search improvments and fixes @Chris
; Kiwix in HTML5
* Release of [https://github.com/kiwix/kiwix-html5/releases/tag/2.0-beta kiwix-html5 2.0 beta] (first version after the merge with Evopedia) @Mossroy
; TED
* Publish 6 new versions "2016" of the TED talks @Emmanuel
; ZimIT
* Lot of progresses on the source code (still need a little bit bug-fixes for a 1.0 release) @Alexis
* Setup a first test version online at http://zimit.openzim.org @Alexis & @Emmanuel
; Apache ZIM module
* First investigation work @Julian


== Impact ==
== Impact ==
* Find synergies between integrators to avoid duplicate works
* Increase collaboration and understanding between Kiwix team and integrators
* Improve kiwix-serve, Kiwix solution uses by all integrators
* Publish first version of "ZIMit", an agnostic Web2ZIM scrapper allowing to create easily a ZIM for from any web site
* Publish new version of the Gutenberg scrapper and publish new versions of the Gutenberg ZIM files (last version is from 2014)
* Publish first version of sotoki, a [https://stackexchange.com/sites# StackExchange network] (most famous IT forums in the world) and publish ZIM files from the most famous site: stackoverflow , stackexchange, askubuntu, serverfault, superuser.
* Bring back ZIM export feature to Wikimedia web site (ZIM support in OCG), so everybody can click and create a custom ZIM file directly from Wikimedia web sites and per extension from any Mediawiki instance
* Improve MWoffliner to avoid duplicate code with OCG
* Add OPDS support to Kiwix infrastructure to allow our solution to scale and deal with >10.000 ZIM files
* Fix a few bugs with the TED scrapper and generate a new set of TED videos
* Extend new features of Kiwix into XSCE/Internet in a Box


== Method ==
== Method ==


5 developers meet during a week to make this steps forward.
~ 12 developers meet during a week to make this steps forward.


== Place ==
== Place ==


[https://meta.wikimedia.org/wiki/Wikimania_2016_bids/Esino_Lario/Meeting_rooms/Primary_school Esino Lario Primary school]
The hackathon will take place in [https://en.wikipedia.org/wiki/Esino_Lario Esino Lario], Italy:
* [http://www.hotelcentra.it Hotel Centrale] for the meals and nights
* [https://meta.wikimedia.org/wiki/Wikimania_2016_bids/Esino_Lario/Meeting_rooms/Primary_school Primary school] (where Wikimania takes place) for the work time
 
:... ''[https://wikimania2016.wikimedia.org/wiki/Journey instructions to go there]'''


== Agenda ==
== Agenda ==


* June 16th: arrivals
* June 16th: arrivals (suggested arrival time 5PM)
* June 17th -> 21th: Kiwix hackathon
* June 17th -> 21th: Kiwix hackathon
* June 22th -> 23th: Wikimania pre-conference hackathon (don't forget to [https://wikimania2016.wikimedia.org register])
* June 22th -> 23th: Wikimania pre-conference hackathon (don't forget to [https://wikimania2016.wikimedia.org register])
Line 53: Line 164:
; Kiwix
; Kiwix
* Emmanuel
* Emmanuel
* Renaud
* Rashiq (remote)
* Rashiq
* Chris (remote)
* Chris
* Dattaz
* Julian Harty* (with Kiwix hat)


; Mediawiki OCG
; Mediawiki OCG
* Scott
* Scott


; KA Lite
; Internet-in-a-Box & Schoolserver.org
* Benjamin
* Adam Holt
 
* Tim Moody
; Schoolserver
* Adam


; Kwiizi
; Kwiizi
* Yannick
* Yannick


; Ideascube
; Ideascube && Biliothèques Sans Frontières (BSF)
* Alexis
* [https://github.com/almet Alexis]
* Yohann
* [https://github.com/mgautierfr Matthieu]
* Florian


; Rachel
; Rachel
* Julian Harty* (with RACHEL hat)
* Julian Harty* (with RACHEL hat)
* Joe


== Costs ==
== Costs ==
~10.000 CHF/USD
== See also ==
* [https://wikimania2016.wikimedia.org/wiki/Programme/Kiwix-dedicated_Hackathon Wikimania hackathon page]
[[Category:Hackathon]]

Latest revision as of 15:31, 27 February 2023

Esino Lario from Grigna
Esino Lario Town Hall

This page summarizes the action plan for the Kiwix Wikimania 2016 hackathon. This hackathon has for main motto to improve the developement of Kiwix serve and ZIM content and bring Kiwix integrators together.

Goals

MWoffliner
  • Build MWoffliner scheduler (server/client)
  • Split MWoffliner in a command line script and a nodejs module
  • Modify MWofliner to apply "readibility" extension
  • Add categories to MWoffliner
OCG
  • Get OCG able to deal correctly with js/css resources
  • Reuse MWoffliner when necessary
  • Get OCG able to generate ZIM files
ZIMit —
  • Release 1.0 (tasks)
    • Add an user interface
    • Add logging to the workers
    • Test it and make it work on a selection of sites
    • ...
kiwix-serve
  • Add opensearch backend
  • Add ability to search in many ZIM files at the seame time
  • Add UI translation
Web server (nginx/Apache) ZIM module
  • kiwix-serve runs as a separate, self-contained web server. Having a way to enable existing web servers to interact directly with ZIM @Julian
  • Create a plugin that can be used by other web servers e.g. nginx, Apache, to interact with zim files.
New catalog
  • Create OPDS backend
  • Integrate OPDS in Kiwix for Android & iOS
Android (& iOS)
  • Allow the app to act as a web server (integrate kiwix-serve) @Julian
  • Release new version of Kiwix for Android
  • Add in-app analytics [provided the team agrees, etc] to help the development and project team learn more about how the app is used so we can help improve the UX and the quality of the mobile apps. @Julian
Gutenberg
Sotoki
  • Release version 1.0
XSCE/Internet in a Box
  • Participate in New Catalog Design
  • Modify Admin Console to use New OPDS Catalog for ZIM Downloads
  • Generate Navigation html snippets for downloaded ZIMs and integrate into menus
  • Integrate ZIMit into XSCE Admin Console
  • Modify impacted Help Files
  • Adapt XSCE to new search strategy
  • Integrate proxied kiwix server into apache in XSCE
  • Test on three platforms: XO, RPi, NUC
  • Stretch Goal: Generate Navigation snippets for downloaded ZIMs for use by WordPress
Kiwix for Android
  • Fix 3 bugs related to localisation @Emmanuel

Achievements

youtube2zim
zimwriterfs
  • Clean-up of README and compilation scripts @Scott
  • Fix many small bugs discovered by hackathon attendees @Emmanuel
  • Split the whole code in multiple files @Matthieu
  • Allow to write Xapian index directly in the ZIM file @Matthieu
  • Speed up seriously the overall speed @Matthieu
PhET
  • Many big improvements to the PhET scrapper @Joe
  • Move of the PhET scrapper source code to Github Kiwix project @Joe
  • Publication and automatisation of monthly update of PhET ZIM files @Emmanuel
  • Publication on Android store of the first PhET app @Emmanuel
Kiwix Desktop
  • Migration to new Xapian 1.4 (port for Windows/OSX still to do) @Emmanuel
  • Add support for new glass single-file Xapian fulltext indexes @Emmanuel
Zimbalaka
Libzim
  • Simplification of the ZIM creation process @Scott
  • Upgrade libzim/refdoc to allow to store fulltext index (get fileoffset of a content) @Matthieu
  • Release libzim 1.3 @Emmanuel
Node-libzim
Catalog
  • Common agreement between attendees of what information should belong to the catalog @Tim @Adam @Matthieu @Alexis @Emmanuel
  • Common agreement to use OPDS
  • First version of OPDS export has been done for Ideascube @Alexis
  • First reader of OPDS import @Tim
Sotoki (Stack Exchange projects ZIM generator)
  • Revamping of the source code for better performanaces @Dattaz
  • Big PR that correct issue #30 #27 #25 #23 #22 #21 #20 #19 #17 #9 @Dattaz
  • Almost ready for 1.0 release @Dattaz
Kiwix for iOS
  • 1.5.1 testing and release @Chris & @Emmanuel
  • 1.6 Testflight release with few search improvments and fixes @Chris
Kiwix in HTML5
TED
  • Publish 6 new versions "2016" of the TED talks @Emmanuel
ZimIT
  • Lot of progresses on the source code (still need a little bit bug-fixes for a 1.0 release) @Alexis
  • Setup a first test version online at http://zimit.openzim.org @Alexis & @Emmanuel
Apache ZIM module
  • First investigation work @Julian

Impact

  • Find synergies between integrators to avoid duplicate works
  • Increase collaboration and understanding between Kiwix team and integrators
  • Improve kiwix-serve, Kiwix solution uses by all integrators
  • Publish first version of "ZIMit", an agnostic Web2ZIM scrapper allowing to create easily a ZIM for from any web site
  • Publish new version of the Gutenberg scrapper and publish new versions of the Gutenberg ZIM files (last version is from 2014)
  • Publish first version of sotoki, a StackExchange network (most famous IT forums in the world) and publish ZIM files from the most famous site: stackoverflow , stackexchange, askubuntu, serverfault, superuser.
  • Bring back ZIM export feature to Wikimedia web site (ZIM support in OCG), so everybody can click and create a custom ZIM file directly from Wikimedia web sites and per extension from any Mediawiki instance
  • Improve MWoffliner to avoid duplicate code with OCG
  • Add OPDS support to Kiwix infrastructure to allow our solution to scale and deal with >10.000 ZIM files
  • Fix a few bugs with the TED scrapper and generate a new set of TED videos
  • Extend new features of Kiwix into XSCE/Internet in a Box

Method

~ 12 developers meet during a week to make this steps forward.

Place

The hackathon will take place in Esino Lario, Italy:

... instructions to go there'

Agenda

  • June 16th: arrivals (suggested arrival time 5PM)
  • June 17th -> 21th: Kiwix hackathon
  • June 22th -> 23th: Wikimania pre-conference hackathon (don't forget to register)

People

Kiwix
  • Emmanuel
  • Rashiq (remote)
  • Chris (remote)
  • Dattaz
Mediawiki OCG
  • Scott
Internet-in-a-Box & Schoolserver.org
  • Adam Holt
  • Tim Moody
Kwiizi
  • Yannick
Ideascube && Biliothèques Sans Frontières (BSF)
Rachel
  • Julian Harty* (with RACHEL hat)
  • Joe

Costs

~10.000 CHF/USD

See also