Automatic testing meeting on 2017-04-05

Automatic testing

On kiwix-html5 repo : travis + saucelabs

saucelabs provides pre-built hardware and software with browser/mobile versions etc.

bitbar android devices test remotely + iOS

Guillaume : test installer on Windows/Mac/Linux Travis only does Linux

Dattaz : curiosity

Matthieu : we should have tests on every project. Lack of tests on Kiwix.

Isaac : fragmentation => need to test on different devices

Emmanuel : non-regression is important. Scale, complexity increases

Julian : Differences between what Matthieu and Isaac do automate the most common user journeys : open the app, open a page, etc BDD frameworks : given, when, then (natural language) Opinion : little point to have common code between all the repos. Java unit tests works well, but do not allow to interact with the system (permissions, SD card etc) UIOtomato Image-based testing : not recommended for us

Which content to use for the tests? Ray Charles archive allows to test some aspects, but not all of them. Careful about the multi-lingual aspect.

Step by step : start with something simple to test, and see what needs to be improved. Android pains : different storage locations, onboarding

Need to record the times taken by each test (with a fine granularity) => reports

Levels at Google : 0- no tests 1- at least one test, and CI that runs it 2- policy : team that agree that, at every change, a procedure must be 3- coverage : minimum 40%

The code coverage is not a primary measure, but can indicate a trend on code quality attention by developers But too much unit tests can be complicated to maintain. Feature coverage can be another measure.

What we're not going to do in the short term for Android testing : - language testing : only English at first - multi-device testing - BDD testing - search (case-sensitivity, etc) : it must be tested by unit tests on the ZIM lib - network connectivity : slow networks can be simulated, but it becomes complicated. We assume the network is working correctly - test permissions of the system - test available storage - test custom apps

For the short term : use preferably Espresso and/or UIOtomato for Android testing.

Emmanuel : the goal is not have a lot of tests, but to have the platform and start.

Maximum time for automated testing : one hour

Choice for Android testing (by the end of the week) : - work on Travis automation (continuous building) - or work on local testing automation, that could be run manually by all Android developers. This one is chosen.

For saucelabs : create a "kiwix" account and use this credential.

No tests on kiwix desktop for now. Maybe for kiwix-serve?