Tell us your story
Tell us your story
How has offline Wikipedia affected you? The Wikimedia Foundation (the non-profit that supports Wikipedia) is looking for personal, diverse and inspiring stories about how offline Wikipedia affects the world. If you have a personal story that you would like to share, please contact: stories@kiwix.org. Thank you!

Difference between revisions of "Automatic testing meeting on 2017-04-05"

From Kiwix
Jump to: navigation, search
(raw paste of my notes)
 
(Minor changes, make the page more readable)
 
Line 1: Line 1:
Automatic testing
+
''These are my raw notes of the meeting :''
  
On kiwix-html5 repo : travis + saucelabs
+
Mossroy : Currently on kiwix-html5 repo : travis + saucelabs
  
saucelabs provides pre-built hardware and software with browser/mobile versions etc.
+
Julian : saucelabs provides pre-built hardware and software with browser/mobile versions etc. bitbar : android devices test remotely + iOS
  
bitbar android devices test remotely + iOS
+
Guillaume : would like to test the installer on Windows/Mac/Linux. Travis only does Linux
 
 
Guillaume : test installer on Windows/Mac/Linux
 
Travis only does Linux
 
  
 
Dattaz : curiosity
 
Dattaz : curiosity
Line 19: Line 16:
  
 
Julian :  
 
Julian :  
 +
 
Differences between what Matthieu and Isaac do
 
Differences between what Matthieu and Isaac do
 +
 
automate the most common user journeys : open the app, open a page, etc
 
automate the most common user journeys : open the app, open a page, etc
 +
 
BDD frameworks : given, when, then (natural language)
 
BDD frameworks : given, when, then (natural language)
 +
 
Opinion : little point to have common code between all the repos.
 
Opinion : little point to have common code between all the repos.
 +
 
Java unit tests works well, but do not allow to interact with the system (permissions, SD card etc)
 
Java unit tests works well, but do not allow to interact with the system (permissions, SD card etc)
 +
 
UIOtomato
 
UIOtomato
 +
 
Image-based testing : not recommended for us
 
Image-based testing : not recommended for us
  
 
Which content to use for the tests? Ray Charles archive allows to test some aspects, but not all of them.
 
Which content to use for the tests? Ray Charles archive allows to test some aspects, but not all of them.
 +
 
Careful about the multi-lingual aspect.
 
Careful about the multi-lingual aspect.
  
 
Step by step : start with something simple to test, and see what needs to be improved.
 
Step by step : start with something simple to test, and see what needs to be improved.
 +
 
Android pains : different storage locations, onboarding
 
Android pains : different storage locations, onboarding
  
Line 36: Line 42:
  
 
Levels at Google :
 
Levels at Google :
0- no tests
+
* 0- no tests
1- at least one test, and CI that runs it
+
* 1- at least one test, and CI that runs it
2- policy : team that agree that, at every change, a procedure must be  
+
* 2- policy : team that agree that, at every change, a procedure must be
3- coverage : minimum 40%
+
* 3- coverage : minimum 40%
 
 
 
The code coverage is not a primary measure, but can indicate a trend on code quality attention by developers
 
The code coverage is not a primary measure, but can indicate a trend on code quality attention by developers
 
But too much unit tests can be complicated to maintain.
 
But too much unit tests can be complicated to maintain.
 +
 
Feature coverage can be another measure.
 
Feature coverage can be another measure.
  
 
What we're not going to do in the short term for Android testing :
 
What we're not going to do in the short term for Android testing :
- language testing : only English at first
+
* language testing : only English at first
- multi-device testing
+
* multi-device testing
- BDD testing
+
* BDD testing
- search (case-sensitivity, etc) : it must be tested by unit tests on the ZIM lib
+
* search (case-sensitivity, etc) : it must be tested by unit tests on the ZIM lib
- network connectivity : slow networks can be simulated, but it becomes complicated. We assume the network is working correctly
+
* network connectivity : slow networks can be simulated, but it becomes complicated. We assume the network is working correctly
- test permissions of the system
+
* test permissions of the system
- test available storage
+
* test available storage
- test custom apps
+
* test custom apps
 
 
 
For the short term : use preferably Espresso and/or UIOtomato for Android testing.
 
For the short term : use preferably Espresso and/or UIOtomato for Android testing.
  
Line 62: Line 67:
  
 
Choice for Android testing (by the end of the week) :
 
Choice for Android testing (by the end of the week) :
- work on Travis automation (continuous building)
+
* work on Travis automation (continuous building)
- or work on local testing automation, that could be run manually by all Android developers. This one is chosen.
+
* or work on local testing automation, that could be run manually by all Android developers. This one is chosen.
 
+
For kiwix-html5 : the tooling seems good enough for now. About saucelabs : create a "kiwix" account and use this credential.
For saucelabs : create a "kiwix" account and use this credential.
 
  
 
No tests on kiwix desktop for now. Maybe for kiwix-serve?
 
No tests on kiwix desktop for now. Maybe for kiwix-serve?

Latest revision as of 15:46, 5 April 2017

These are my raw notes of the meeting :

Mossroy : Currently on kiwix-html5 repo : travis + saucelabs

Julian : saucelabs provides pre-built hardware and software with browser/mobile versions etc. bitbar : android devices test remotely + iOS

Guillaume : would like to test the installer on Windows/Mac/Linux. Travis only does Linux

Dattaz : curiosity

Matthieu : we should have tests on every project. Lack of tests on Kiwix.

Isaac : fragmentation => need to test on different devices

Emmanuel : non-regression is important. Scale, complexity increases

Julian :

Differences between what Matthieu and Isaac do

automate the most common user journeys : open the app, open a page, etc

BDD frameworks : given, when, then (natural language)

Opinion : little point to have common code between all the repos.

Java unit tests works well, but do not allow to interact with the system (permissions, SD card etc)

UIOtomato

Image-based testing : not recommended for us

Which content to use for the tests? Ray Charles archive allows to test some aspects, but not all of them.

Careful about the multi-lingual aspect.

Step by step : start with something simple to test, and see what needs to be improved.

Android pains : different storage locations, onboarding

Need to record the times taken by each test (with a fine granularity) => reports

Levels at Google :

  • 0- no tests
  • 1- at least one test, and CI that runs it
  • 2- policy : team that agree that, at every change, a procedure must be
  • 3- coverage : minimum 40%

The code coverage is not a primary measure, but can indicate a trend on code quality attention by developers But too much unit tests can be complicated to maintain.

Feature coverage can be another measure.

What we're not going to do in the short term for Android testing :

  • language testing : only English at first
  • multi-device testing
  • BDD testing
  • search (case-sensitivity, etc) : it must be tested by unit tests on the ZIM lib
  • network connectivity : slow networks can be simulated, but it becomes complicated. We assume the network is working correctly
  • test permissions of the system
  • test available storage
  • test custom apps

For the short term : use preferably Espresso and/or UIOtomato for Android testing.

Emmanuel : the goal is not have a lot of tests, but to have the platform and start.

Maximum time for automated testing : one hour

Choice for Android testing (by the end of the week) :

  • work on Travis automation (continuous building)
  • or work on local testing automation, that could be run manually by all Android developers. This one is chosen.

For kiwix-html5 : the tooling seems good enough for now. About saucelabs : create a "kiwix" account and use this credential.

No tests on kiwix desktop for now. Maybe for kiwix-serve?