JS-ctypes
With the unavailability of XulRunner in Ubuntu Oneiric and the possible removal of it from other distro, we decided to switch to js-ctypes components. The main reason is that our code won't be tied to a XR version (which are released every 6 weeks) as components are. It also has some side-advantages
JS-ctypes
ctypes is a Foreign Function Interface which is available in several languages. It allows one to call C functions (of a library) from Javascript or Python or other language supporting it. It doesn't require specific C code. It just calls arbitrary functions.
Porting
Porting Kiwix components to js-ctypes requires a lot of work so we decided to incorporate other changes. Porting a component consist of the following:
- Rewrite the C++ component into a C++ library with no mozilla dependency.
- Write a C++ program testing the C++ API.
- Write a C Wrapper around the C++ API.
- Write a C program testing the C Wrapper/API.
- Write a JS module interfacing with js-ctypes and the C wrapper.
- Fix existing JS code (gui.js, etc) to use the JS module.
Note: Because of the number of layers, it is important to respect the convention on naming which differs from layer to layer. Examples:
- component: zimAccessor
- component path: src/components/zimAccessor
- library name: zimAccessor
- library path: src/zimAccessor
- C++ library files: zimAccessor.cpp zimAccessor.h
- C Wrapper files: ZimAccessorWrapper.cpp ZimAccessorWrapper.h
- Tester path: src/zimTester/
- C++ tester file: zimTester.cpp
- C Wrapper tester file: zimCTester.c
Rewrite the C++ component into a C++ library
The main rule here is to remove Mozilla dependency. It is very important to remove it completely otherwise it's a waste of time. Advantages of removing MOZ dep:
- We build a shared/static lib which is not a component.
- We don't need the mozilla stack to build it (ease Windows, and other system setup like arm)
- We don't need an every-release recompile of our code
- We don't even need to release anything to have it work with newer XR.
- We can use it to build a Kiwix UI with webkit (for example on Android). We'll code all component code and just need a new UI.
- We can use the same code for kiwix-serve
- We could use it to create a Kiwix server instead of a kiwix-http-server (allow one to administer the server library using a Web UI)
- We can then split kiwix code with components code and have a kiwix-libs package.
Writing this library is quite easy. It's mostly a copy-paste of the component code then cleaning.
Note: The goal is to replace the component but until all components are ported, we'll have a duplication of the code in the tree. We need to port-back every addition to the comonent to the new library until it replaces the component completely.
Rules to embrace/respect:
- nsAString is replaced with string
- nsACString is replaced with char *
- nsIURI is replaced with string and code adapted consequently.
- No more retVal nor NS_OK. Methods uses return type (bool mostly)
- Keep case on method names.
- don't add features while you port to keep it understandable and revert-able.
The component uses no header file (it is generated from IDL at compile time and is really verbose) so you also need to write it manually. Refer to example.
Write a C++ program testing the C++ API
Instead of writing a program to test the features of the API, we should write unit tests but I have no prior experience with unit testing with C/C++ so I choose the loose way. Feel free to improve that.
The idea here is just to make sure that all methods are available and working as expected.
- C++ library header file
- C++ tester source
I'm not very proud of it, a lot of stuff are hard coded and while it gets a ZIM as argument, it expects the swahili zim to test everything.
Write a C Wrapper around the C++ API
js-ctypes (and ctypes) only works with C code and not with C++. Because our code base is in C++ (for good reasons), we need to add a C layer to interface with the C++ library. This is done by creating a C++ library (exported as C) containing only functions which would each call the C++ API. We'll add a couple functions to create/destroy the instance of the C++ class and each function will accept a pointer to the instance as parameter.
Rules:
- Create a void pointer type to match the class pointer.
- Create functions to crete/destroy the class instance.
- All functions are prefixed with library name (and keep case)
- All functions have previously created type as first argument
- expect C types as input (char *)
- convert char * to string inside your function
- convert outputed string to char *
- return the actual data. Most of the time, we only have one return variable. Use that as return type for the wrapper. It's easier and cleaner from the JS perspective.
- if you need to return multiple values, use a struct example
Header example
typedef void * HZIMCLASS; HIMCLASS ZimAccessor_Create( void ); void ZimAccessor_Destroy( HZIMCLASS h ); const char* ZimAccessor_GetPageUrlFromTitle( HZIMCLASS h, char* url);
Source example
/* creates instance of ZimAccessor */ HZIMCLASS ZimAccessor_Create( void ) { ZimAccessor * zimAccessor = new ZimAccessor(0); // Return its pointer (opaque) return (HZIMCLASS)zimAccessor; } /* Delete instance of ZimAccessor */ void ZimAccessor_Destroy( HZIMCLASS h ) { assert(h != NULL); // Convert from handle to ZimAccessor pointer ZimAccessor * zimAccessor = (ZimAccessor *)h; delete zimAccessor; } /* Return a page url fronm title */ const char* ZimAccessor_GetPageUrlFromTitle( HZIMCLASS h, char* title) { assert(h != NULL); ZimAccessor * zimAccessor = (ZimAccessor *)h; string cptitle = string(title); string url = ""; zimAccessor->GetPageUrlFromTitle(cptitle, url); return url.c_str(); }
Write a C program testing the C Wrapper/API
Instead of writing a program to test the features of the API, we should write unit tests but I have no prior experience with unit testing with C/C++ so I choose the loose way. Feel free to improve that.
The idea here is just to make sure that all methods are available and working as expected.
Note: This testing is more different than the previous C++ one as the API has changed while switching C in order to simplify it (make it look more like JS).
- C++ library header file
- C++ tester source
I'm not very proud of it, a lot of stuff are hard coded and while it gets a ZIM as argument, it expects the swahili zim to test everything.
Write a JS module interfacing with js-ctypes and the C wrapper
Creating a JS module is not required but it's cleaner and convenient: ctypes calls requires the pointer as first argument and the return value of the calls is specific to ctypes. Our module will use js-ctypes abstract the C librar so that the whole js-ctypes is hidden.
- JS module example
Documentation:
Rules:
- Open library and declare functions in register method.
- you need to declare every function of the C API.
- char * returned by API are available via .readString()
- int and bool returned are available via contents (raw).
- declare() method gets name, abi (we use default so that it's multiplatform), return type (you need to choose this), then parameter (first one is our pointer to the class).
- retrieving values from a struct example
- Change case on method names to differentiate with original API.
Tips
- .so libs in xulrunner folder
- jsm are compiled and cache at registration. You need to remove cache (or whole profile) after changes.
Remaining work
- unicode
- makefiles
- Replace XR with FF
- Compatibility with XR 1.9.2?