JS-ctypes

From Kiwix
Revision as of 11:02, 29 November 2011 by Reg (talk | contribs)
Jump to navigation Jump to search

With the unavailability of XulRunner in Ubuntu Oneiric and the possible removal of it from other distro, we decided to switch to js-ctypes components. The main reason is that our code won't be tied to a XR version (which are released every 6 weeks) as components are. It also has some side-advantages

JS-ctypes

ctypes is a Foreign Function Interface which is available in several languages. It allows one to call C functions (of a library) from Javascript or Python or other language supporting it. It doesn't require specific C code. It just calls arbitrary functions.

Porting

Porting Kiwix components to js-ctypes requires a lot of work so we decided to incorporate other changes. Porting a component consist of the following:

  1. Rewrite the C++ component into a C++ library with no mozilla dependency.
  2. Write a C++ program testing the C++ API.
  3. Write a C Wrapper around the C++ API.
  4. Write a C program testing the C Wrapper/API.
  5. Write a JS module interfacing with js-ctypes and the C wrapper.
  6. Fix existing JS code (gui.js, etc) to use the JS module.

Note: Because of the number of layers, it is important to respect the convention on naming which differs from layer to layer. Examples:

  • component: zimAccessor
  • component path: src/components/zimAccessor
  • library name: zimAccessor
  • library path: src/zimAccessor
  • C++ library files: zimAccessor.cpp zimAccessor.h
  • C Wrapper files: ZimAccessorWrapper.cpp ZimAccessorWrapper.h
  • Tester path: src/zimTester/
  • C++ tester file: zimTester.cpp
  • C Wrapper tester file: zimCTester.c

Rewrite the C++ component into a C++ library

The main rule here is to remove Mozilla dependency. It is very important to remove it completely otherwise it's a waste of time. Advantages of removing MOZ dep:

  • We build a shared/static lib which is not a component.
  • We don't need the mozilla stack to build it (ease Windows, and other system setup like arm)
  • We don't need an every-release recompile of our code
  • We don't even need to release anything to have it work with newer XR.
  • We can use it to build a Kiwix UI with webkit (for example on Android). We'll code all component code and just need a new UI.
  • We can use the same code for kiwix-serve
  • We could use it to create a Kiwix server instead of a kiwix-http-server (allow one to administer the server library using a Web UI)
  • We can then split kiwix code with components code and have a kiwix-libs package.

Writing this library is quite easy. It's mostly a copy-paste of the component code then cleaning.

Note: The goal is to replace the component but until all components are ported, we'll have a duplication of the code in the tree. We need to port-back every addition to the comonent to the new library until it replaces the component completely.

Rules to embrace/respect:

  1. nsAString is replaced with string
  2. nsACString is replaced with char *
  3. nsIURI is replaced with string and code adapted consequently.
  4. No more retVal nor NS_OK. Methods uses return type (bool mostly)
  5. Keep case on method names.
  6. don't add features while you port to keep it understandable and revert-able.

The component uses no header file (it is generated from IDL at compile time and is really verbose) so you also need to write it manually. Refer to example.

Write a C++ program testing the C++ API

Instead of writing a program to test the features of the API, we should write unit tests but I have no prior experience with unit testing with C/C++ so I choose the loose way. Feel free to improve that.

The idea here is just to make sure that all methods are available and working as expected.

I'm not very proud of it, a lot of stuff are hard coded and while it gets a ZIM as argument, it expects the swahili zim to test everything.

Write a C Wrapper around the C++ API

js-ctypes (and ctypes) only works with C code and not with C++. Because our code base is in C++ (for good reasons), we need to add a C layer to interface with the C++ library. This is done by creating a C++ library (exported as C) containing only functions which would each call the C++ API. We'll add a couple functions to create/destroy the instance of the C++ class and each function will accept a pointer to the instance as parameter.

Rules:

  1. Create a void pointer type to match the class pointer.
  2. Create functions to crete/destroy the class instance.
  3. All functions are prefixed with library name (and keep case)
  4. All functions have previously created type as first argument
  5. expect C types as input (char *)
  6. convert char * to string inside your function
  7. convert outputed string to char *
  8. return the actual data. Most of the time, we only have one return variable. Use that as return type for the wrapper. It's easier and cleaner from the JS perspective.
  9. if you need to return multiple values, use a struct example

Header example 

typedef void * HZIMCLASS;
HIMCLASS ZimAccessor_Create( void );
void ZimAccessor_Destroy( HZIMCLASS h );
const char* ZimAccessor_GetPageUrlFromTitle( HZIMCLASS h, char* url);

Source example 

/* creates instance of ZimAccessor */
HZIMCLASS ZimAccessor_Create( void ) {
    ZimAccessor * zimAccessor = new ZimAccessor(0); 

    // Return its pointer (opaque)
    return (HZIMCLASS)zimAccessor;
}

/* Delete instance of ZimAccessor */
void ZimAccessor_Destroy( HZIMCLASS h ) {
    assert(h != NULL);

    // Convert from handle to ZimAccessor pointer
    ZimAccessor * zimAccessor = (ZimAccessor *)h;

    delete zimAccessor;
}

/* Return a page url fronm title */
const char* ZimAccessor_GetPageUrlFromTitle( HZIMCLASS h, char* title) {
    assert(h != NULL);

    ZimAccessor * zimAccessor = (ZimAccessor *)h;

    string cptitle = string(title);
    string url = "";
    zimAccessor->GetPageUrlFromTitle(cptitle, url);
    return url.c_str();
}

Write a C program testing the C Wrapper/API

Instead of writing a program to test the features of the API, we should write unit tests but I have no prior experience with unit testing with C/C++ so I choose the loose way. Feel free to improve that.

The idea here is just to make sure that all methods are available and working as expected.

Note: This testing is more different than the previous C++ one as the API has changed while switching C in order to simplify it (make it look more like JS).

I'm not very proud of it, a lot of stuff are hard coded and while it gets a ZIM as argument, it expects the swahili zim to test everything.

Write a JS module interfacing with js-ctypes and the C wrapper

Creating a JS module is not required but it's cleaner and convenient: ctypes calls requires the pointer as first argument and the return value of the calls is specific to ctypes. Our module will use js-ctypes abstract the C librar so that the whole js-ctypes is hidden.

Documentation:

Rules:

  1. Open library and declare functions in register method.
  2. you need to declare every function of the C API.
  3. char * returned by API are available via .readString()
  4. int and bool returned are available via contents (raw).
  5. declare() method gets name, abi (we use default so that it's multiplatform), return type (you need to choose this), then parameter (first one is our pointer to the class).
  6. retrieving values from a struct example
  7. Change case on method names to differentiate with original API.
  8. abstract the API by providing direct value output to your methods.
  9. export your jsm as libXXX

Beware that wrong declaration can cause your jsm not to load (error message is misleading: File Not Found). Comment your declarations out and add them back one by one to find the problem. I chose arbitrary types which worked for my simple tests and some might be wrong. Think for your self and adjust.

I chose:

  • int16_t (16b integer) for boolean returned
  • ctypes.int32_t.ptr (pointer to 32b integer) for instance pointer
  • ctypes.char.ptr pointer to char for all strings.

Fix existing JS code to use the JS module

This part hasn't been done yet. Only local tests. Due to the way the current code mixes the various components it is very difficult to change that code until all components are converted. Once all are converted, we can rewrite part of that code and make it a lot simpler & cleaner than it currently is.

Tips

  • To test js-ctypes and your jsm, you need to put your statically built CWrapper .so lib in xulrunner folder. We'll search later if there's a way to place it elsewhere and have it discovered. Anyway, js-ctypes.open() can take a full path as parameter.
  • jsm are compiled and cache at registration. You need to remove cache (or whole profile) after changes.

Remaining work

unicode

Absolutely zero work as been done regarding unicode and string handlings. It just work on Linux but I'm 100% sure it will fail badly on windows or with very exotic paths. Because of the pain it is to manage string properly, it has been agreed that we would wait until the components are ported then we'll find a way to fix the strings.

I believe we might want then to duplicate code from nsString (mozilla) or libICU.

makefiles

library, wrapper and tester makefiles are hardcoded right now. Well need to move them to autotools.

Replace XR with FF

This is the target. Once we have those compo working, we need to investigate using FF as a XR engine. This is theoretically possible but I haven't found example and initial tests showed difficulties with main.xul.

Compatibility with XR 1.9.2

js-ctypes have been introduced with XR 1.9.2 but the ABI was incomplete or different. Once we have everything working we'll need to dig that and check if we need it.


Progression

  • zimAccessor: complete
  • contentManager: jsm not finished
  • xapianAccessor: not started
  • zimXapianIndexer: not started

In the JSM we open() libXX.so. We'll need to replace that with appropriate extension for each platform. There's an xpcom component which guess the extension based on platform but can't remember the name.

Retrieved from "https://wiki.kiwix.org/w/index.php?title=JS-ctypes&oldid=4435"