Difference between revisions of "JS-ctypes"

From Kiwix
Jump to navigation Jump to search
Line 48: Line 48:
'''Note''': The goal is to replace the component but until all components are ported, we'll have a duplication of the code in the tree. We need to port-back every addition to the comonent to the new library until it replaces the component completely.
'''Note''': The goal is to replace the component but until all components are ported, we'll have a duplication of the code in the tree. We need to port-back every addition to the comonent to the new library until it replaces the component completely.


* C++ Component code [http://kiwix.svn.sourceforge.net/viewvc/kiwix/moulinkiwix/src/components/zimAccessor/zimAccessor.cpp?revision=2920&view=markup example]
* C++ Component code [http://kiwix.svn.sourceforge.net/viewvc/kiwix/moulinkiwix/src/components/zimAccessor/zimAccessor.cpp?revision=HEAD&view=markup example]
* C++ library [http://kiwix.svn.sourceforge.net/viewvc/kiwix/moulinkiwix/src/zimAccessor/zimAccessor.cpp?revision=2920&view=markup example]
* C++ library [http://kiwix.svn.sourceforge.net/viewvc/kiwix/moulinkiwix/src/zimAccessor/zimAccessor.cpp?revision=HEAD&view=markup example]
* C++ library header [http://kiwix.svn.sourceforge.net/viewvc/kiwix/moulinkiwix/src/zimAccessor/zimAccessor.h?revision=2811&view=markup example]
* C++ library header [http://kiwix.svn.sourceforge.net/viewvc/kiwix/moulinkiwix/src/zimAccessor/zimAccessor.h?revision=HEAD&view=markup example]


Rules to embrace/respect:
Rules to embrace/respect:
Line 69: Line 69:
The idea here is just to make sure that all methods are available and working as expected.
The idea here is just to make sure that all methods are available and working as expected.


* C++ library [http://kiwix.svn.sourceforge.net/viewvc/kiwix/moulinkiwix/src/zimAccessor/zimAccessor.h?revision=2811&view=markup header file]
* C++ library [http://kiwix.svn.sourceforge.net/viewvc/kiwix/moulinkiwix/src/zimAccessor/zimAccessor.h?revision=HEAD&view=markup header file]
* C++ tester [http://kiwix.svn.sourceforge.net/viewvc/kiwix/moulinkiwix/src/zimTester/zimTester.cpp?revision=2920&view=markup source]
* C++ tester [http://kiwix.svn.sourceforge.net/viewvc/kiwix/moulinkiwix/src/zimTester/zimTester.cpp?revision=HEAD&view=markup source]


I'm not very proud of it, a lot of stuff are hard coded and while it gets a ZIM as argument, it expects the swahili zim to test everything.
I'm not very proud of it, a lot of stuff are hard coded and while it gets a ZIM as argument, it expects the swahili zim to test everything.
Line 133: Line 133:
}
}
</source></blockquote>
</source></blockquote>
=== Write a C program testing the C Wrapper/API ===
Instead of writing a program to test the features of the API, we should write unit tests but I have no prior experience with unit testing with C/C++ so I choose the loose way.
Feel free to improve that.
The idea here is just to make sure that all methods are available and working as expected.
'''Note''': This testing is more different than the previous C++ one as the API has changed while switching C in order to simplify it (make it look more like JS).
* C++ library [http://kiwix.svn.sourceforge.net/viewvc/kiwix/moulinkiwix/src/zimAccessor/zimAccessor.h?revision=HEAD&view=markup header file]
* C++ tester [http://kiwix.svn.sourceforge.net/viewvc/kiwix/moulinkiwix/src/zimTester/zimCTester.cpp?revision=HEAD&view=markup source]
I'm not very proud of it, a lot of stuff are hard coded and while it gets a ZIM as argument, it expects the swahili zim to test everything.


== Tips ==
== Tips ==

Revision as of 10:01, 29 November 2011

With the unavailability of XulRunner in Ubuntu Oneiric and the possible removal of it from other distro, we decided to switch to js-ctypes components. The main reason is that our code won't be tied to a XR version (which are released every 6 weeks) as components are. It also has some side-advantages

JS-ctypes

ctypes is a Foreign Function Interface which is available in several languages. It allows one to call C functions (of a library) from Javascript or Python or other language supporting it. It doesn't require specific C code. It just calls arbitrary functions.

Porting

Porting Kiwix components to js-ctypes requires a lot of work so we decided to incorporate other changes. Porting a component consist of the following:

  1. Rewrite the C++ component into a C++ library with no mozilla dependency.
  2. Write a C++ program testing the C++ API.
  3. Write a C Wrapper around the C++ API.
  4. Write a C program testing the C Wrapper/API.
  5. Write a JS module interfacing with js-ctypes and the C wrapper.
  6. Fix existing JS code (gui.js, etc) to use the JS module.

Note: Because of the number of layers, it is important to respect the convention on naming which differs from layer to layer. Examples:

  • component: zimAccessor
  • component path: src/components/zimAccessor
  • library name: zimAccessor
  • library path: src/zimAccessor
  • C++ library files: zimAccessor.cpp zimAccessor.h
  • C Wrapper files: ZimAccessorWrapper.cpp ZimAccessorWrapper.h
  • Tester path: src/zimTester/
  • C++ tester file: zimTester.cpp
  • C Wrapper tester file: zimCTester.c

Rewrite the C++ component into a C++ library

The main rule here is to remove Mozilla dependency. It is very important to remove it completely otherwise it's a waste of time. Advantages of removing MOZ dep:

  • We build a shared/static lib which is not a component.
  • We don't need the mozilla stack to build it (ease Windows, and other system setup like arm)
  • We don't need an every-release recompile of our code
  • We don't even need to release anything to have it work with newer XR.
  • We can use it to build a Kiwix UI with webkit (for example on Android). We'll code all component code and just need a new UI.
  • We can use the same code for kiwix-serve
  • We could use it to create a Kiwix server instead of a kiwix-http-server (allow one to administer the server library using a Web UI)
  • We can then split kiwix code with components code and have a kiwix-libs package.

Writing this library is quite easy. It's mostly a copy-paste of the component code then cleaning.

Note: The goal is to replace the component but until all components are ported, we'll have a duplication of the code in the tree. We need to port-back every addition to the comonent to the new library until it replaces the component completely.

Rules to embrace/respect:

  1. nsAString is replaced with string
  2. nsACString is replaced with char *
  3. nsIURI is replaced with string and code adapted consequently.
  4. No more retVal nor NS_OK. Methods uses return type (bool mostly)
  5. Keep case on method names.
  6. don't add features while you port to keep it understandable and revert-able.

The component uses no header file (it is generated from IDL at compile time and is really verbose) so you also need to write it manually. Refer to example.

Write a C++ program testing the C++ API

Instead of writing a program to test the features of the API, we should write unit tests but I have no prior experience with unit testing with C/C++ so I choose the loose way. Feel free to improve that.

The idea here is just to make sure that all methods are available and working as expected.

I'm not very proud of it, a lot of stuff are hard coded and while it gets a ZIM as argument, it expects the swahili zim to test everything.

Write a C Wrapper around the C++ API

js-ctypes (and ctypes) only works with C code and not with C++. Because our code base is in C++ (for good reasons), we need to add a C layer to interface with the C++ library. This is done by creating a C++ library (exported as C) containing only functions which would each call the C++ API. We'll add a couple functions to create/destroy the instance of the C++ class and each function will accept a pointer to the instance as parameter.

Rules:

  1. Create a void pointer type to match the class pointer.
  2. Create functions to crete/destroy the class instance.
  3. All functions are prefixed with library name (and keep case)
  4. All functions have previously created type as first argument
  5. expect C types as input (char *)
  6. convert char * to string inside your function
  7. convert outputed string to char *
  8. return the actual data. Most of the time, we only have one return variable. Use that as return type for the wrapper. It's easier and cleaner from the JS perspective.

Header example

typedef void * HZIMCLASS;
HIMCLASS ZimAccessor_Create( void );
void ZimAccessor_Destroy( HZIMCLASS h );
const char* ZimAccessor_GetPageUrlFromTitle( HZIMCLASS h, char* url);

Source example

/* creates instance of ZimAccessor */
HZIMCLASS ZimAccessor_Create( void ) {
    ZimAccessor * zimAccessor = new ZimAccessor(0); 

    // Return its pointer (opaque)
    return (HZIMCLASS)zimAccessor;
}

/* Delete instance of ZimAccessor */
void ZimAccessor_Destroy( HZIMCLASS h ) {
    assert(h != NULL);

    // Convert from handle to ZimAccessor pointer
    ZimAccessor * zimAccessor = (ZimAccessor *)h;

    delete zimAccessor;
}

/* Return a page url fronm title */
const char* ZimAccessor_GetPageUrlFromTitle( HZIMCLASS h, char* title) {
    assert(h != NULL);

    ZimAccessor * zimAccessor = (ZimAccessor *)h;

    string cptitle = string(title);
    string url = "";
    zimAccessor->GetPageUrlFromTitle(cptitle, url);
    return url.c_str();
}

Write a C program testing the C Wrapper/API

Instead of writing a program to test the features of the API, we should write unit tests but I have no prior experience with unit testing with C/C++ so I choose the loose way. Feel free to improve that.

The idea here is just to make sure that all methods are available and working as expected.

Note: This testing is more different than the previous C++ one as the API has changed while switching C in order to simplify it (make it look more like JS).

I'm not very proud of it, a lot of stuff are hard coded and while it gets a ZIM as argument, it expects the swahili zim to test everything.


Tips

  • .so libs in xulrunner folder

Remaining work

  • unicode
  • makefiles