TED (Technology, Entertainment, Design) is a global set of conferences under the slogan "ideas worth spreading". They address a wide range of topics within the research and practice of science and culture, often through storytelling. The speakers are given a maximum of 18 minutes to present their ideas in the most innovative and engaging ways they can. Its web site is www.ted.com.
The purpose of this project is to create a sustainable solution to create ZIM files providing the TED and TEDx videos in a similar manner like ted.com
- A script (python) able to create easily following ZIM files of the TED and TEDx videos with the possibility to filter by language/conference/topic
- A list of TED talks can be found here
- The data should be scraped from ted.com.
- Videos should be available in HTML5 and subtitles need to be supported
- The ZIM should provide a simple filtering/search solution to find content (by author, language, title, conference, topic, ....)
As it currently stands there is a redesign of the TED site, that is currently available at http://new.ted.com.
We should focus on scraping that site, because the old one will eventuelly get discontinued.
http://new.ted.com/talks gives you a list of all the TED talks sorted by playlists and categories.
http://new.ted.com/talks/browse?sort=popular will give you list of all the TED talks in one place sorted by popularity.
It would be best to scrape this site and add the metadata (Category, playlist etc.) by ourselves later on.
Ideas to achieve it
- Retrieve the list of TED(x) presentations with medatas in a local database
- Download videos and re-encode them if necessary
- Retrieve the video subtitle files from www.amara.org
- Subtitle don't make so much sense for TEDx
- TED has a translation program here
- Interesting to read this to get an idea how to store a database client side.
- Fill the HTML templates with the data from the XML/RDF and write the index pages in a target directory
- Run zimwriterfs to create the corresponding ZIM file of your target directory
- First three days (17.-19.02.2014):
- Rest of the first week (20. - 23.02.2104):
- Collection of all the data
- Writing the Scraper, that scrapes TED.com
- Writing the Scraper, that scrapes the TED translation page on ww.amara.org
- Writing the html templates
- Writing a python script, that dumps the scraped data into the HTML pages, creating static content
- Collection of all the data
- First three days of the second week (24. - 26.02.2014):
- Implementing the local database, that manages all the content
- Finally: Creating the first prototype zim files
- Rest of the second week (27.02 - 2.03.2014):
- Improving everything
- Fixing possible bugs
- Possible other things:
- Implement a way to play html5 videos on the Android version of Kiwix (Bug can be found here)
The networking library we are going to use will be requests. Requests is pretty easy to use and straightforward.
The scraping library, that we are going to use will be Beautifulsoup4 . You can easily go through all nodes of an HTML document with it. HTML elements can be either selected by CSS selectors or by regular expressions.
Downloading videos from TED is easy and straightforward. An example of an URL to a video can be found here
The subtitles of videos are harder to get. They are all available on here in multiple formats. We will use the caption format SRT.
Building HTML sites out of the scraped content
We want to 'export' our scraped data to html, so we can run the zim tool on it and create compressed zim files off it. Out of all the possibilities Jinja2 seems to be the best library for that.