Wikipedia on Rockbox?

[grandinferno]

I've been reading this thread eagerly since it's conception. This is so exciting. I so wish I could program so I could help out. I am a graphic designer though so if there's anything that needs prettying up (like a background for it or something) just PM me.

Looking forward to seeing a finished result. Great work connell!! You're a legend! Rockbox ... well... rocks!

[connell]

OK, in-page links (jumps) now work, and the back/forward mechanism is well on its way.

I've thought about a couple of ways to simplify the use of the plugin. The interface itself should/will be easy to use, however getting the files into the wiki folder (and updating the list of wikis file) is a tad over complicated for what should be an easy process. An application could be written which displays a list of available wikis on a server on the internet, downloads them upon request, and installs them into the correct folder. In the future it could be extended to only retrieve revisions. For this I'm thinking a cross-platform application would be best so I think Java is the way to go. Any Java programmers out there willing to take up the challange? I think I may start a thread in the General Rockbox forum for this idea (but not until the plugin is nearing readyness).

Connell

[BaZ]

Hi All,
Been following this and the Rockbox H300 thread from the start, but haven't had anything to contribute: I do have one question now. Is the project, which Connell is developing actually more of a cut down web-browser than simply a Wiki reader.

The reason I ask is that this really explands what my iRiver can do, I'm a research student and to be able to view note which I organise into HTML while on the go would be fantastic, as well as being able to access a downloaded Wikipedia.

Connell, the work you're doing here is amazing, I can't wait to try the first release. I have some very basic C skill if I can be of any help at all.

Regards,

BaZ

[connell]

Originally Posted by BaZ

actually more of a cut down web-browser than simply a Wiki reader

Yes, I would say so. The current program I have created converts simple HTML into the format used by my plugin. An application that managed the wikis on the device could also possibly convert html files into the correct format.

PS - Back/forward now works. I'm going to clean up the code a bit now.

Connell

[Tide]

Originally Posted by connell

Yes, I would say so. The current program I have created converts simple HTML into the format used by my plugin...

Connell, is it possible to implement your program to convert also PDF files to a format that could be read by Rockbox or by your reader? I think doing so would make the possibilities even greater for future document publications - specially for those downloading e-books and etc.

[connell]

Originally Posted by Tide

convert also PDF files

Yea, there is a GPL program called pdftohtml which could be used as an intermediate converter. I'll take a look at it tonight (GMT) to see if the html it produces is clean enough for my converter.

Connell

[FlashFlood]

Originally Posted by connell

OK, in-page links (jumps) now work, and the back/forward mechanism is well on its way.

I've thought about a couple of ways to simplify the use of the plugin. The interface itself should/will be easy to use, however getting the files into the wiki folder (and updating the list of wikis file) is a tad over complicated for what should be an easy process. An application could be written which displays a list of available wikis on a server on the internet, downloads them upon request, and installs them into the correct folder. In the future it could be extended to only retrieve revisions. For this I'm thinking a cross-platform application would be best so I think Java is the way to go. Any Java programmers out there willing to take up the challange? I think I may start a thread in the General Rockbox forum for this idea (but not until the plugin is nearing readyness).

Connell

I'm currently in a Java programming class. Although I probably do not have the skill to do it, I could ask my teacher to help me write a program like this... if that would be helpful....

[TheCowBoX]

It would be very helpful. Just remember that even if something happens (school obligations, whatnot), at least you'll have gotten as far as you did, and that's more than we were before.

[mpx1]

I like this Project very much.
Great work Connell!

[digitalb0y]

I m SO excited for this plugin. Being a student and audio engineer, this should be more usefull than anything I have seen so far. Thanks Connell.

Waiting eagerly,

C

[partisan]

Hi,

I'm new here, but a Rockbox user since the project started.
I joined this forum due to this interesting thread.

Do you know there is already a dictionary plugin part of Rockbox (see http://www.rockbox.org/twiki/bin/vie...kboxDictionary)?
So, why reinvent the wheel? The plugin is very fast, even with big files.
The interface is kinda basic though and scrolling long descriptions isn't implemented yet.

To be constructive, I prepared the latest Wiktionary dump (English, with nearly 100.000 articles) for this plugin.
You can download it at http://www.yourfilelink.com/get.php?fid=52700 (10MB zipped, 30MB unzipped).

It would be no big deal to convert the Wikipedia to the Rockbox dictionary format, but without scrolling it's somewhat pointless. Any volunteers?

- partisan

[golf7]

thanks for that wiktionary dump, would be great if someone implemented scrolling that would be awesome

[Crossfire238]

::edit::
damn i need to start reading before i post!
i just answered two of my questions....i'll shut up now

[jewman]

This project could be BIG!!! Check this out! Apparently, this Wikimedia Foundation includes wiki sites that provide: books, media, quotations, a library, and even an animal species directory. We just need to get the reader off the ground. The possibilities are endless!

[connell]

Hi,

Sorry for the lack of updates guys (my internet connection has been erratic).

For jewman - the HTML conversion software basically means that anything in simple web page format can be viewed in the wiki (within limits). This means much more than just Wikimedia content! The only restriction is that it must be available to download in a convenient way (which Wikimedia provides).

I noticed someone mentioned Wiktionary. Over the last few days I have already been converting the wiktionary into the format used by my wiki reader. (I had thought that it was a good starting point to test the reader). It works well (looks good although there are a few glitches with the conversion). It takes a while to do the conversion (about 1-2 topics a second) but this can be greatly improved with better software. Screenshots of the plugin running the Wiktionary to follow in the next day or so . I have implemented a very basic search feature (only returns topics with the search string at the beginning of the name and it is quite slow) but there is room for loads of improvement.

Also for an interesing thought...a program which compiles the contents of a mailbox and generates a wiki allowing you to view e-mails on your Rockbox! Could be sync'ed with a program when you connect your jukebox to your PC. (I have slapped together a program to do this on my mailbox (which is in unix Maildir format) but it would need to be converted to be available with more popular formats (eg POP & IMAP). But so far...looks really cool.

Am I right in saying that TheCowBoX created a website for the project? (I'm too lazy to look through the whole thread). Is it still a go-er? Something that we could use to collect our ideas would be great (I think it is too big a project to have integrated with rockbox yet). An update mailing list would also be useful so that people can be informed of any major headway/releases.

Connell

[connell]

Originally Posted by partisan

Do you know there is already a dictionary plugin part of Rockbox

With a dedicated wiki plugin, we can have rich formatting (bold, italics etc), in-line topic links, a proper wiki interface (wiki selection, searching optimized for searching millions of articles etc). Many of these have already been implemented in my plugin. The dictionary plugin can be left to do what it has been designed to do - act as a fast, efficient method of displaying a relatively small amount of controlled information.

Connell

[Tide]

great work man!
Looking forward to get my hands on it! Specially the mail sync (hope people like me, a OSX user, won't be forget)

[Daavid]

Suffering suckatash! Sounds like its getting closer. Great work cannell. I'm really looking forward to having this reader on my Rockbox.

[partisan]

Originally Posted by connell

I tried to import Wikipedia into a local copy of MediaWiki (so that I can parse it more easily into a viewable form) using MySQL 4...
Over the last few days I have already been converting the wiktionary into the format used by my wiki reader. (...) It works well (looks good although there are a few glitches with the conversion). It takes a while to do the conversion (about 1-2 topics a second) but this can be greatly improved with better software.

So you need 28-55 hours for the conversion of the wiktionary (with approx. 100.000 entries) ?!! This can be greatly improved by taking the already existing tools on the wikipedia site and do some postprocessing afterwards to suit your format. It took me only 10-15 minutes to do the conversion to the rockbox dictionary format (see my last post).

Originally Posted by connell

With a dedicated wiki plugin, we can have rich formatting (bold, italics etc), in-line topic links, a proper wiki interface (wiki selection, searching optimized for searching millions of articles etc). Many of these have already been implemented in my plugin. The dictionary plugin can be left to do what it has been designed to do - act as a fast, efficient method of displaying a relatively small amount of controlled information.

Formatting is nice, but personally my primary interest is the content, not the presentation... in-line topic links would be very useful though. I thought enhancing the dictionary plugin could be an alternative to your approach. The speed of it's search is hardly beatable.

cheers
-partisan

[Mad_Big_Sausage]

But one of the greatest features (as mentioned previously) is wikipedias similar and "further reading" topic linking.
The dict plugin couldn't do this.

[connell]

Originally Posted by partisan

28-55 hours for the conversion of the wiktionary

Yea, with the current software this is a problem (it would take _ages_ for wikipedia) but there are reasons for this:

1 - The Wiktionary is converted into HTML using the MediaWiki software so essentially there is 2 conversions taking place (mediawiki to html then html to rockipedia). With more time, a mediawiki to rockipedia converter could be written to drastically reduce conversion time).

2 - The MediaWiki and MySQL content store that I am using are on seperate machines (due to file space restrictions). This introduces the extra slow down of a complete TCP/IP transfer of information across a network. This slow down would be completely removed with the converter consolidation in 1.

3 - The conversion software is written in Perl using the HTML::Tree module which is over-bloat for what is needed.

Originally Posted by partisan

I thought enhancing the dictionary plugin could be an alternative to your approach

I'm not sure if the users of the dictionary plugin would want to bloat that the extra features bring. The fact remains, however, that the most of the nitty-gritty work of the plugin is done (although the code is still very messy and unoptimised) and IMO an alpha release is not 100 years away. Having scanned the dictionary plugin code, it appears that the contents of the dictionary are stored in 2 files, an index file and a content file. This causes problems when large amounts of data are being used because FAT32, the filesystem used by the jukeboxes, only allows individual files up to 2GB. Being able to split the content into many files overcomes this, and uses the filesystems design to aid in the organisation of the topics. The dictionary plugin has only 315 lines of code and does not have the structures required to easily add formatting or linking support. In contrast the wiki plugin requires over 600 lines of code to implement the features it has so far (although this is likely to change as more are implemented and optimisation takes place).

Perhaps the best approach would be to analyse how the dictionary plugin achieves its search speed and incorporate that into the wiki plugin.

Connell

[soppen]

Man, this looks amazing! I'm really looking forward to it! Great work!

[gdap]

Hey,

I've registered on this forum to say good luck with the project. My one request would be that you use a good text input mechanism (e.g. encylopodia on the ipod displays a hollow box and you can use the wheel to scroll through characters). The dictionary's text input is annoying to use in comparison.