Wikipedia on Rockbox?

[TheCowBoX]

I thought I emailed you already. I sent another just in case.

[TheCowBoX]

I just had a conversation with someone that fruited a great idea:

FRIEND: Hmm. You know you could make your life easier by just constructing a (XML) parser with Lex.
Chad48309: Well, to work with the ROCKbox plugin format, it has to be written in C among other things. And we'd like to get a search engine done first (I'm assuming).
FRIEND: Lex outputs a parser in standard .c format; then you take that and plug it into your search engine code, and put whatever plugin wrapper around you need to get the module running.

[abraxas]

I noticed a searchengine.rock in the viewers directory. I'll do some research
login details to sourceforge posted to cowbox

[theking12345]

I take it if the articles themselves are actually going to be in *.txt format, the hyperlinks wont be there? The blue links that pop up every other word that make it really hard to leave the site because there's always another interesting article.

[Daavid]

An idea for that splash screen yuo wanted. What do you think?

[TheCowBoX]

Awesome! I love it!

[Daavid]

Cool. Well feel free to use it. Glad I could do somethnig since I can't code.
Dave

[wraypa1]

In order to stop the links not functioning, wouldnt it be easier to use a simple cut down web browser to do it. While i dont actually know anything about coding one could be converted to run on the h3xx architecture.

P.S. sorry if this was a stupid suggestion

[TheCowBoX]

Not at all, but it would blow up the size of what's needed. However, I think that an XML parser written in Lex could interpret the information from Wikipedia (including formatting) and show it just how it is.

[the_winch]

Originally Posted by theking12345

I take it if the articles themselves are actually going to be in *.txt format, the hyperlinks wont be there? The blue links that pop up every other word that make it really hard to leave the site because there's always another interesting article.

Using text files sounds sensible as wikipedia articles are really just text with some markup. Links to other wikipedia pages should be possible as it will just need to load another article, something the plugin will need to do anyway.

Looking at a web browser might be a good idea as many of the problems will be similar. I would think most existing browsers will be so complex that stripping them down would be impractical. HTML is a lot more complicated than the simple markup on most wikipedia articles.

I'm not really sure what the advantages of using xml over the existing wiki markup are. It looks like it would increase the size of the files and make them harder to parse.

[abraxas]

Hello,
Anyone looked between the <page></page> tags of our 4G expanded download yet?
Regarding embedded hyperlinks in text, they are not html tags, but are pointers to other document names.

There be 5394066 lines of data. I'm sure you're not going to be able to get inside this with a generic open(FILE_HANDLE) and will probably have to read directly from the disk in nice blocks.
regards
A
PS Nice logo

[TheCowBoX]

I still say splitting it up into:

\rockwiki
|-\a
  |-aa.txt
  |-ab.txt
  |-ac.txt
...

Would be best.

[abraxas]

Great idea for distributing it TheCowBox, I feel that during the parsing process we'll run into problems with it's size and any xml parser we choose. If or when you get the file you may need this to look at it

http://www.movsd.com/download/pfe101i.zip
( here is the calling page http://www.movsd.com/tools.htm )

My usual stable of viewers and editors freaked out!

Also, and possibly more importantly to xml parsers, the file does not appear to be a "well formed" document. In fact the file looks truncated. It stops in mid-word.

DrippyDonut did you notice this? Just ctrl End and you'll see what I mean

Regards
A

[possumking]

So, how's everything coming along? I appreciate all the work that you all are putting into this.

[TheCowBoX]

Hmmm...
I dunno what to do. If it's that large truncated, I'd hate to see how big it is normally. Should we start looking at compression schemes? Also, would it be possible to write a script to divide the files into the format I specified? I don't use *NIX anymore, but I have Cygwin, so I'll look into it.

[Matthias88]

here is my idea (funny: I thought about this about one week before this thread was opened):
Use PC to generate a striped-down version of WikipediaXML file, if you want to. Only interesting this is the <page></page> and the <title></title> before that.
An all in one File is possible (because fopen only returns a pointer to fileposition; i think we can open a 2GB file on rockbox?)

next you need a table-file: Every record (in alphabetical position) and the desired fseek-Position (byte offset from beginning of datafile), so we can seek to that. And voila, we have our entry
The designthing of the wikipedia articles: tables and headlines are the most important thing. only a small interpreter is needed, but i'm not really able to code it.

[TheCowBoX]

The big problem with a one-file system isn't the ability to open it (though it could be a factor), it's speed. And besides, it's alot bigger than two gigs, as we've seen. I think that the iPod people are using a compression scheme. Or maybe if you strip it down enough, you might get it to their alleged size. Either way, start small, work your way up.

[Matthias88]

Hm sry, I'm from germany. OUR wiki XML is 1,7GB uncompressed...(only articles without previous versions)

[connell]

Hi,

I'm new in this forum and joined due to this thread.

I thought about having wikipedia on my iriver last year, and downloaded the xml wikipedia output file pages-articles.xml.bz2. I then re-imported this into a MySQL database using MediaWiki 1.5. Then using the ipod linux script, with a few modifications, I started getting the HTML output from the version of wikipedia now on my computer. This script uses the *nix utility html2text to strip out all html tags (including links). I had to stop this as my version of MySQL was too old (and causing problems). What I was going to do after that was write a shell script to sort the files into folders depending on the title (and with a maximum depth of say 4-8 directories:
For Apples:
A\P\P\L\Apples.txt
For Bus:
B\U\S\Bus.txt
This would ensure that it would be quite quick to navigate to an article in the fileview and it would be simple to write a plugin that observes the trend and provides a very simple search facility.
An index could be created which allows you to more comprehensively search for titles.

I think I'll try and update MySQL and try this again. I'll post if I get any further.

Thanks,
Connell
(inspired by Doug. Adams to do this aswell

[digitalb0y]

wow, that's a pretty clever file structure idea.

C

[TheCowBoX]

Is there a directory limit? If not, this would be a great idea. Kind of like:

\a
  |-\a
  |   ...
  |-\r
  |  |-d
...aardvark.txt

Seems kinda confusing, but I'm sure it could be done with a shell script.

[kagelump]

haha... so did they beat us?

[JungMin]

Well, apparently so.....It looks really cool.

[moesbarn]

i don't suppose their work would be able to help us in any way? it looks like they've used a dual boot configuration, way to make it complicated!

[nexxyz]

Well, since it is on sourceforge and written for iPodLinux, it should be quite useful, right?