Modwiki

[deadite4]

I would love a little community input on a new modwiki.  In early March, 2011 our programmer at the time did an HTML scrape of the modwiki site while it was still up.  I've had it this whole time and just remembered about it not too long ago.

It needs cleaning up as the scrape turned the menus/headers/footers of the wiki template into HTML.  I've spent the last week cleaning out all of the menu/footer stuff and am just about done.  Another week and I can probably have it mostly clean to just the information.  The scrape also did not include image files, but I've been able to collect a lot of them from web.archive.  They don't have all of them, but they have a lot of them and I've been successfully able to get things back in order from the tests I've done.  thankfully, most of the interwiki page linking all works and doesn't need any work.

So my question is what to do with the information that makes it useful to the community again?  I figure there's 3 primary options - HTML site, downloadable PDF or back to a wiki.

The HTML site is the quickest, but crudest.  I could probably have the full site up in another 1-2 weeks and be mostly functional.  There would be broken links and they will have to stay that way.  There wouldn't be any type of community updating, no search feature.  It would essentially just be a cleaned up dump of the old information so it's at least kind of accessible to people.

A PDF would basically be me taking the clean HTML directory and compiling it all in a full PDF to download and use locally.  It's a bit nicer than a static site because you should get search capabilities back through the PDF format.  Updating still wouldn't be sensible and it would remain mostly as-is.

Putting it all back in a wiki would allow for long term maintenance and community updating.  It certainly would be the most robust route as we would gain all of the nice features of a wiki.  It also might take the longest to implement as I'm not familiar at all with getting an HTML site imported into a wiki.  I've read there are scripts and it seems possible but I ultimately have zero practice in it.

The initial package of data contained about 6000 files.  I removed 4000 which were just blank stubs, 1 for each game entity which seems useless at least in current scope.  With 2000 files left, it's been more manageable in terms of cleanup.

I think my ultimate goal would be to have a wiki back, and also the ability to export it all to PDF for a local download(either monthly or quarterly).  However, if that's just not useful for anyone and the community mostly feels like something like that isn't needed I won't waste my time on it.  I'll just dump the HTML site up and we can call it a day.

What's the general desire of the old information?

[Radegast]

It also might take the longest to implement as I'm not familiar at all with getting an HTML site imported into a wiki.  I've read there are scripts and it seems possible but I ultimately have zero practice in it.

What we really need is a wiki, but if you can upload a html dump then do it as soon as you can and someone else will write a custom script that extracts the info into a wikitext (if you are unsure how to do it yourself).

[The Happy Friar]

What we really need is a wiki,

Not trying to say no to a wiki, but what is the reason you feel this should be in a wiki?  The last wiki had issues with spammers, lots of dead ends, and very few actual contributors and was fairly empty (had less information then the doom3 script help file). 

More content would help but I've gotten more useful information on modding Quake 2 off a page of tutorials then I got for Doom 3 from the Wiki. 

[deadite4]

Spammers can be easily controlled through administration permissions.  If there was spam issues in the last one, I'd initially chalk it up to failed duties.

From what I've found from cleaning all of this stuff up, a lot of the dead ends  were from the wasteful scripts that were run just to generate pages as empty stubs.  I don't understand the point of a script generating 4000 empty stub pages for entities that no one is every going to fill in(or would even need to fill in).  Makes no sense as I'm looking through it all personally.

There's no reason why people couldn't write up tutorials and put them there.  Or at a minimum link them in there.  There is/was a tutorial part of the wiki.  I see a separate tutorial page in here that has to link 60-70 tuts easy.  Unfortunately most link to D3W, however since I have the actual links stored I might be able to track them over at web.archive......I haven't tried it yet.

A wiki is good for information with minimal discussion like when you need to know the keywords you can use in a material, etc......things that are definite.

[deadite4]

I've cleaned up all of the HTML files and linked most of the major images back.  Still need to spend some more time getting specific images in individual articles.

A static site certainly won't really work.  The whole thing behaves very slow as each page has to load everything when you go to it and still no search.  I'm going to see how a PDF looks/handles this week.  Still looking into a conversion to wiki too.

[The Happy Friar]

google/yahoo could be used to search the site.  I think it's like this "search_term site:webpage"

[Radegast]

So it has finally happened! Caedes and Kordex have resurrected the wiki:

https://modwiki.xnet.fi/

[tron]

Awesome, I notice the d3w archive as well!

edit: as things are it's extremely hard to navigate as there doesn't seem to be any indication of what pages have content.  Seem to remember the old modwiki had some kind of contents page/list of useful pages,don't suppose that got scraped as well?

[kordex]

Yea we know, there are even some pages missing but now that it's backed up with git and github repository it should stay online quite a while longer than the original did.

The restoration was done from a crawl to the archive of internet and then converted around with first HTML::WikiConverter to MediaWiki syntax and then with pandoc to mark down. After few lines of scripting we have it like this. The wiki engine however supports multiple categories and all that stuff which existed on the original wiki but i haven't had the time to go through all the category indexes that came from the crawl.

I just setup a crawl with greater scope so we should get most what is archived by the great archives so we can compare directly what pages are missing and add those but updates to existing and all that kind of stuff requires manual work which is not so easy.

Now that we have the ship to sail with we need some crew to operate it, so please if you have the motivation help us.

If there are major things you'd like to contribute one can fork the github repository at https://github.com/OpenTechEngine/modWiki.git and make a pull request once done. Perl HTML::WikiConverter and after that pandoc can quite nicely convert HTML version of wikipages into markdown.

If there is any problems or anything like that please open issue at https://github.com/OpenTechEngine/modWiki/issues and let's address them!

[nbohr1more]

Great stuff.

I'm not a code expert but I was in the habit of editing modwiki articles. Before it went down I think I fixed about
50% of the stub articles for material keywords using references from iddev and D3W.

I'll sign-up for editing when I get a chance. The one major article that is missing useful images is the Blend (stage keyword)
article. The wayback archive has most of them:

http://web.archive.org/web/20110111010457/http://www.modwiki.net/wiki/Blend_%28Material_stage_keyword%29

[motorsep]

Make a book - sell it on Amazon. I'll be the first buying it

[kordex]

Make a book - sell it on Amazon. I'll be the first buying it

Sorry but I don't have interest in making money by ripping off the community and books get out-dated pretty fast and are quite useless as scripting reference and pdf version would be on tpb in seconds

[motorsep]

That's a backward thinking 

Community cared sooo much about modwiki and d3w that modwiki stayed off the grid for as long as I can remember 

Plus, I don't mean literally putting it the way it is into a book. It has to be organized, examples have to be made, etc. No worries about it getting outdated - idtech 4 is a dinosaur anyway. Can't get more dated than that

Btw, there are maybe 10 people from the old community.

[kordex]

Great stuff.

Thank you, it's really motivating to hear that the work is worth of something!

I am currently in progress of restoring the rest of what is on archive.org. Let's hope it turns out easy to get those articles there.

If you have an github account already, I would like to invite you into the organization so you could edit the pages

[caedes]

Btw, there are maybe 10 people from the old community.

Hooray, let's write a book for 10 people. That makes sense.