Mikan Engine: ScummVM-like HTML5 VN engine to port everything to

[mnakamura]

7 hours ago, Scorp said:

I can, but it do not have much meaning, as there is lot of real VM implementations around. For example this one: https://github.com/vnvm/vnvm So you can use it to inspect format and create your own implementation in .story.

Thanks again! I had no idea that such a project existed. By any chance, could you enlighten me if you know any other such projects (i.e. custom VMs for 3rd party novels)?

[Scorp]

24 minutes ago, mnakamura said:

Thanks again! I had no idea that such a project existed. By any chance, could you enlighten me if you know any other such projects (i.e. custom VMs for 3rd party novels)?

RLVM, Anise, ViLE (already gave you link in previous messages), ONScripter (nScripter seems was open-source from beginning, but it is sort of evolution for modern machines). Probably I know more, just cannot remember right now.

Another thing is that some VNs have plain text script (like Doddler remakes he made for MG or like C's Ware Eve Burst Error PC-98 Windows version), sort of what I made for Divi-Dead.

/btw you can just press "Like this" button for each message was useful, you know.

[mnakamura]

16 hours ago, Scorp said:

RLVM, Anise, ViLE (already gave you link in previous messages), ONScripter (nScripter seems was open-source from beginning, but it is sort of evolution for modern machines). Probably I know more, just cannot remember right now.

Another thing is that some VNs have plain text script (like Doddler remakes he made for MG or like C's Ware Eve Burst Error PC-98 Windows version), sort of what I made for Divi-Dead.

/btw you can just press "Like this" button for each message was useful, you know.

I had no idea on RLVM and Anise. Hmm, now I'm getting it where Asenheim project took their supported titles

[Scorp]

1 hour ago, mnakamura said:

I had no idea on RLVM and Anise. Hmm, now I'm getting it where Asenheim project took their supported titles

Actually no, if you take a look at list of supported titles - Asenheim titles do not cross that much with RLVM or Anise.

[Mad Pierre]

On 26.06.2016 at 1:20 PM, Scorp said:

Actually no, if you take a look at list of supported titles - Asenheim titles do not cross that much with RLVM or Anise.

But Asenheim's list seems to be pretty close to ViLE - the same set of older C's ware engine titles, the same Nocturnal Illusion, Divi-Dead, etc?

Do you know who used who's work? Who was the first reverse engineer of these games: Zus, ViLE guys, or someone long before that?

[Scorp]

1 hour ago, Mad Pierre said:

But Asenheim's list seems to be pretty close to ViLE - the same set of older C's ware engine titles, the same Nocturnal Illusion, Divi-Dead, etc?

Do you know who used who's work? Who was the first reverse engineer of these games: Zus, ViLE guys, or someone long before that?

BaSF (one of the ViLE author) is also among the asenheim staff. So asenheim could be considered as ViLE 2.0, lets say so. Zus coded web engine and integrated them, I think. Anyway, why not ask Zus? He does not bite, quite a good guy, btw.

Who was first... I suppose BaSF reverse-engineered Nocturnal Illusion first, as he is the author of that infamous patch for Nocturnal Illusion (Jast release) which restores all ripped-out content with blackened scenes.

[mnakamura]

4 hours ago, Scorp said:

BaSF (one of the ViLE author) is also among the asenheim staff. So asenheim could be considered as ViLE 2.0, lets say so. Zus coded web engine and integrated them, I think. Anyway, why not ask Zus? He does not bite, quite a good guy, btw.

Actually, I've tried to contact both ViLE team and Zus multiple times — and I haven't got any answer. Zero activity on ViLE since 2012, lots of unanswered questions on ViLE blogs / comments, and several other people reporting failure to contact suggest that BaSF is no longer active in the community. By the way, someone still pays for vilevn.org domain - and that is probably "Vegard Fiksdal", as per whois info.

Zus seems to be alive and kicking (at least he answers questions in a dialog box on tss.asenheim.org), but seems to ignore me. Hey, @Scorp, if you have a solid method to contact him, could you relay a message from me? Or at least confirm that he does not want to have any discussion with me?

[Scorp]

3 minutes ago, mnakamura said:

Zus seems to be alive and kicking (at least he answers questions in a dialog box on tss.asenheim.org), but seems to ignore me. Hey, @Scorp, if you have a solid method to contact him, could you relay a message from me? Or at least confirm that he does not want to have any discussion with me?

What's the problem to contact via email? Do not work for you?

[mnakamura]

12 hours ago, Scorp said:

What's the problem to contact via email? Do not work for you?

As I said, the problem is that he ignores my mails. I don't know if there's a transmission problem, he's busy, or he doesn't want to reply.

[Scorp]

2 hours ago, mnakamura said:

As I said, the problem is that he ignores my mails. I don't know if there's a transmission problem, he's busy, or he doesn't want to reply.

His email works fine... Okay, I will ask him and will give him link to this thread.

So, he replied and told, that he used BaSF algorithms for Divi-Dead and probably for Nocturnal Illusion (of course BaSF gave his permission for this). BaSF is not part of Asenheim staff, all other games were reversed and mantained by Zus. Speaking about "list seems to be pretty close to ViLE" - most likely that's just a coincidence, because there is quite a few classic eroge with english translation. And no, he never got your emails. If you still want to speak with him - you can pass me your email in PM, I will send to him, he will try to contact you. Zus is friendly and cooperative guy, as I already told. That's weird you were not able to get any response from him.

Seems all is clear now?

[mnakamura]

In actual news, I've finished porting demo of Aiyoku no Eustia today - demo quality (lacks some fancy animations), but playable altogether. It uses newer BGI engine.

Obligatory screenshot:

Ports and engine-specific tools are published at:

• https://github.com/mnakamura1337/engine_bgi - engine-specific tools
• https://github.com/mnakamura1337/port_eustia - Eustia-specific scripts, metadata, etc (basically, they call "engine_bgi" stuff with proper arguments, that's all)

More documentation will probably follow. If anyone wants to take a look without doing all the conversion with scripts on your box - PM me, as usual.

[Mad Pierre]

Sugoooi! Looks really cool. Downloading trial now to take a try.

What's your general idea on repository / code structure for porting? From what you've published, I reckon something like that:

• engine_* - engine-specific tools, one repository per engine, written mostly in Ruby (is it obligatory? I'd rather use Python)
• *2story - ??? - seems to be also engine-specific tools
• port_* - game-specific tools, one repository per game

Your scripts seem to be written in bash, so they are Linux-only. Are you ok with that? I guess Window userbase is also strong, most Windows users would want a straightforward .exe converter, like ones that were provided by VNDS.

[Mad Pierre]

On 25.06.2016 at 6:47 PM, mnakamura said:

Thanks again! I had no idea that such a project existed. By any chance, could you enlighten me if you know any other such projects (i.e. custom VMs for 3rd party novels)?

Just stumbled upon yet another VN remake project - http://www.nonakap.org/drs/ recreates Kazoku Keikaku - https://vndb.org/v155

[mnakamura]

16 hours ago, Mad Pierre said:

What's your general idea on repository / code structure for porting? From what you've published, I reckon something like that:

• engine_* - engine-specific tools, one repository per engine, written mostly in Ruby (is it obligatory? I'd rather use Python)
• *2story - ??? - seems to be also engine-specific tools
• port_* - game-specific tools, one repository per game

Your scripts seem to be written in bash, so they are Linux-only. Are you ok with that? I guess Window userbase is also strong, most Windows users would want a straightforward .exe converter, like ones that were provided by VNDS.

Yeah, you're hit the point with structure. The idea is to have one repo per game (port_*) and one repo per engine (engine_*). *2story is obsolete and they will be joined into engine_* repos. Ruby is not obligatory by any means, of course.

As for scripts in bash... Well, do you know of any alternatives?

This whole VN reverse engineering tool business is pretty weird, if you'd ask me. Take a look at tlwiki, for example: it's a very diverse collection of scripts in various languages (most of them are just uploaded as is, so no VCS, no change history, no nothing) and binary tools written in hardcore languages like C or C++ (the sources are frequently missing, sometimes they're GUI only). It's nightmare with licenses and copyrights - nobody seems to care.

And, the ugliest thing - all these tools are meant to be ran manually. Almost no automation or anything like that exists. The closest possible stuff I've seen is VNDS converters - they're written in Java (and include sources) and they're huge UI applications. Just checked in, Never7Converter is ~400KB of sources, ~12K LOC.

In contrast, my noratoto port actually is 287 LOC for the port + 608 LOC for the engine tools < 1K LOC totally.

I guess we can try to cater for Windows users later by doing some sort of "converter shell" that would actually bring shell scripts / Ruby / Python interpreters to Windows box and call them in a pretty GUI. Or, screw that, 99.9% of Windows users would just download a ready-made port from torrents anyway.

16 hours ago, Mad Pierre said:

Just stumbled upon yet another VN remake project - http://www.nonakap.org/drs/ recreates Kazoku Keikaku - https://vndb.org/v155

Yeah, thanks for the heads up! I'll put that into Mikan engine README soon.

[Scorp]

7 minutes ago, mnakamura said:

Take a look at tlwiki, for example

Apache2 Debian Default Page

 

It works!

[mnakamura]

Wow, ok, https://tlwiki.org/. It seems that they have a whole fanscinating world of web servers configs to learn before them, I guess.

[Scorp]

@mnakamura Speaking about N7/E17/etc converters - I never saw so much complex arithmetics in simple eroge engine just to add/compare a flag  It looked like they wanted to do a TBS, not an eroge.

[mnakamura]

1 minute ago, Scorp said:

@mnakamura Speaking about N7/E17/etc converters - I never saw so much complex arithmetics in simple eroge engine just to add/compare a flag

BGI is pretty weird too, especially modern BGI with their subroutines. They have like 5 or 6 ways to just change a background, for heaven's sake... And I'm not talking about any fancy transitions, or anything like that.

Another fun fact: their compiler always compiles negative literal numbers like -42 as "push 42; push -1; mul".

Care to show a bit of N7/E17 arithmetics disassembly?

[Scorp]

24 minutes ago, mnakamura said:

BGI is pretty weird too, especially modern BGI with their subroutines. They have like 5 or 6 ways to just change a background, for heaven's sake... And I'm not talking about any fancy transitions, or anything like that.

Another fun fact: their compiler always compiles negative literal numbers like -42 as "push 42; push -1; mul".

Care to show a bit of N7/E17 arithmetics disassembly?

Unfortunately I do not have any right now, I remember when I checked Cartagra remake for PS2, there were insanely large arithmetic subroutines, covering almost every math operation possible, xor/or/and operations, including ( ) priority and stuff... And all this used just to do "Flag A += 1"

[mnakamura]

Just now, Scorp said:

Unfortunately I do not have any right now, I remember when I checked Cartagra remake for PS2, there were insanely large subroutines, covering almost everything, xor/or/and operations, including ( ) and stuff... And all this used just to do "Flag A += 1"

I just wonder why people keep writing such stupid compilers... And actually why people keep reinventing the wheel and write so many engines in the first place...

[Scorp]

1 minute ago, mnakamura said:

I just wonder why people keep writing such stupid compilers... And actually why people keep reinventing the wheel and write so many engines in the first place...

Erm... Told the guy, who decided to reinvent the wheel with Mikan engine 

Yes, I also do not understand, why I still see in 2016 releases compression methods from 1989, instead of using built-in libraries like gzip, why someone still uses custom audio compressions or try to obfuscate C# code... Japanese programmers think differently, that's all I can think of.

[mnakamura]

Just now, Scorp said:

Erm... Told the guy, who decided to reinvent the wheel with Mikan engine 

Yes, I also do not understand, why I still see in 2016 releases compression methods from 1989, instead of using built-in libraries like gzip, why someone still uses custom audio compressions or try to obfuscate C# code... Japanese programmers think differently, that's all I can think of.

You know, Mikan is different Prior to inventing .story format, I've seen through various open / semi-open options available (like tjs format, VNDS format, ren'py format) - they are either too primitive to convey modern games (with animations, layers and stuff) properly, or they are too generic (i.e. they are actually a proper Turing-complete programming language - and thus programmers are tempted to invent yet-another-engine-inside-engine with them. That's actually an architectural error: they fail to properly differentiate "presentation" vs "business logic" vs "actual story data".

Obfuscation and stuff like that is, unfortunately, blatant ignorance either by programmers or (more frequently) by publishers who still think that publishing a offline DVD in 2016 with some "copy / extraction protection" will stop their game from being pirated and/or reversed.

[Scorp]

55 minutes ago, mnakamura said:

You know, Mikan is different Prior to inventing .story format, I've seen through various open / semi-open options available (like tjs format, VNDS format, ren'py format) - they are either too primitive to convey modern games (with animations, layers and stuff) properly, or they are too generic (i.e. they are actually a proper Turing-complete programming language - and thus programmers are tempted to invent yet-another-engine-inside-engine with them. That's actually an architectural error: they fail to properly differentiate "presentation" vs "business logic" vs "actual story data".

Obfuscation and stuff like that is, unfortunately, blatant ignorance either by programmers or (more frequently) by publishers who still think that publishing a offline DVD in 2016 with some "copy / extraction protection" will stop their game from being pirated and/or reversed.

Lets say, that I saw lot of engines, and yes, every engine have its own restrictions. The most complex of them all is krkr2, of course, as it allows you to extend/program/recode almost everything, so you can do even a 3d shooter on it, I believe (but would be slow, of course. And stupid). But in terms of development this actually makes development times much less rather than doing it from scratch.

Anyway, lets see how it will progress. Hope you would have passion to develop Mikan at least for couple of years.

[Mad Pierre]

5 hours ago, mnakamura said:

Yeah, you're hit the point with structure. The idea is to have one repo per game (port_*) and one repo per engine (engine_*). *2story is obsolete and they will be joined into engine_* repos. Ruby is not obligatory by any means, of course.

Well, I've managed to build like 80% of Eustia, but it really took me a while to figure out:

• The initial unpacking part is missing. One's on your own to unpack BGI arc files. I've managed to do it using "Ethornell" module of ExtractData.
• Image conversion part is missing. You have generated pngs somehow, I'm still at loss how to do that.
• Whole extracted.d / extracted.s stuff is very confusing - I have no idea which is which and there's no documentation.
• BGI2STORY works, but it requires one to manually compile your .ksy format description files first
  • ... and now that's a very untrivial matter, as it means that one has to find Kaitai Struct project, install it, and figure out that you need to run "ksc -t ruby file.ksy" to compile .ksy => .rb

Noratoto port is much better documented, and, thanks to that, I've managed to get through everything, besides .png conversion. Thanks for your impressive work, but it's still *way* beyond being user-friendly or anything.

[mnakamura]

On 01.07.2016 at 0:49 AM, Mad Pierre said:

Well, I've managed to build like 80% of Eustia, but it really took me a while to figure out:

• The initial unpacking part is missing. One's on your own to unpack BGI arc files. I've managed to do it using "Ethornell" module of ExtractData.
• Image conversion part is missing. You have generated pngs somehow, I'm still at loss how to do that.
• Whole extracted.d / extracted.s stuff is very confusing - I have no idea which is which and there's no documentation.
• BGI2STORY works, but it requires one to manually compile your .ksy format description files first
  • ... and now that's a very untrivial matter, as it means that one has to find Kaitai Struct project, install it, and figure out that you need to run "ksc -t ruby file.ksy" to compile .ksy => .rb

Noratoto port is much better documented, and, thanks to that, I've managed to get through everything, besides .png conversion. Thanks for your impressive work, but it's still *way* beyond being user-friendly or anything.

Yeah, you're right, and I'm sorry for that. Actually, I've kind of cheated and just used some external program to do unpacking, let me check which one was that. I think it was either Crass or arc_unpacker. Probably the latter.

UPDATE: Actually it was arc-reader by Alexander Roper.

I'll try to bring up some docs up to date in the days to come. Hopefully it will help understanding what's going on with all that porting stuff.