Mikan Engine: ScummVM-like HTML5 VN engine to port everything to

[Mad Pierre]

3 hours ago, mnakamura said:

Yeah, you're right, and I'm sorry for that. Actually, I've kind of cheated and just used some external program to do unpacking, let me check which one was that. I think it was either Crass or arc_unpacker. Probably the latter.

UPDATE: Actually it was arc-reader by Alexander Roper.

I'll try to bring up some docs up to date in the days to come. Hopefully it will help understanding what's going on with all that porting stuff.

Thanks, now I've got it to work! Patiently waiting for your design specs.

[mnakamura]

Just got yet another port request basically running. This time it's Kyonyuu Fantasy - running NekoPack / NekoSDK / NEKOSDK_ADVSCRIPT2 engine.

Obligatory screenshot:

Most basic things like backgrounds / sprites / texts / voices / background music work.

Partial English translation is "made by Zyxas. Special thanks to xtenext."

This particular VN is actually quite a number for me, because it's the first one that I'm doing completely blind: I couldn't make it run on any of my Windows desktops, so I just dug in and tried to guess whatever's going on.

As always, feel free to request demos. As far as I understand, Mangagamer's official translation is going on to happen somewhere in 2016, so nobody's interested in translating it into English anymore?

[tymmur]

14 minutes ago, mnakamura said:

As far as I understand, Mangagamer's official translation is going on to happen somewhere in 2016, so nobody's interested in translating it into English anymore?

There is the unwritten rule that work on translation patches as well as download links to (partial) patch downloads will vanish once an official translation has been announced. The goal is to coexist, not to compete.

[mnakamura]

1 hour ago, tymmur said:

There is the unwritten rule that work on translation patches as well as download links to (partial) patch downloads will vanish once an official translation has been announced. The goal is to coexist, not to compete.

I doubt that it is the case here: probably the team that started the translation in 2011 just faded out years ago, it's not about official translation announcement.

In fact, I'm curious if Mangagamer or other official translators would be interested in making use of porting to HTML engine? It could be beneficial to the business, as it dramatically hastens deployment and combats piracy, as it's less trivial to copy a website than to copy a local file(s).

Neko* is actually a pretty poor engine from the technical POV, compared to Ren'Py or krkr2. Any image effects support (blur, colormapping, etc) is basically nonexistent, lots of incompatibilities, traditional problems with script operators written in Japanese (yes, they literally use [立ち絵ロード] command to load a sprite), etc, etc. The main pro of the engine is that it *is* a high-level engine.

[tymmur]

I would however be sorry to see that happening because I really like the ability to play offline as well as with the lag we get from the HD rather than the internet. Looking at how people in general flame the "always online" need in some modern games, going all browser based could very well be a big mistake. They already felt the wrath of people hating DRM and online activation.

 

Still I wouldn't rule out that they would have no interest at all as it would bring VNs to more devices, like iPads and mac/linux. They would however have the added costs of running a server to host such a service, which once again could make it risky for them. There could also be some issues with the VN licenses. They might not be allowed to "stream" the VNs online or port it to a different engine. They also have the cost of support. There are so many factors to consider other than how well the engine works that makes it tricky to figure out if MG is interested. My guess is that they will not be interested. Too risky from a business perspective.

[Mad Pierre]

4 hours ago, tymmur said:

There is the unwritten rule that work on translation patches as well as download links to (partial) patch downloads will vanish once an official translation has been announced. The goal is to coexist, not to compete.

Actually, in this particular case they're alive and kicking:

http://www.hongfire.com/forum/forum/hentai-lair/hentai-game-discussion/game-patch-hentai/402509-waffle-091023-111216-巨乳ファンタジー-kyonyuu-fantasy-english-patch/page6

http://www.hongfire.com/forum/forum/hentai-lair/hentai-game-discussion/game-patch-hentai/418893-waffle-091023-111216-巨乳ファンタジー-kyonyuu-fantasy-english-patch-kuroibaka

The fallen flag was picked up by KuroiBaka, latest news suggest that he's still on it, as of May, 2016.

They even have some tools published: https://github.com/xtenext/nekoKF

[Mad Pierre]

2 hours ago, tymmur said:

Still I wouldn't rule out that they would have no interest at all as it would bring VNs to more devices, like iPads and mac/linux. They would however have the added costs of running a server to host such a service, which once again could make it risky for them. There could also be some issues with the VN licenses. They might not be allowed to "stream" the VNs online or port it to a different engine. They also have the cost of support. There are so many factors to consider other than how well the engine works that makes it tricky to figure out if MG is interested. My guess is that they will not be interested. Too risky from a business perspective.

Well, if I was in their shoes, I'd actually embraced it ASAP:

• server costs are negligible nowadays; CDNs like Cloudflare basically offer free traffic for static content
• additional platforms are a huge plus, especially mobile ones
• support costs — I'd say that they're spending fortunes now to do support of dozens of different engines hardly written for English language in the first place; won't turning everything to one straightforward engine would drastically cut the costs of support and translation?

I don't think that licensing terms should care about platform. After all, they are licensing artistic content: it's up to Western publisher to decide proper ways to deliver it to the clients. Consider a regular paper book publisher licensing a manga translation: won't it be weird if original publisher licensed it to be printed only on one particular brand of paper (the same one produced in Japan, which had to be imported specifically for this project)? That's exactly what a platform is: it's purely technological, non-artistic stuff that just aids in deliverance of content.

[tymmur]

1 minute ago, Mad Pierre said:

I don't think that licensing terms should care about platform. After all, they are licensing artistic content: it's up to Western publisher to decide proper ways to deliver it to the clients. Consider a regular paper book publisher licensing a manga translation: won't it be weird if original publisher licensed it to be printed only on one particular brand of paper (the same one produced in Japan, which had to be imported specifically for this project)? That's exactly what a platform is: it's purely technological, non-artistic stuff that just aids in deliverance of content.

You are being rational. Legal stuff doesn't work that way. They might write in the license contract that it has to be the same engine because then it will be the same file format/encryption. There could also be some complains from Japan that the new engine fail to display some effect correctly, hence altering the visual impression.

 

3 minutes ago, Mad Pierre said:

• support costs — I'd say that they're spending fortunes now to do support of dozens of different engines hardly written for English language in the first place; won't turning everything to one straightforward engine would drastically cut the costs of support and translation?

While that is true, it will move to an always online solution, which will cause quite some negative talk. Plenty of people do not want that for multiple reasons, be it ideological or practical reasons.

[Mad Pierre]

3 minutes ago, tymmur said:

You are being rational. Legal stuff doesn't work that way. They might write in the license contract that it has to be the same engine because then it will be the same file format/encryption. There could also be some complains from Japan that the new engine fail to display some effect correctly, hence altering the visual impression.

Sure, that's civil law anyway. I'm an optimist Just because it's possible to make a contract on a very weird set of terms, I don't think it's a right thing to do. Negotiate while you can, I doubt that Japanese people can't be rational.

[mnakamura]

On 05.07.2016 at 7:39 PM, Mad Pierre said:

The fallen flag was picked up by KuroiBaka, latest news suggest that he's still on it, as of May, 2016.

They even have some tools published: https://github.com/xtenext/nekoKF

I've missed that, thanks! I've imported first chapters they've completed translating and looking through their tools now. Actually, I'd love to contact these guys and show them my demo. Any ideas on how to do that? xtenext doesn't seem to have e-mail published on Github.

[Scorp]

55 minutes ago, mnakamura said:

I've missed that, thanks! I've imported first chapters they've completed translating and looking through their tools now. Actually, I'd love to contact these guys and show them my demo. Any ideas on how to do that? xtenext doesn't seem to have e-mail published on Github.

I doubt it is possible to use github without email  Check first lines of any commit and you will see email... https://github.com/xtenext/nekoKF/commit/24fa830e2ef0138d955cca3260ff93ce2823bdb5.patch

On 05.07.2016 at 4:31 PM, mnakamura said:

In fact, I'm curious if Mangagamer or other official translators would be interested in making use of porting to HTML engine? It could be beneficial to the business, as it dramatically hastens deployment and combats piracy, as it's less trivial to copy a website than to copy a local file(s).

Mangagamer sell VNs, so they should work offline. Also they have Doddler, who can recode them any VN on Unity, which also can work online (actually I think they still have web version of Kara no Shoujo 2 trial). So I would say they would not be interested.

[mnakamura]

On 07.07.2016 at 10:24 PM, Scorp said:

I doubt it is possible to use github without email  Check first lines of any commit and you will see email... https://github.com/xtenext/nekoKF/commit/24fa830e2ef0138d955cca3260ff93ce2823bdb5.patch

My bad, how could I forgot about that Thanks!

On 07.07.2016 at 10:24 PM, Scorp said:

Mangagamer sell VNs, so they should work offline. Also they have Doddler, who can recode them any VN on Unity, which also can work online (actually I think they still have web version of Kara no Shoujo 2 trial). So I would say they would not be interested.

I've read that article by Doddler, and, honestly, I don't think that he's all that almighty. I'm sorry, but rationale like "The main reason is, I don’t know Ren’Py. I do know Unity though." sounds like a pretty immature attitude.

Of course, things are somewhat easier for them, given that they (probably) have access to original VN sources, but still, their current technologies make them spend like 4-5 months per port. For comparison, I typically one usually in less than a week (KFHD, for example, took me literally 1 evening) and that includes all the reversing stuff.

And, yeah, IMHO, Unity's "online" versions have *a long* way to go...

[tymmur]

2 hours ago, mnakamura said:

I've read that article by Doddler, and, honestly, I don't think that he's all that almighty. I'm sorry, but rationale like "The main reason is, I don’t know Ren’Py. I do know Unity though." sounds like a pretty immature attitude.

I wouldn't call it immature. If you read Building a Better Higurashi by Doddler, then you will see that it writes about realtime rescaling of graphics to match screen resolution. Unity does this quite well while Ren'py... well Ren'py isn't written to maximize the performance in the GPU. This mean going with Unity will allow a better graphical experience.

 

2 hours ago, mnakamura said:

their current technologies make them spend like 4-5 months per port. For comparison, I typically one usually in less than a week (KFHD, for example, took me literally 1 evening) and that includes all the reversing stuff.

1: how do you know porting itself takes 4-5 months for each port?

2: what is included in the porting time for MG? Doing what you did in a day will not take months. They might, you know, actually test what they made more than "the first 5 lines appears to work fine".

 

You make it sounds like MangaGamer consists of a bunch of people who has no idea what they are doing. Considering they run a business and they have an engine, which seems to perform better than what you can offer, I say your words carry little weight. You might value their engine less than Mikan, but they optimized their engine for the best offline reader experience and in this field they beat your engine. You said your goal is cross platform support and online reading. Their aim is offline reading in windows, though Unity allows them to go cross platform without too much hassle.

 

Don't get me wrong. Mikan is an impressive piece of software. It provides some features the unity based engine doesn't do or does worse. The problem I have with what you say is that Mikan is better. It looks like MG made an engine, which is significantly better than Mikan or most other engines when used as they intend it to be used: offline playing in windows.

[Scorp]

10 hours ago, mnakamura said:

Of course, things are somewhat easier for them, given that they (probably) have access to original VN sources, but still, their current technologies make them spend like 4-5 months per port. For comparison, I typically one usually in less than a week (KFHD, for example, took me literally 1 evening) and that includes all the reversing stuff.

Lets see Cat2 Grisaia Vita port. Doddler recreated 100% same experience in  his port. If you look into Cat2 sources (that's free engine), it have sophisticated layers/image transition effects and to have it recreated properly it takes lot of time (and lot of memory, I doubt HTML5 would be able to do it, but I am not a web pro, could be wrong). If we speak about simple game - this would take not more than a week on any engine. And his approach is not like your approach - he does exact ports, so whole initial script remains, he reimplement needed functions into his engine, to get 100% same experience for users. Mikan does it exactly opposite - converts initial script to Mikan format (so I believe some effects could look a bit different). And no, he does not have code source in lot of cases (at least in case of Cartagra or Kara no Shoujo 2), he have only script, so do not think he have huge advantage. I believe japanese guys do not like to share code sources, prefer to build all in-house, so in some worst cases Doddler have to build these ports.

Anyway, why not ask @Doddler himself?

[mnakamura]

First of all, I'd like to apologize if I sounded rude anywhere or offended someone. I didn't mean to.

On 09.07.2016 at 10:51 AM, tymmur said:

I wouldn't call it immature. If you read Building a Better Higurashi by Doddler, then you will see that it writes about realtime rescaling of graphics to match screen resolution. Unity does this quite well while Ren'py... well Ren'py isn't written to maximize the performance in the GPU. This mean going with Unity will allow a better graphical experience.

I'm sorry, but (a) you're citing May, 2015 article, but I'm citing June, 2014 article, (b) context or outcomes do not actually matter here.

If a person chooses a tool blindly, just because he/she knows a tool - that *is* immature. If it turns out that he/she was right afterall - that's great, but that's luck, not a well-weighted decision.

On 09.07.2016 at 10:51 AM, tymmur said:

1: how do you know porting itself takes 4-5 months for each port?

Quoting the article: "So that’s how I did it, the whole port done in C#/Unity, over the course of about 4-5 months."

On 09.07.2016 at 10:51 AM, tymmur said:

You make it sounds like MangaGamer consists of a bunch of people who has no idea what they are doing. Considering they run a business and they have an engine, which seems to perform better than what you can offer, I say your words carry little weight. You might value their engine less than Mikan, but they optimized their engine for the best offline reader experience and in this field they beat your engine. You said your goal is cross platform support and online reading. Their aim is offline reading in windows, though Unity allows them to go cross platform without too much hassle.

Please don't attribute me things that I haven't said and make me look like I'm an evil person. I haven't ever said anything bad about MangaGamer, much less calling them "a bunch of people who has no idea what they are doing" or anything like that. I fully respect them and recognize their role in current upbringing of VNs in the West. They really do their best and I'm in no way competing with their business. All the words about "X is performing better than Y" without measurable proofs carry little weight. You're completely right about different aims: they do business (and consequently do huge PR work), I do not. They earn money (thus they're professionals), I don't (thus I'm not).

There are quite a few other aspects beside technical merits of Unity or not. For example, chances are that using Unity makes it much easier to pass all the required tight certifications for proprietary gaming platforms such as PS / XBox / PSVita / Wii / whatever. Or it's easier / cheaper to hire additional help (thus scaling the team) for Unity vs other competitive solutions.

You're also completely right about the sizes of the games and amount of testing.

Actually, it makes an interesting challenge for me: to take the same Really? Really! that is actually the object of 2014 article, port it to Mikan and measure the time. That's probably the only way to get hard results.

[mnakamura]

On 09.07.2016 at 6:30 PM, Scorp said:

Lets see Cat2 Grisaia Vita port. Doddler recreated 100% same experience in  his port. If you look into Cat2 sources (that's free engine), it have sophisticated layers/image transition effects and to have it recreated properly it takes lot of time (and lot of memory, I doubt HTML5 would be able to do it, but I am not a web pro, could be wrong). If we speak about simple game - this would take not more than a week on any engine. And his approach is not like your approach - he does exact ports, so whole initial script remains, he reimplement needed functions into his engine, to get 100% same experience for users. Mikan does it exactly opposite - converts initial script to Mikan format (so I believe some effects could look a bit different). And no, he does not have code source in lot of cases (at least in case of Cartagra or Kara no Shoujo 2), he have only script, so do not think he have huge advantage. I believe japanese guys do not like to share code sources, prefer to build all in-house, so in some worst cases Doddler have to build these ports.

I'm not sure that Doddler aims to "recreate 100% same experience". A quote from the 2014 article: "In the original, a shattering sound played and the screen dropped away piece by piece, but I found it very difficult to pull off that effect in Unity as I would need to do pixel-by-pixel manipulation of a texture (something that’s really slow to do). So instead, I came up with my own effect. I used a unity asset store script “Ultimate Fracturing and Destruction” to create an object that I could apply a snapshot of the screen to, and then used the built in Unity physics to handle it falling away."

Unity is somewhat high-level and obviously has its pros and cons. Actually, it's pretty similar to HTML + WebGL in that aspect (save for the fact that Unity obvious has *tons* of ready-made effects / assets / scripts / etc already implemented). It would be hard to achieve some effects, so substitutes some naturally. Some of them might actually look better, but it's still not "100% same".

I haven't yet reverse-engineered any of the Doddler's ports to check out whatever's inside, but it doesn't look like that he's "keeping the original script", as you mentioned. 2015 article mentions that he actually does some compilation into a bytecode and conversions of the original script.

[Scorp]

34 minutes ago, mnakamura said:

I haven't yet reverse-engineered any of the Doddler's ports to check out whatever's inside, but it doesn't look like that he's "keeping the original script", as you mentioned. 2015 article mentions that he actually does some compilation into a bytecode and conversions of the original script.

Believe me then, he keeps it. He have original script in plaintext, as least for Cartagra, Kara no Shoujo 2, Free Friends 1-2, and Really Really.

Anyway you can see it yourself, I have a tool to extract data from containers. Unity could be decompiled with any Unity decompiler of your choice (like dotPeek or ILSpy), so no problem to see, what happens there, if you're interested.

 

[tymmur]

1 hour ago, mnakamura said:

If a person chooses a tool blindly, just because he/she knows a tool - that *is* immature.

It's commercial production. Learning a new engine and hence new "coding language" because it might be better involves the risk of completely wasting the time (hence money because he is paid) because it turned out that Unity would be the best choice. A free tool isn't free if you have to pay a worker to learn how to use it. Going with a tool, which requires no training of your staff is a valid approach. Another risk assessment they would have to do would be: what is the risk that we need something the engine can't provide? Unity allows coding it while Ren'py could be more problematic if that happened.

 

In other words: Ren'py was likely dropped due to financial risk assessment rather than somebody being immature and too lazy to study a new engine.

 

2 hours ago, mnakamura said:

Quoting the article: "So that’s how I did it, the whole port done in C#/Unity, over the course of about 4-5 months."

Sounds like it is building the engine and then port, which mean it isn't the timeframe for each port.

2 hours ago, mnakamura said:

Please don't attribute me things that I haven't said and make me look like I'm an evil person.

You said stuff, which hinted in that direction, which is what I'm reacting to. However the word evil wasn't on my mind, in fact it would be the opposite if anything. I sense the issue is more like ignorance of how commercial code production works. It's very different from how people work for free on open source projects where moving in the wrong direction is just wasted time, but without financial consequences.

[mnakamura]

2 hours ago, Scorp said:

Believe me then, he keeps it. He have original script in plaintext, as least for Cartagra, Kara no Shoujo 2, Free Friends 1-2, and Really Really.

Anyway you can see it yourself, I have a tool to extract data from containers. Unity could be decompiled with any Unity decompiler of your choice (like dotPeek or ILSpy), so no problem to see, what happens there, if you're interested.

Wow, what else can I say? Thanks!

I feel really lucky that I've met you. I admire your skills, hands down. You always seem to have answers for everything I just keep wondering if there's anything you haven't tried reversing yet?

[mnakamura]

1 hour ago, tymmur said:

It's commercial production. Learning a new engine and hence new "coding language" because it might be better involves the risk of completely wasting the time (hence money because he is paid) because it turned out that Unity would be the best choice. A free tool isn't free if you have to pay a worker to learn how to use it. Going with a tool, which requires no training of your staff is a valid approach. Another risk assessment they would have to do would be: what is the risk that we need something the engine can't provide? Unity allows coding it while Ren'py could be more problematic if that happened.

In other words: Ren'py was likely dropped due to financial risk assessment rather than somebody being immature and too lazy to study a new engine.

You know, in my day life, I'm a team leader in an infosec division of a large Japanese corporation. I do know a thing or two about time management, risk evaluation, delivering the results, fulfilling the promises and meeting the deadlines. If we're choosing the tool for a new, not-so-well-known area of work, it's only natural to spend some time to do preliminary research. It could take a week or two, but ultimately you would be choosing a tool X because it "performs better", "has larger userbase (thus we can just hire them and save on the training)", "has best SLA", "can be super-quick for prototyping purposes", etc, etc, and *not* because "I only know X and screw everything else". I've already been through multiple situation when this time spent on research paid off well.

Anyway, I guess it's kind of pointless and off-the-topic to discuss who's more "mature" or whatever. The best thing I've got from this conversation is probably a new challenge for me: to try doing "Really? Really!" port for Mikan

[Scorp]

12 minutes ago, mnakamura said:

Anyway, I guess it's kind of pointless and off-the-topic to discuss who's more "mature" or whatever. The best thing I've got from this conversation is probably a new challenge for me: to try doing "Really? Really!" port for Mikan

Not better to finalize porting AI5 (Elf/Silky's engine) first?

 

32 minutes ago, mnakamura said:

Wow, what else can I say? Thanks!

I feel really lucky that I've met you. I admire your skills, hands down. You always seem to have answers for everything I just keep wondering if there's anything you haven't tried reversing yet?

I reversed only things I was interested in. Also some I did not finished, mostly the ones which required recoding stuff done on PSP or Saturn (means translation re-implementation). I am actually not that knowledgeable, I lack fundamental knowledge about reverse-engineering (like I cannot easily do a pseudocode reverse, put correct structures, guess the arguments and in the end get readable code) and it requires too much time for me to learn that.

[mnakamura]

Just now, Scorp said:

Not better to finalize porting AI5 (Elf/Silky's engine) first?

Of course, that comes first. I still have a solid backlog of things I've already promised, so probably I won't even touch Doddler's engine in a month or so.

Just now, Scorp said:

I reversed only things I was interested in. Also some I did not finished, mostly the ones which required recoding stuff done on PSP or Saturn. I am actually not that knowledgeable, I lack fundamental knowledge about reverse-engineering (like I cannot easily do a pseudocode reverse, put correct structures, guess the arguments and in the end get readable code) and it requires too much time for me to learn that.

You're really modest Still, you seem to have at least touched virtually everything I've tried so far And still I haven't tackled basically anything somewhat exotic (like, for PS games I've only barely touched 2 titles: Gran Turismo and Subete ga F ni naru). From what I've seen there, it's all pretty complex - i.e. custom sound formats, custom image formats, weird multi-layered textures + embedded 3D models for even the simplest stuff like overlaying the eyes over face sprite, etc, etc.

[Scorp]

@mnakamura speaking about Subete ga F ni naru it looks like they used same engine as in Ever17 PC, all is packed with usual lzss compression, images most likely use same format, so all could be extracted I think. At least when I checked that it was looking pretty understandable. And I am not sure that they use 3d models there, albeit did not digged much.

[tymmur]

While it's interesting to add support for more and more engines, what about improving support for the already supported engines? For instance make buttons and similar, which feels like the original VNs. Once Mikan gives a decent playing experience, which isn't degraded relative to the original VN, then it would make sense to add support for more engines.

[mnakamura]

1 minute ago, Scorp said:

@mnakamura speaking about Subete ga F ni naru it looks like they used same engine as in Ever17 PC, all is packed with usual lzss compression, images most likely use same format, so all could be extracted I think. At least when I checked that it was looking pretty understandable. And I am not sure that they use 3d models there, albeit did not digged much.

Well, here you go again - another title named just of the the blue - and you've already tried and checked it

To be honest, I've really just wanted to take a look at it without purchasing a real PSP while the series were on air, but as the series concluded (IMHO, in a very boring way), I kind of lost interest to even trying that.