guide Reverse engineering visual novels: the basics

[mnakamura]

Hi, fellow VN explorers,

I've just published an article on basic reverse engineering of unknown file formats of visual novels.

It requires little technical knowledge (actually, everything except for the optional latest part can be done without programming) and it tackles the basic process of discovering information about simple container format and exploring its structure to extract files from a game archive.

Hope it would be useful Feel free to ask any questions.

[Scorp]

11 hours ago, mnakamura said:

I've just published an article on basic reverse engineering of unknown file formats of visual novels.

That's just a translation of your old article to english? Or something is different.

Quote

If you’re doing this in plain Ruby (or Python, or PHP, or whatever), at the very least you’ll have to do the whole thing twice: first, you write a script that outputs dumps to the screen, then you rewrite it to actually extract the data. If you’re employing “advanced” hex editors like Hexinator or 010 Editor — chances are you’d actually do this work thrice (yet another time you’ll have to write a template in your editor).

Really? I always write an extraction script from the beginning, after thinking a bit about structure in hex viewer. Why would anyone waste time on mindless hex data dumping on the screen, when they already saw that?

[mnakamura]

2 hours ago, Scorp said:

That's just a translation of your old article to english? Or something is different.

98% is just a translation.

2 hours ago, Scorp said:

Really? I always write an extraction script from the beginning, after thinking a bit about structure in hex viewer. Why would anyone waste time on mindless hex data dumping on the screen, when they already saw that?

I've just described my way. If you're able to do more than me just by applying your brain — congrats, you're probably smarter than me. I'm not really comfortable with doing lots of repetitive hex arithmetics in my head or applying zlib decompression / XOR / bit shifts / rotations / etc just by looking at the hex dump.

[Scorp]

6 hours ago, mnakamura said:

98% is just a translation.

I've just described my way. If you're able to do more than me just by applying your brain — congrats, you're probably smarter than me. I'm not really comfortable with doing lots of repetitive hex arithmetics in my head or applying zlib decompression / XOR / bit shifts / rotations / etc just by looking at the hex dump.

Any hex editor could help you with testing all these xor/bit shifts/rotations as well. Also you will do these arithmetic in same extract script you writing and you can dump results in it, not making new script.

[mnakamura]

5 minutes ago, Scorp said:

Any hex editor could help you with testing all these xor/bit shifts/rotations as well.

Ok, could you please name the hex editor that could show me at least zlib, AES, 3DES substreams (preferably whole lot more, like RLE, LZ/LZW/LZSS, etc)?

[Scorp]

16 minutes ago, mnakamura said:

Ok, could you please name the hex editor that could show me at least zlib, AES, 3DES substreams (preferably whole lot more, like RLE, LZ/LZW/LZSS, etc)?

You know it yourself. 010 editor with templates. But why do these zlib extractions there? You do them in your script, like I do. That's enough to understand, that this is zlib or AES, not actually required to apply them on data in hex editor to do a code.

[mnakamura]

1 hour ago, Scorp said:

You know it yourself. 010 editor. But why do these zlib extractions there? Do them in your script, like I do.

Maybe I don't know something, but last time I checked 010 had no concept of substreams at all, let alone applying common algorithms to it.

It's kind of getting us nowhere. You keep sticking to your approach, that's fine with me by all means.

[Scorp]

3 hours ago, mnakamura said:

Maybe I don't know something, but last time I checked 010 had no concept of substreams at all, let alone applying common algorithms to it.

It's kind of getting us nowhere. You keep sticking to your approach, that's fine with me by all means.

Yes, I just told that your sentence about "you have to write 3 times same thing to make an extract" is an exaggeration, that's all.

[sanahtlig]

On 7/14/2016 at 6:17 AM, mnakamura said:

Hi, fellow VN explorers,

I've just published an article on basic reverse engineering of unknown file formats of visual novels.

It requires little technical knowledge (actually, everything except for the optional latest part can be done without programming) and it tackles the basic process of discovering information about simple container format and exploring its structure to extract files from a game archive.

Hope it would be useful Feel free to ask any questions.

It could be helpful to explicitly describe your target audience near the beginning and the significance your article will have for them (and by extension, the community at large).  Given that you've targeted this to the "Fan translation" forums, I think you have a very specific audience in mind.  This was not at all clear when you posted it on Reddit.

[mnakamura]

On 15.07.2016 at 10:30 PM, sanahtlig said:

It could be helpful to explicitly describe your target audience near the beginning and the significance your article will have for them (and by extension, the community at large).  Given that you've targeted this to the "Fan translation" forums, I think you have a very specific audience in mind.  This was not at all clear when you posted it on Reddit.

Thanks for your suggestion! You see, I'm a very beginner writer (actually, it's my first article I've ever written for wide public), and I'm by no means a native English speaker. Can you recommend me what would be a better way to introduce such an article to translators / reverse-engineers-wanna-bes?

[sanahtlig]

3 hours ago, mnakamura said:

Thanks for your suggestion! You see, I'm a very beginner writer (actually, it's my first article I've ever written for wide public), and I'm by no means a native English speaker. Can you recommend me what would be a better way to introduce such an article to translators / reverse-engineers-wanna-bes?

Organization of ideas crosses language boundaries.  Not being a native speaker will not impede you.

For my solution to a similar problem, take a look at my Windows 10 Visual Novel Compatibility Guide.  Notice how I explicitly outline an existing problem and then my approach to solve it.  This is a standard writing approach that should be useful in any sort of technical writing.  I also make a habit of summarizing the article at the start so readers know what to expect and can see at a glance whether it'll be relevant to them before reading further.

In science, the significance of any proposal outweighs all other factors.  That's what inspired my current approach to technical writing.  The quality of your content won't matter if you can't convince readers it is relevant to them.

[Scorp]

@sanahtlig only specific audience would be interested in an article with title "Reverse engineering visual novels: the basics", so I do not quite understand why he should outline or convince anyone, when all is told in the title. I doubt anyone would even read it, if he is not interested in reverse engineering.

[sanahtlig]

1 hour ago, Scorp said:

@sanahtlig only specific audience would be interested in an article with title "Reverse engineering visual novels: the basics", so I do not quite understand why he should outline or convince anyone, when all is told in the title. I doubt anyone would even read it, if he is not interested in reverse engineering.

If your audience is merely those who already know about your subject and its significance, then you're probably underselling your message.  That's what I learned from my internship in science.

[Scorp]

10 minutes ago, sanahtlig said:

If your audience is merely those who already know about your subject and its significance, then you're probably underselling your message.  That's what I learned from my internship in science.

I really doubt you can sell anything just with catchy tagline, if person is not interested. At least if person do not understand what means "Reverse engineering" and will walk away.

[DrDaxxy]

Articles like this aren't gonna be anywhere near as useful to people who already have reverse engineering skills as people who'd like to get into it.

Back when I started, I'd definitely have appreciated more resources like this.

However, someone who wants to learn how to "hack VNs" (or "mod games" or "view .something files") might not know that reverse engineering is what that's called.

[Kelebek1]

Is it just me that found the article confusing as fuck starting half way down? And I already do a lot of this stuff.

[Scorp]

9 hours ago, Kelebek1 said:

Is it just me that found the article confusing as fuck starting half way down? And I already do a lot of this stuff.

That's where that Kaitai Script comes into action?

[Forgetful Frank]

Trying to read that, I began to foam at the mouth whilst having a seizure, it almost felt as if I was looking into the Matrix itself.

But great work on writing such an awesome technical list allowing others to learn how to do some cool stuff in extracting files and whatnot from VN's. I'm sure it will help a lot of future FTL projects in the future.