Data extraction thread

[Nikdo]

3 hours ago, Nanashi3 said:

AFAICT it is an output parameter that is used to for the --repack corresponding command

It produces such a file

<archive>
  <header entriesCount="602" dataOffset="0x2f10">
    <undecodedData>some base64 stuff</undecodedData>
  </header>
  <entries entriesCount="602">
    <entry unknown0="0x1" offset="0xea0d0" size="0x940" unknown1="0x250dc57a"/>
    ...
  </entries>
  <files entriesCount="602">
    <file offset="0x2f10" size="0x1f40" filename="file_0000.tm2" type="TM2"/>
    ...
  </files>
</archive>

 

Haha, I know what XML in general is. The problem is that there is no such file or information there.

Maybe it is because I have the PS2 version. Did you encounter the same problem when using this tool?

 

EDIT: Yeah, I get it now. The xml file is generated, not required. Now, I have to figure out what is it with the tm2 files and why have them all exactly same size... :/

EDIT: Yeah, I've got it complete. Thank you very much!!!

Edited February 14, 2019 by Nikdo

[Hackrabbits]

I need some immediate help. I looked through this thread and the Script Extraction thread, and it doesn't really help much at all.

The scripts seem to be in the .dat files. Are there any recommendations on extracting as well as encoding the scripts so they are editable .txt files?

ExtractDatav1.20 doesn't work out for me because it's so confusing. GARbro-v1.5.42.2842 cannot open the archives.

Edited February 18, 2019 by Hackrabbits

[Zakamutt]

Nobody cares about this I assume but I figure I'll write about extracting / reinserting ( .pd files ) for Umi Kara Kuru Mono / 海からくるもの ,

an old denpaらしい vn none of you have heard of.

It seems to use the same formats as Cross Channel, so insani's cross channel tools work for it: http://www.insani.org/tools/

They need python and insani.py in the same directory to run, I think I have v2.7 python on my path which probably helped. crosschannel-extract3.py extracts with argument 1 being filename and 2 being output directory, crosschannel-repack1.py repacks with a folder containing the resources as its first argument and filename as its second.

The script files seem flexible; the files come out with a lot of blank lines but they work fine with those removed (only tested on first scenario file). Font is monospace; linebreaks can be added with the enter key. Edit with notepad++

Not sure if removing blank lines will fuck up saving/loading, have not tried this yet. probably not but you never know.

[sena_kaito]

Hi, 
How to extract sprites files from Kanojo to Ore to Koibito to.?
Seems using e-mote engine.
Can't extract .dat and .psp files both GARbro and arc_unpacker.
Any solution?

Here sample files.

I'm working on VN extraction.
Thank you :3

[Haiyami]

I'm needing to extract files from Sister scheme HD. They use .nme files, .det files and .atm files. I used gar-bro to open up the object.det file since that was the only file i could open with gar bro, and inside had the string file and scenario file i'm needing to fix standing CG problems that the developers didn't fix in the HD. However the files to extract are .o files. so it would be something like main01.o and I can't further open those files to extract. I need an extraction program and a a repacker program that can deal with those types of files so I can fix the issues with the CG. I'm going to need to change either the scenario file or string file to reference the proper cg. But yeah i don't know how to extract .o files.

 

Any help?

[Ulysses]

On 2019/2/23 at 1:05 PM, sena_kaito said:

Hi, 
How to extract sprites files from Kanojo to Ore to Koibito to.?
Seems using e-mote engine.
Can't extract .dat and .psp files both GARbro and arc_unpacker.
Any solution?

Here sample files.

I'm working on VN extraction.
Thank you :3

E_AYA07_A.dat & E_AYA07_B.dat are WMV video files, open directly via PotPlayer.

C2_EA.PSP is PSB file, however it is either incomplete or encrypted or obfuscated (there seems to be many 0xFF inserted), it's not possible to be loaded in my FreeMote.

If you believe C2_EA.json is JSON file, it could be XORed, too.

Edited February 25, 2019 by Ulysses

[Bathhouse_Owner]

Hello, does anyone here have the System-NNN/Gore Screaming Show tools? They were hosted on TLwiki, and can't find them anywhere else.

[sena_kaito]

On 2/25/2019 at 2:25 PM, Ulysses said:

E_AYA07_A.dat & E_AYA07_B.dat are WMV video files, open directly via PotPlayer.

C2_EA.PSP is PSB file, however it is either incomplete or encrypted or obfuscated (there seems to be many 0xFF inserted), it's not possible to be loaded in my FreeMote.

If you believe C2_EA.json is JSON file, it could be XORed, too.

Solved with latest GARro
Thank you.

[onii]

SiglusEngine issue with g00 files: how do I create "No Data" layers?

I tried using dummy transparent images but it causes a glitch, it may be something to do with the file output format as well but I can't find a documentation for G00PackMax and vaconv-cpp doesn't seem to support multi-layer PSD files.

Now, I have almost completely finished editing the interface and half the scene file, that is the only obstacle I couldn't manage to overcome so far... I made this thread here with more details if you need.

Edited March 5, 2019 by onii
reduced to relevant info

[thrae]

On 24. 5. 2014 at 10:34 AM, maefdomn said:

YU-NO

Download Yuno_arctools.zip and Yuno_gp8tools.zipnot the latest version.

Is there any good soul that still has this "Yuno_arctools" and could reupload it somewhere? The link is not working and I'm not able to google it anywhere else. I'm trying to extract music.arc from this game to turn down BGM volume a bit (because it drowns out voice overs) with no luck for the last two days and I would much appreciate any help.

Edited March 6, 2019 by thrae

[Nanashi3]

12 minutes ago, thrae said:

Is there any good soul that still has this "Yuno_arctools" and could reupload it somewhere? The link is not working and I'm not able to google it anywhere else. I'm trying to extract music.arc from this game to turn down BGM volume a bit (because it drowns out voice overs) with no luck for the last two days and I would much appreciate any help.

You may download the v0.4 here: https://web.archive.org/web/20171201163843/https://tlwiki.org/index.php?title=File:Yuno_arctools.zip

[thrae]

On 6. 3. 2019 at 11:49 PM, Nanashi3 said:

You may download the v0.4 here: https://web.archive.org/web/20171201163843/https://tlwiki.org/index.php?title=File:Yuno_arctools.zip

Thanks mate, you saved me! It didn't cross my mind to try the other versions when I previously found out the archive link to the newest one isn't working. Considering how much hours I've spent googling and trying of different arc tools, this is really emabarassing. Anyway, I just finished editing the files and the game is so much better with background music actually being in the background! Once again thank you!

Edited March 7, 2019 by thrae

[Eco-Friendly]

I'm trying to unpack squeez cgpack by using arc_unpacker , but only thing i get is 0_nopics.gr or something like that. Can anyone help me ?

[Nikdo]

Hi, it's me again. This time I'm trying to unpack Hentai Ouji to Warawanai Neko visual novel (PSP, 2013), again looking for character images.

I have found it for some reason in the INSDIR directory, inside the RES.DAT (it has 47 50 44 41 / GPDA signature). The only program able to extract it was the old good arc_conv-master, so here I've got the script_charactor.dat file, again with GPDA signature.

But there goes the problem.

If you open the script_charactor.dat file in a hex editor, you can see it contains the desired *.gim files, which can be opened with GimConv or Gim Viewer:

But unpacking it again with arc_conv-master only creates a "damaged" files. It has no extension and if you open it with a hex editor, you can see it's got a GPDA signature again and the content is clearly messed up (a lot of zeros after every few lines).

Isn't there a better way to extract those files? Anything I've found on the internet was about emulators... :/

EDIT:

OK, I have found something called "tenoritool" (http://rsk.twilight.free.fr/ps2/home/tenoritool-1.0alpha.7z), which seems to go well with other PSP games, where you need to extract res.dat (like Oreimo or Toradora).

However, when I've tried it for this game, it's showed some kind of error written in my native language (something about "spaces in the string"), so I guess it is more like cmd error than anything, but... What the heck?

EDIT:

OK, everything I said before was wrong.

To use the tenoritool, you must be just very direct with the command, so this one actually worked:

tenoritool.exe --list="" -v -x -d extract res.dat

And with that, I was able to extract the script_charactor.dat file. And from there I've finally got the desired *.gim files.

But those files wasn't the usual gims. It was just gzip files, that contained... Guess what? The dat file that I've got from the arc_conv-master at the beginning!

But I know the solution now.

Using tenoritool on it again creates 22 dat files. One of them has clearly the biggest size, so all you have to do is to change its extension to *.gim and you can open it/convert it to an image.

Wow, this was harsh. Sorry for bothering you when I was able to solve it myself the same day I wrote it here, but maybe someone else will find these instructions helpful.

 

On 13. 3. 2019 at 11:59 PM, Eco-Friendly said:

I'm trying to unpack squeez cgpack by using arc_unpacker , but only thing i get is 0_nopics.gr or something like that. Can anyone help me ?

Try GARbro. It isn't omnipotent, but it's the best user-friendly unpacker I know so far...

Edited March 17, 2019 by Nikdo

[Eco-Friendly]

15 hours ago, Nikdo said:

Try GARbro. It isn't omnipotent, but it's the best user-friendly unpacker I know so far..

Thanks , but it didn't helped . Saying it can't be opened as archive 

[Nikdo]

15 hours ago, Eco-Friendly said:

Thanks , but it didn't helped . Saying it can't be opened as archive 

OK, I'm gonna look at it.

Can you send here another pack of the same game, that isn't that large?

[Nanashi3]

4 hours ago, Nikdo said:

OK, I'm gonna look at it.

Can you send here another pack of the same game, that isn't that large?

Try https://github.com/xyzz/taiga-aisaka

Here is a precompiled copy with Java 8: https://mega.co.nz/#!7s90TQwD!JeRYuxxB62f9bt19RU_-mwYqX0eWgHwNAn0Uu4L8YTE

Example usage:

java -jar Gpda.jar res.dat

Compilation howto:

javac Gpda.java
jar cfe Gpda.jar Gpda *.class

Slightly modified to extract regardless of whether ".dat" is lowercase or not.

[Eco-Friendly]

7 hours ago, Nikdo said:

OK, I'm gonna look at it.

Can you send here another pack of the same game, that isn't that large?

I opened it with GARbro , it just worked for a third time !
Thank you for response !!))

[Veshurik]

Oh, I think, extracting files from PSP, PSVita, PS3/PS4 eroge is terrible... Even harden then for PC... If you are totally noob.

[Nikdo]

19 hours ago, Nanashi3 said:

Try https://github.com/xyzz/taiga-aisaka

Here is a precompiled copy with Java 8: https://mega.co.nz/#!7s90TQwD!JeRYuxxB62f9bt19RU_-mwYqX0eWgHwNAn0Uu4L8YTE

Example usage:

java -jar Gpda.jar res.dat

Compilation howto:

javac Gpda.java
jar cfe Gpda.jar Gpda *.class

Slightly modified to extract regardless of whether ".dat" is lowercase or not.

Well, thank you.

This program seems to work similary to tenoritool, but it also has the same "bug" when it comes to Hentai Ouji to Warawanai Neko.

While it normaly extracts res.dat and inner script_charactor.dat (which contains the base for characters images), the faces are stored inside script_charactor_face.dat.

Both tenoritool and Gpda extract the latter as "no data".

In tenoritool case, it contains the names and filesizes, but there is nothing inside (just zeros).

In Gpda case, it contains only name. The file sizes are always 0 B.

I have upload the file on a cloud, so you can download it here: https://dfiles.eu/files/cociazoa9

Just looking inside it with a hex editor, you can clearly tell that it does have the data there, both programs just fail to extract them.

Do you have any idea why?

[Nanashi3]

2 hours ago, Nikdo said:

Well, thank you.

This program seems to work similary to tenoritool, but it also has the same "bug" when it comes to Hentai Ouji to Warawanai Neko.

While it normaly extracts res.dat and inner script_charactor.dat (which contains the base for characters images), the faces are stored inside script_charactor_face.dat.

Both tenoritool and Gpda extract the latter as "no data".

In tenoritool case, it contains the names and filesizes, but there is nothing inside (just zeros).

In Gpda case, it contains only name. The file sizes are always 0 B.

I have upload the file on a cloud, so you can download it here: https://dfiles.eu/files/cociazoa9

Just looking inside it with a hex editor, you can clearly tell that it does have the data there, both programs just fail to extract them.

Do you have any idea why?

Ok so it seems there was an unhandled case in the file parsing.

I have compiled a new one for you @ https://github.com/mchubby/taiga-aisaka/releases (source changes at https://github.com/mchubby/taiga-aisaka/commit/14e066)

The resulting files could be further processed by the SDK's GimConv (if you need to mod & convert back) and Gim Viewer http://hsreina.shadosoft-tm.com/display/post/56

 

[Nikdo]

On 19. 3. 2019 at 8:29 PM, Nanashi3 said:

Ok so it seems there was an unhandled case in the file parsing.

I have compiled a new one for you @ https://github.com/mchubby/taiga-aisaka/releases (source changes at https://github.com/mchubby/taiga-aisaka/commit/14e066)

The resulting files could be further processed by the SDK's GimConv (if you need to mod & convert back) and Gim Viewer http://hsreina.shadosoft-tm.com/display/post/56

 

Haha, and there goes all my expectations.

Thank you so much for creating this version for me, but the only information I've got from this is, that the script_charactor.dat file nor any other of those now accessible files actually contain what I'm looking for.

You see, inside the script_charactor.dat file you can find images like this:

So I have logically expected, that the script_charactor_face.dat file will contain what is missing, but there are just "teasers" like this:

I'm starting to hate this game, really.

There is even someting like emote, but you just can't use it to finish the face, because it creates completely different images.

I know that the desired part may be assembled in just one image that I just happened to miss, but... There is actually still a file that wasn't extracted correctly.

It's name is "adummy.dat" (35,938 kB) and even now it contains only zeros.

Unfortunately, it's path is res.dat\adummy.dat so I have to share the whole resource now: https://dfiles.eu/files/hwpn85k6r

If it's not there, then I'm just lost.

Edited March 21, 2019 by Nikdo

[Nanashi3]

18 hours ago, Nikdo said:

Haha, and there goes all my expectations.

Thank you so much for creating this version for me, but the only information I've got from this is, that the script_charactor.dat file nor any other of those now accessible files actually contain what I'm looking for.

You see, inside the script_charactor.dat file you can find images like this:

Well the tools are not perfect but after extracting the RES.Dat using the java tool

cmd

cd RES\script_charactor.dat

ren *.gim *.gim.gz

 

Extract those gz files using 7-zip for example:

=> AZU_1B.dat, etc. to YOKO_3L.dat are created.
These are GPDA-headered files too but the java tool fails with a heap memory error.

Tenoritool can split those but with the wrong extensions

- the first files are textual configs [AZU_1B.0001.dat and AZU_1B.0002.dat]

- The largest file, like the one you posted,  is a GIM image [AZU_1B.0003.dat]

- the rest are eyemasks as GIM images [AZU_1B.0004.dat to AZU_1B.0022.dat]

 

 

[Nikdo]

4 hours ago, Nanashi3 said:

Well the tools are not perfect but after extracting the RES.Dat using the java tool

cmd

cd RES\script_charactor.dat

ren *.gim *.gim.gz

 

Extract those gz files using 7-zip for example:

=> AZU_1B.dat, etc. to YOKO_3L.dat are created.
These are GPDA-headered files too but the java tool fails with a heap memory error.

Tenoritool can split those but with the wrong extensions

- the first files are textual configs [AZU_1B.0001.dat and AZU_1B.0002.dat]

- The largest file, like the one you posted,  is a GIM image [AZU_1B.0003.dat]

- the rest are eyemasks as GIM images [AZU_1B.0004.dat to AZU_1B.0022.dat]

 

 

OMG you are kidding me.

So I had it in my palm the whole time, but I just toss it away as a trash... I'm such an idiot. Thank you, really.

But still, for the education sake, why that one file isn't extracted?

Is it because it's too large and some kind of space-limit overflow?

I have created some data-handling programs myself (mostly just wav or bmp though), so I have some basic idea what are those extractors doing, but... I don't know Java or C#, so those source codes tells nothing to me...

[pmrowla]

I've been working on a tool for extracting and patching LiveMaker/LiveNovel games and figured I should post it here. It started out as a fork of tinfoil's irl which I guess was kickstarted and then abandoned a while back?

Current status is:

• LiveMaker assets can be extracted from .exe or .dat files
• LiveMaker's binary .lsb format for their "chart" scripting can be dumped as text or xml
• LiveNovel scripts can be extracted from LiveMaker's binary .lsb format
• (Translated) LiveNovel scripts can be compiled and inserted back into a .lsb file
• Modified .lsb's can be patched back into a livemaker game and run with (cp932 encoded) english text, but currently the application will lock up when you try to exit.

My current code is here: https://github.com/pmrowla/pylivemaker/tree/wip (you have to use the wip branch link since I'm not pushing anything into master until I figure out the crash on exit thing)

The code itself is mostly fully commented but I haven't published any of the actual docs yet, so there's a github issue w/a summary of what works right now and what doesn't: https://github.com/pmrowla/pylivemaker/issues/1

 

Honestly idk if there's even any livemaker games worth patching but I was bored and felt like reversing something so I picked an obscure engine that afaik hasn't been dug into very much before. I think GARbro supports viewing .gal CG's from LiveMaker archives, but that doesn't cover the text extraction and patching side of things.