Tropico 3/4/5 HPK Archiver

[666jason666]

Thanks for listening!! I realized that when I generate the hpk file, the file size decreases, it is being compressed. I read in your post that this could not be done in Tropico 5. I downloaded the translation in Spanish (it has 850k) I extract it and compress it again (to test) it has decreased its size, getting 808k. The file is being compressed, right? Can you help me with this?

 

[FordGT90Concept]

If you're not checking the "Compress It" box then it's likely just getting rid of junk data.

 

[666jason666]

I'm not checking this box. but somehow the file is being compressed because it is losing some kbytes. I extract from a .hpk file of 850k and then compress (without making any changes) when I re-compress it to .hpk format it gets 808k. The game does not read the file anymore ... what happens? Plz help me =(

 

[FordGT90Concept]

Like I said, most likely junk data. There's a table in the file of orphaned data. This happens when Kalypso added a file to the archive that's bigger than the file it replaced. Instead of rewriting the whole file to add that extra data, it instead marks the old file as orphaned and adds a new file entry adding it to the end of the archive. When you extract the file, it doesn't extract the orphaned data so when you archive it again, the file ends up being smaller even though the usable contents of it are the same.

 

[666jason666]

I understood! But when I generate the file (English.hpk) and put it in the local folder replacing the file for the translation, the game does not read it. What can it be?

 

[FordGT90Concept]

It's better to ask specific questions about modding here:
http://forum.kalypsomedia.com/forumdisplay.php?fid=87

Via this thread:
https://www.techpowerup.com/forums/threads/tropico-5-small-modding-tutorial.201529/

Which at the bottom says you don't even need to repack the HPK. The game apparently loads the local files first before trying to get the asset from the HPKs.

 

[jianeddie]

Hello, the new game Surviving Mars just release. And I try to translate it, but when I extract the english.hpk, it just got a Game.csv with a new compression format.

Is there any idea to deal with these?

Here the english.hpk

Here the lua.hpk

 

[FordGT90Concept]

If it dumped absolutely everything into one file then there's a major problem with extracting.

 

[jianeddie]

If it dumped absolutely everything into one file then there's a major problem with extracting.

I think there should be just one file in the english.hpk, as the same as Tropico 5.

Besides I can't figure out if there's some problem with extracting or I got the wrong lua decompiler, I got a empty file when I decompiled the lua file which just extracting.

 

[FordGT90Concept]

The HPK header starts with "BPUL" which is missing from your picture. That means it was successfully extracted; however, look at what the picture starts with: "LZ4" You're right, that's a compression algorithm. I never updated my program to handle LZ4 but atom0s did here: https://www.techpowerup.com/forums/threads/tropico-3-4-5-hpk-archiver.113705/page-5#post-3679753

You can try his build with LZ4 added and see if that works.

 

[ChoGGi]

It doesn't seem to extract properly either (Edit: it meaning atom0s version)

LuaS “

xV           (w@@Mars/Lua/MarsMarkers.lua             @ K@  ‹ € ÁÀ  «@€ J€
@€€ A l  
@€‚& €    DefineClassRCRoverMarker
__parents EditorMarkerOnMsgChangeMapDone                      F @ ‹€  Š€À€ì   ŠÀ€d@ & €    ForEachclassRCRoverMarkerexec                   F @ Œ@@ ¤€ Æ€@ ÇÀÀd€€† A Á€  ¤€ Ì@A@€ ä@€& €    GetPassablePointNearbyGetPosRCRoverpfclassPlaceObjectSetPos                                               marker       p     r            _ENV  
  
           
         map          _ENV                                                   _ENV

 

[jianeddie]

The HPK header starts with "BPUL" which is missing from your picture. That means it was successfully extracted; however, look at what the picture starts with: "LZ4" You're right, that's a compression algorithm. I never updated my program to handle LZ4 but atom0s did here: https://www.techpowerup.com/forums/threads/tropico-3-4-5-hpk-archiver.113705/page-5#post-3679753

You can try his build with LZ4 added and see if that works.

Thanks, I got the right Game.csv.

But the lua files still have some problem, unluac or luadec not work at all. I got from google that the lua files seems to be modified by Haemimont Games, not the source file.

 

[FordGT90Concept]

They could have easily updated to a newer version of Lua that unluac/luadec don't support.

 

[jianeddie]

Well, It seems miss two bytes at the header of the lua file, I don't know if the problem cause by the tool, or Haemimont Games did this.

Trying to find some way to deal with this (Finally I solved this, and the unluac works well, I think). Or maybe you can fix this in your application?

 

[FordGT90Concept]

That's part of the LuaScript file. Haemimont games likely added it so it's something unluac would have to account for.

 

[nickelc]

Well, It seems miss two bytes at the header of the lua file, I don't know if the problem cause by the tool, or Haemimont Games did this.

Trying to find some way to deal with this (Finally I solved this, and the unluac works well, I think). Or maybe you can fix this in your application?

View attachment 98530

Can you give me more details about the missing two bytes of the lua files?
I have not yet entered the world of lua. Did you build unluac from source?

i've created an alternate hpk archiver tool https://github.com/nickelc/hpk and i think i could support the processing of the lua files to fix them.
It's written in Rust and requires Microsoft VC++ runtime.

It supports:

• Tropico 3-5
• Omerta: City of Gangsters
• Grand Ages: Rome
• Victor Vran
• Surviving Mars

While Victor Vran & Surviving Mars dont like repacking with the used lz4 compression algorithm for some reason, both are fine with zlib compressed files.

 

[FordGT90Concept]

My fear is that those bytes may indicate another chunk in the Lua file that may be important. I'd approach the authors of the decompilers so, at bare minimum, they can gracefully consume those bytes.

 

[jianeddie]

Can you give me more details about the missing two bytes of the lua files?
I have not yet entered the world of lua. Did you build unluac from source?

i've created an alternate hpk archiver tool https://github.com/nickelc/hpk and i think i could support the processing of the lua files to fix them.
It's written in Rust and requires Microsoft VC++ runtime.

It supports:

• Tropico 3-5
• Omerta: City of Gangsters
• Grand Ages: Rome
• Victor Vran
• Surviving Mars

While Victor Vran & Surviving Mars dont like repacking with the used lz4 compression algorithm for some reason, both are fine with zlib compressed files.

As I know Surviving Mars were built by LUA 5.3, I just compare the extracted one with the normal 5.3 version lua file built by the luac.

All information the picture has showed, no more detail I can tell, because I am not familiar with Lua too. Beside I just rewrite the lua files to let unluac can read them, for me this is the easiest way.

Maybe luadec or unluac could give you more idea.

 

[ChoGGi]

All information the picture has showed, no more detail I can tell, because I am not familiar with Lua too. Beside I just rewrite the lua files to let unluac can read them, for me this is the easiest way.

Could you show the before/after of a rewritten lua?

 

[nickelc]

So I've looked into the lua source code and found this https://www.lua.org/source/5.3/ldump.c.html#DumpHeader

static void DumpHeader (DumpState *D) {
  DumpLiteral(LUA_SIGNATURE, D);
  DumpByte(LUAC_VERSION, D);
  DumpByte(LUAC_FORMAT, D);
  DumpLiteral(LUAC_DATA, D);
  DumpByte(sizeof(int), D);
  DumpByte(sizeof(size_t), D);         // missing `0x04` or `0x08` depends on the used lua interpreter (32bit/64bit)
  DumpByte(sizeof(Instruction), D);
  DumpByte(sizeof(lua_Integer), D);    // missing 0x08 - lua_Integer is defined by default as `long long` or `__int64`
  DumpByte(sizeof(lua_Number), D);
  DumpInteger(LUAC_INT, D);
  DumpNumber(LUAC_NUM, D);
}

I haven't tested it yet but probably the game doesn't like the valid bytecode headers when repacking the scripts.
Still investigating a nice and clean solution for the extraction.

Could you show the before/after of a rewritten lua?

Before:

After:

 

[ChoGGi]

Thank you, that helped.
Anyone have any luck decoding Data.hpk\TechTree.lua?
The best I can do is get luadec sucking up 26 gigs of ram before crashing with a not enough memory error.

 

[nickelc]

unluac also doesnt like it and exits with The size of an integer is outside the range that unluac can handle.

ok, unluac failed because i set the byte offset 13 [sizeof(size_t)] in the bytecode header to 0x08 instead of 0x04.
it works fine with the right header.

 

[jianeddie]

Thank you, that helped.
Anyone have any luck decoding Data.hpk\TechTree.lua?
The best I can do is get luadec sucking up 26 gigs of ram before crashing with a not enough memory error.

Here TechTree.lua, decompiled by unluac.

 

[ChoGGi]

Ah sorry should've edited my post, got it decompiled thanks to nickelc.

 

[nickelc]

I pushed a new release v0.2 of my hpk tool: https://github.com/nickelc/hpk

New command line options to fix the invalid lua bytecode headers of Surviving Mars.

With Surviving Mars they switched from Lua 5.1 to 5.3 but the bytecode
headers are missing two bytes to make it possible to decompile the scripts.

|Surviving Mars' Lua Bytecode Header
Broken | 1B4C 7561 5300 1993 0D0A 1A0A 0404 ____ 0878 5600 ...
Valid | 1B4C 7561 5300 1993 0D0A 1A0A 0404 0408 0878 5600 ...

Use hpk extract --fix-lua-files ... to fix the scripts on extraction.
Use hpk create --cripple-lua-files ... to change the bytecode header of the compiled lua scripts on repacking.
The game can also load plain Lua files instead of a compiled version if you run into any problems.