Tropico 3/4/5 HPK Archiver

[canefire7]

@nickelc I am having trouble running the tool, how do you go about running it? a bit confused :S I have installed rust and VC runtime ++ but can't figure out what to do next (tried using run function in rust with the main.rs provided)

 

[nickelc]

@canefire7
You can find compiled versions on the release page https://github.com/nickelc/hpk/releases
but if you want to compile by yourself then use cargo run to run the debug version (release: cargo run --release).
The compiled executables are in target/debug or target/release

Full examples:

cargo run -- extract path/to/file.hpk destination_dir
cargo run -- create source_dir dest.hpk
// run release build
cargo run --release -- list path/to/file.hpk

// build only
cargo build
// or
cargo build --release

 

[Viktor_Riven]

Hello everyone and nice to be here!

I really appreciated the HPK Archiver since it allowed me to open a file from the first chapter of the saga in which the game "Grand Ages: Rome" is the 3rd one. The game I'm working on is, in fact, "Imperium Civitas I" or "Glory of the Roman Empire".
I decided to start modding the game by adding and modifying some trade routes. I was able to decompress, with your tool, the .hpk file where files about trade routes are located but I get two errors:

Spoiler: "error.txt" Contents

C:\Users\Username\Downloads\hpk_archiver_1_0_14\Missions\MainMenu\no_localization Warning: File too small to decompress, outputting blank file.
C:\Users\Username\Downloads\hpk_archiver_1_0_14\Missions\FX-missions\scripts\20_tradepost.hgs Warning: File too small to decompress, outputting blank file.

The problem is that I cannot re-compress the file anymore since, because of these two errors in decompressing, I miss those two files cited by the log and, by re-creating a compressed hpk version of the folder, it will result in a crash-to-desktop because of the blank files missing.

I'm wondering if there is a way to solve this problem

Thanks in advance

 

[FordGT90Concept]

I don't have that game so I can't really look into it. How large is the HPK file you're extracting?

 

[Viktor_Riven]

Many thanks for the quick reply!

Some information about the file: it is large 177 Mb. Called "Missions.hpk", it contains all the levels of the game.

 

[FordGT90Concept]

20_tradepost.hgs

is literally

5A4C49420000000000800000

which is

ZLIB    0       32768

ZLIB means it is compressed and how
0 is the file length
32768 is the ZLIB chunk size (almost always that number).

Another compressed file starts immediately after this so there's no mistakes on my end.

"Warning: File too small to decompress, outputting blank file." is the best we can do here.

no_localization is the same: 5A4C49420000000000800000

I packed it back up and unpacked the freshly packed file and it preserved those two empty files.

So, whatever the problem is, it isn't those warnings.


AFAIK, the game doesn't need them packed in an HPK to work. Can just unpack Missions.hpk then rename it (e.g. Missions.hpk.bak) so the game can't find it. It will then use the unpacked files instead.


-----------------------------------------------------------------


Just updated the original post with version 1.1.0. It adds support for LZ4 (inflate and deflate) but in doing so, I rewrote significant amounts of code. It should theoretically all work but...yeah...could be bugs.

Spoiler: Technical Details

The program was set up to determine if a file was compressed and then choose the right method to decompress and extract it. Now one method does it all: discovers if it is empty, uncompressed, zlib compresssed, or lz4 compressed and extracts it.

On the archival side, the single method that used to compress it before now does LZ4 or ZLIB based on the game. Compressing the entire archive technically supports LZ4 as well but no code path will trigger it because I don't know that the newer games even support it.

It does not implement nickelc's workaround for lua because that's outside of the scope of what the program does.


I noticed the LZ4 compressed file is significantly bigger than the extracted file. No idea if the games will be able to load it. It's using Encode right now.
original: 378 KB
EncodeHC: 421 KB
Encode: 491 KB

 

[Viktor_Riven]

I really thank you for the support! I made some tests and here are the results.

Summary:

Errors for files too small disappeared.
Problem in re-packing or re-presenting files to the game.

Deep explanation:

- With the new version, during the extraction process, no error log appears, even if the two incriminated files, once uncompressed, are 0 Kb as before. Said that, these two files are not, in my opinion, crucial for the game (as the one I'm trying to modify).
- Once modified the file with the trade routes specifically for the Venice map, I tried to re-compress the hpk archive and I tried, with your last version of HPK Archiver, the compression method for Tropico 3/Grand Ages: Rome (with or without compression box selected) and for Tropico 4 (just to try).
- Unfortunately, once re-added the Mission.hpk file (the resulting file has the same weight of the original) in the Packs directory of the game, an error occurs and Imperium Civitas crashes in the loading screen. It crashes at a specific point: at the debugging stage, as told by this errors log (see file attached with Drive: Error Log), this log is about the crash when I put the uncompressed folder as it were after the compression. This means that the game has problems to recognise files if they are not compressed (or maybe there is something more complex that can be understood by reading the log).

Hope these information can be of any use.

 

[FordGT90Concept]

Make sure you're extracting to and repacking from the same, empty directory.

 

[666jason666]

I just extracted and compressed the file without making any changes. Notice how the file loses data. What could be happening? The game does not read the file. Please help me. Attached files ...

https://drive.google.com/file/d/1yuxs8lB5VQ6OsYO4UC0FYJzm_safMTlD/view?usp=sharing
https://drive.google.com/file/d/1GI46CeT5N-Ov_wpTM8B7069p9k9XMbnC/view?usp=sharing

 

[FordGT90Concept]

It's normal for the file to shrink some because the original HPK tools can add files to the archive marking off segments of the file to ignore. When I extract them, I don't extract those segments so they don't end up back in the repacked archive.

What game is this?

 

[666jason666]

This is Tropic 5. I make changes to the file and compress, but the game does not read it. There is no change in the language of the game. Changes that can make recordings, like the game, can not read the file I created. What can it be? Can you do the editing and compress and test for me? If you can see what the problem is.

 

[FordGT90Concept]

Are you for sure checking the Tropico 5 box?

The folder you select to extract to is the same folder you should select to repack it.

 

[Wiotbilu]

Hey, I'm trying to make a mod for grands Age Rome, I decompressed the data.hpk file, but I can not open the .lua files right, I'm using notepad ++ in the moon language, but I'm not understanding anything, a lot of NUL, NUL , EOT, do you use any other application to perfectly open .lua files? because I use Notepad ++ to open the .lua file of Crusader kings 2 perfectly.

 

[ChoGGi]

Sounds like they're compiled, you'd need to use something like unlarc or luadec to decompile them.

 

[ashik2611]

Can any help me with detail description,
How Can I extract Surviving mars terrains with textures to a 3d software?
(Like obj or fbx)

 

[ChoGGi]

Probably not doable, but you can use Terraformer to edit them in-game.

Edit: you can get the textures out easily enough: file names are in TerrainTextures, and then you'd have to extract the dds files from Textures*.hpk

 

[Madbey9]

When I unpacked the file from GA:Rome, textures were damaged and I don't know why? Anybody help?

 

[FordGT90Concept]

I don't know what format the textures are in but it is likely something non-standard. If it's DDS, for example, you'd need to use something like Paint.NET to open them.

 

[Madbey9]

The format is .dds and I have paint.net, and thats why I know they are damaged. Sry for my bad english

 

[FordGT90Concept]

Could be a different version from what Paint.NET supports or they contain metadata that Paint.NET doesn't know how to properly handle.

All HPK Archiver does is pack and unpack files/folders. It doesn't modify the content of the files.

 

[Madbey9]

Ok so probably it's my weak computer's fault. Thx for reply

 

[ChoGGi]

Might be an issue with hpk archiver, I got a corrupt looking dds as well. It works fine with this.

 

[FordGT90Concept]

Looks like nickelc recently updated his to make it work.

I'd need the source HPK to debug mine.

 

[ChoGGi]

Not sure how recently you mean, the game is from 2009
https://mega.nz/#!p94zxYoK!km3sRTli7X5ZSpN1J0kLBfQvoUrjoXVGaLdn3M5tzvc

 

[FordGT90Concept]

Well now I'm curious why it doesn't work. Checking...

Edit: It successfully decompresses the first chunk and then fails on the second: "Old input was not completely processed"

It was expecting 14916 but got 14912. In other words, the version of zlib I'm using doesn't match whatever the game uses. Updating it could break previous games like Tropico 3.

Edit: I tried updating SharpZipLib for giggles and it is giving the same error. I'd have to have multiple zlib versions each targeting their respective games.

Since nickelc already has a working version, I think this application will become abandonware. I'll provide a prominent link to nicklec's solution in the original post.