Tropico 3/4/5 HPK Archiver

[Zur13]

Is it the same error or something else?

They aren't going to repack the same size because...
1) I eliminate fragmentation (1/8 the size of the file system).
2) I eliminate residual data (in Lua.hpk, that is 1029 bytes).
3) I don't necessarily compress the same file types.
4) I have a minimum file size for compressing.
5) I added _filedates which preserves the "last modified" time.

Anyway, I did more tinkering (none of it should have impacted the file size) but I get 3,718,144 bytes, not "3,667,848 bytes." I'll upload it (1.0.9) shortly.

I've used v.1.0.9 to repack Lua.hpk, and again Tropico 5 failed to launch. I've captured software breakpoint under debug:

842702: Software breakpoint exception (exc.code 80000003, tid 2156)
Debugged application message: [LUA ERROR] Attempt to use an undefined global 'TerrainTextures'
        --- end of stack

At that part of code where this exception thrown, there must already run second app thread for lua, but its not.

Lua.hpk after repack has 3 667 757 bytes, that size defference from you've got on your system is weird, i'm using Win 8.1 maybe there some zlib implementation changes .

 

[FordGT90Concept]

I can't think of any reason why they would change sizes unless the source was different. The MD5 of the file I have is C78DEB43F796D7951A812D076D3E3AFE

Do we know what file TerrainTextures is in?

Edit:

Spoiler: List

-- Find in Files: TerrainTextures in *.lua; excl --
\CommonLua\Classes\AnimMomentHook.lua (27,1321)
\CommonLua\Classes\AnimMomentHook.lua (33,22)
\CommonLua\Classes\TerrainConfig.lua (18,243)
\CommonLua\Classes\TerrainConfig.lua (20,85)
\CommonLua\Classes\TerrainConfig.lua (26,63)
\CommonLua\Classes\TerrainConfig.lua (28,22)
\CommonLua\Classes\TerrainConfig.lua (31,459)
\CommonLua\Classes\TerrainConfig.lua (35,64)
\CommonLua\Classes\TerrainConfig.lua (38,188)
\CommonLua\Classes\TerrainConfig.lua (42,22)
\CommonLua\Classes\TerrainConfig.lua (45,4)
\CommonLua\Core\Terrain.lua (4,53)
\CommonLua\Core\Terrain.lua (5,22)
\CommonLua\Core\Terrain.lua (15,22)
\CommonLua\Core\Terrain.lua (20,22)
\CommonLua\Core\Terrain.lua (23,35)
\CommonLua\Dev\uaDev.lua (79,2128)
\CommonLua\Editor\EnrichTerrain.lua (78,22)
\CommonLua\UI\uiTerrainBrushesDlg.lua (11,22)
\CommonLua\UI\uiTerrainBrushesDlg.lua (27,541)
\Lua\Config\Terrain.lua (16,27)
\Lua\Dev\uaExtraDev.lua (5,2369)
\Lua\Game\OverlayGrids\BeautyGrid.lua (10,22)
\Lua\Game\OverlayGrids\CropGrids.lua (22,22)
\Lua\Game\OverlayGrids\HumidityGrid.lua (6,22)
\Lua\Game\OverlayGrids\PasturesGrid.lua (17,22)
26 occurrences have been found in 12 files.
Output completed (2 sec consumed)

12 files...that's a lot.

Edit: I extracted, archived, extracted again, and compared the two archived files. All 12 files above are 100% matched resync compared. In other words, find out why your HPK is smaller and you'll likely find your problem.

Edit: I'm going to make a program that compares all of the files in the two directories and looks for any inconsistencies....

Edit: 800 files compared and none deviated. The archive's contents are identical on my end.

 

[Zur13]

Thanks again for your help. I've found why there was problem with repack. I've forget to uncheck Compress it checkbox when packed Lua.hpk for Tropico 5.

 

[FordGT90Concept]

Does Tropico 5 not support compressed HPK? I'm really not sure which versions do.

 

[Zur13]

Does Tropico 5 not support compressed HPK? I'm really not sure which versions do.

It seems so. At least when I've used repacked hpk without compression tropico 5 had launched normally, so HPK Archiver can be used for Tropico 5 modding now.

Now I'm going to decompile lua scripts, as far as I saw some scripts can't be decompilled correctly so there are still some problems for modding ahead.

 

[FordGT90Concept]

Released 1.0.10 which separates the Tropico 5 option. When checked, it disables "Compress It" because compressed archives apparently aren't supported in Tropico 5. Otherwise, it is functionally identical to Tropico 4.

 

[OPTWood]

The .lua files unpacked from Tropico 5 are unreadable. I have uploaded the bank.lua file from the \Data\BuildingTemplate directory. I have added a .txt extension to allow it to be uploaded.

 

[FordGT90Concept]

It looks correct to me. Unreadable as in a Lua decompiler can't make sense of it or...? I think it's already been mentioned that the decompilers available don't work on the version Tropico 5 uses. I'm no expert on Lua so I can't tell whether or not the file fully extracted.

 

[Zur13]

The .lua files unpacked from Tropico 5 are unreadable. I have uploaded the bank.lua file from the \Data\BuildingTemplate directory. I have added a .txt extension to allow it to be uploaded.

Here the modding tutorial, there is description how to decompile lua files.
http://www.techpowerup.com/forums/threads/tropico-5-small-modding-tutorial.201529/

 

[BloodnAshes]

Hello good people of the internet,
I have Tropico 3 Gold Edition and I'm trying to edit some of the capacities for buildings ( ex. grade school to 12) and such, if that's even possible, but when I extract the .hpk files all I get back is .dds files which I can't edit.

Question: Am I an idiot or is there something wrong with the output - shouldn't they be being converted to usable .txt files or something?

 

[cxd5]

Hi

i have your program for localization and I have a problem, because after unpack and repack english.hpk (I try to change to other language ofc) I don't see any changes. Furthermore when I delete this file permanently from local folder I still have english language... so I don't think that file is necessary... any options to localize english.hpk?

edit: Tropico 5 of course :]

 

[FordGT90Concept]

Hello good people of the internet,
I have Tropico 3 Gold Edition and I'm trying to edit some of the capacities for buildings ( ex. grade school to 12) and such, if that's even possible, but when I extract the .hpk files all I get back is .dds files which I can't edit.

Question: Am I an idiot or is there something wrong with the output - shouldn't they be being converted to usable .txt files or something?

No. As far as I know, that stuff is in LUA scripts and LUA scripts are compiled. You'd have to find a modding tutorial for Tropico 3 that explains how to decompile and recompile LUA scripts for Tropico 3.

Hi

i have your program for localization and I have a problem, because after unpack and repack english.hpk (I try to change to other language ofc) I don't see any changes. Furthermore when I delete this file permanently from local folder I still have english language... so I don't think that file is necessary... any options to localize english.hpk?

edit: Tropico 5 of course :]

I have no idea. All I do is extract and archive HPK. I never really researched/cared about what was inside.

 

[samusya]

Where can I find support for Turkish language tropico 5?

 

[Luk]

Hi there
I'm new here and I have noticed that I changed the language of the game (I do not care, now is English and I get it too) but I wish someone could climb 2 hpk files that do not have English, then do not listen to Penultimo or anything voice in the game, the files are in Tropico 5 --> local--> english.hpk and Tropico 5 -->local--> Voices --> english.hpk. If someone uploaded the "local" folder would be sufficient

Samusya, you can find support on support@kalypsomedia.com, but you must send the message in English or German. If the game is the same, the language is not a problem, unless the problem is related to the translation.

 

[jianeddie]

Hi, I got some trouble in extracting the dlc HPK.
There are some HPK files extracted from the dlc HPK, but there are not extracted correctly.
Just get the error message: "Could not decompress the file. Writing compressed file."
When I try to extract this HPK files, HPK Archiver just crash.

here is the dlc file : http://1drv.ms/1vl0Zc3

I find my translation is not compatible with the new dlc. So I need to get the translation text in the dlc.
I hope you can help me. And thanks.

 

[FordGT90Concept]

My guess is it is another one of those compressed files that either were 0 length or too short to compress. I'll take a look quick and see how serious it is...

revisions.lua = text file, no compression in the first place
Local\language.hpk = obvious issues here but too tired to investigate now. Curious that they are HPKs inside of an HPK...
Music\31767_Soca pati_246.opus = Opus...wait, what? It's obviously still compressed but that's still different as far as I know
Maps\DLC_Guerra.hpk = obvious trend here forming...
Lua.hpk = ...HPK inside of the HPK...


So, nothing wrong with revisions and 8 of the remaining items appear to be related to the fact we have HPK inside of HPK. The 9th (Opus) may be a case of the same thing but mislabeled.

Edit: Those alleged HPKs are not HPKs at all. They use the generic compression used since Tropico 3 (at least) but there is something breaking the decompression code and I'm not sure what. I'll have to investigate later but I suspect once I find the issue, it will fix all 9 files.

 

[FordGT90Concept]

Well, I spotted one problem but it only looks to apply to the language LUAs. These zipped HPKs have a tiny table of contents at the end with Game.csv and CurrentLanguage at the end. I imagine Game.csv is the file you're after. I'm not entirely sure what CurrentLanguage is about (might be a folder). There is one outstanding problem though preventing me from fixing it: I don't see anything anywhere telling me that that little bit is there at the end. Maybe... HPK file that begins with ZLIB instead of BPUL may be a clue... but I still haven't found anything pointing to that little table.

Edit: I have an idea. Perhaps all they need is the HPK header slapped on it.

 

[jianeddie]

I think these HPK files just like those in other folders of the game. Maybe just a little or even nothing different.
If it is so, the language hpk should contain the folder “CurrentLanguage” and the file "Game.csv".

Maybe these three files can help
english.hpk: http://1drv.ms/1u3aFtx
lua.hpk: http://1drv.ms/1mynnPx
data.hpk: http://1drv.ms/1mynqLh

 

[FordGT90Concept]

I'm increasingly becoming convinced that these files are FUBAR. They are zipped when they shouldn't be zipped and even then, they are only partially zipped instead of completely zipped. I'm having a hard time believing the game can even access these files at all. It's a disaster what they did here.


Edit: Good news is I got Italian.lua decompressed (it was the first one in the archive). Bad news is I see no way to automate the process. I assume you want English the most?

 

[FordGT90Concept]

I got it fixed and it was stupidly simple (literally two lines of code). Original:

                    try
                    {
                        inflate.SetInput(source);
                        result = inflate.Inflate(destination);
                    }
                    catch { };

Fixed:

                    try
                    {
                        // attempt to inflate
                        inflate.SetInput(source);
                        result = inflate.Inflate(destination);
                    }
                    catch
                    {
                        // inflate failed, write uninflated data to output
                        Array.Copy(source, destination, source.Length);
                        result = source.Length;
                    };

Download (1.0.11) is on the original post.

 

[jianeddie]

Thanks, it works well now.

 

[FordGT90Concept]

I got to thinking...

How much demand is there for extracting/archiving via command line arguments? It already has -extract, -archive, and -zip. I'm going to add some more just to try to figure out the problem associated with the files below...

How much demand is there for an extract all button/command line argument (e.g. point at Tropico 3 directory and extract them all to this directory)? If there is, how should it handle that in terms of file/folder structure? Could all of the HPK files technically be extracted and packed into one giant HPK?


Edit: I went looking through old posts in this thread and the changes made up to 1.0.11 apparently fixed all but 3 out of 1334 errors in Tropico 4\dlc\Expansion.hpk pointed out here:
http://www.techpowerup.com/forums/threads/tropico-3-4-hpk-archiver.113705/page-2#post-2831529

C:\Users\Admin\Desktop\New folder\ex\game\CommonEntities.txt	File too small to decompress, outputting blank file.
C:\Users\Admin\Desktop\New folder\maps\__EXPgamaSAND.hpk	Could not decompress the file.  Writing compressed file.
C:\Users\Admin\Desktop\New folder\VoiceEx\German\31067.ogg	Could not decompress the file.  Writing compressed file.

1st one is simply just a broke file.
2nd one appears to use uint32 for offsets instead of the normal uint16 (might be able to write an exception/fix for that)
3rd one is the same as the second.

Edit: Not sure now what is wrong with them.

 

[FordGT90Concept]

I got #2 fixed! It was actually directly related to the previous fix. In those two files, zlib apparently returns error result instead of outright crashing and that's why my code failed on them (extremely rare apparently). So, I did zlib a favor and threw an exception before calling it if output size = input size (chunk is not deflated). This should make it marginally faster too on heavily compressed, but erring archives.

I also made the program start as console and show command results in the console. See the original post for more information. I'm going to try to work some more on #3.

Edit: got #3 fixed and it was yet again the same code. Now the code effectively never breaks: it succeeds at inflating, doesn't try to inflate, or it outputs the supposedly deflated content because it was never deflated in the first place.

1333 out of 1334 errors fixing in Expansion.hpk and the last is a non-error. I'll do a few more tests to make sure it is working right then upload 1.0.12.

If you're curious, here is what the code before now looks like:

                    try
                    {
                        if (source.Length == header.ChunkSize)
                            throw new Exception("Can't inflate when expected size matches input size."); // Send to catch

                        // attempt to inflate
                        inflate.SetInput(source);
                        result = inflate.Inflate(destination);
                    }
                    catch { result = 0; }; // fall through to the code below
                    if (result == 0)
                    {
                        // dump the chunk to file -- it may not be compressed at all
                        Array.Copy(source, destination, source.Length);
                        result = source.Length;
                    }

Edit: Just uploaded 1.0.12. I'm open to adding more argument commands if there's demand for it.

 

[RatsU]

Can this work for Grand Age : Rome? ROA?
When I try it. It stops at 82% and fatal error.

i'm using windows 8.1 64bit.

Below is the error
C:\gar\Data\Precache.ini File too small to decompress, outputting blank file.

 

[FordGT90Concept]

Got it fixed in 1.0.13 as well as a few other things.

General note: If you see "Applying last modified dates to files..." for a long time or the dates match today, it likely means you have the wrong game checked. For example, Grand Ages: Rome with Tropico 5 checked added about 10 seconds to the extract time because it was catching an error over 1000 times.