Hex-Editing Prerequisites

[NotHereMan]

I'm trying to understand how to use Hex Editors when it comes to non ASCII-character related bytes to see what the program is doing. Seems like most tutorials I come across are just getting to string messages. They do teach how to use the hex editor and understanding binary/hexidecimal, word size and Big/Little Endian(Some better than others). But, as far as interpreting the unreadable stuff I have not been able to find. Closest thing I came across was the idea of learning assembly.

What are the prerequisites to fully utilizing a hex-editor and knowing where to look for certain things? I'm familiar with C-style programming, some OOP, and I understand Binary math/Hexidecimal and just learned Big/Little Endian. Still working on determining when there's a byte/multibyte sequence in a program except for ASCII character, which are always just two nibbles/1 Byte.

 

[silentbogo]

Just read up some tutorials on IDA. Unless we are talking something simple, like analyzing blinky firmware for AVR micro, HEX editor is useless on its own.
HexRays now have a freeware version of the latest IDA, which is probably the best thing out there(at least for Windows).

Another good tool is Ghidra. Haven't worked with it yet, but it's FOSS with sources on github (though, developed by NSA).

Also, here's a cool YT channel on topic.

https://www.youtube.com/@LowLevelLearning

 

[Wirko]

What kind of files are you trying to analyse, is it program code or is it data (such as multimedia files, PDFs, binary database files, resource DLLs, etc)?

 

[NotHereMan]

Program files, in particular a Windows game executable. I'm guessing mostly program code. I'm trying to get data within the program code such as Console variables/commands and additional command line arguments that are part of the engine. Currently I'm trying to find the parts where the CD-Check occurs. This was removed in the next patch, but the files are too different to compare.