CQ0 on h.264 will be really unnecessarily huge, because it's literally lossless quality. I'm not sure if mpeg-2 actually has a lossless setting. If you're really bored and curious, go back and try h.264 with CQ1 instead?
Weirdly enough, CQ1 either stayed the same or increased size. But CQ2 and after, all yielded some size reduction.
I ran this test using Handbrake "Production Max" preset, with two config subsets. The "normal" one, while changing only the CQ, and the "special" one where I also removed a few encoder settings that were set manually into Handbrake by the developer. I had to do a little research for this, but it was kinda fun
The encoder additional options that I removed were:
keyint and
min-keyint, which control the interval between IDR frames. Just to explain it quickly here for those that do not know it (and if someone is well versed in this stuff and spots an error, do tell), H264 uses "reference" frames (IDR frames) that contain a full picture (this type of frames which are "full pictures" are called I-frames, but not all I-frames are IDR frames, depending on encoding settings), while the following frames only tell the decoder what has changed. They also act as a separator of sorts, since a frame can't reference any other frame that is on the other side of the IDR frame. They're quite useful for the fast-seeking feature of nearly every media player out there, but they're rather large, so letting the encoder use a larger interval saves a few bytes here and there. The original setting was 12 frames per IDR frame, removing this option let the encoder use one IDR frame per 300 "normal" frames.
ref. This controls the size of the Decoded Picture Buffer. This affects the so-called P-frames (frames that only contain differences compared to previous frames). The normal Prod. Max preset only lets a P-frame reference the previous frame. Removing this setting lets a P-frame use more (in this case, since I used only the "fast" encoder preset, the encoder used 2 ref frames. The slower/higher encoding quality presets allow for up to 16 ref frames, but the 3820x2160 resolution enforces the 5.1 encoder level, which is limited to a maximum of 5 ref frames)
bframes. This one controls B-frames (which are similar to P-frames, but can also use the future frames as reference, not just the previous ones). Again, the fast preset uses only 3 frames, but H264 supports up to 16 frames for this.
dct-decimate. DCT Decimation allows the x264 encoder to skip encoding any of the blocks which it views as redundant information. These blocks are the not written to the output file, effectively reducing the bitrate. Blocks are basically arrays of pixels, as I understand it. The option was set to 0 (no decimation), but I removed it. Again, I think this might be affected by both the encoder preset and the quality control, so maybe using a lower quality or a slower encoder preset would have yielded a bigger size saving?
fast-pskip. By default, x264 will skip macroblocks in P-frames that don't appear to have changed enough between two frames to justify encoding the difference. This considerably speeds up encoding. When this parameter is not set explicitly, it's controlled by the encoder preset and quality control, from what I have observed.
I left the other options (qcomp, aq-strength and deblock) alone, since they are still a little beyond my understanding and do not really change the size of the video stream. They deal mostly with quality concerns according to the content you're encoding, or at least that's what I understand after looking through a bunch of pages.
Now, if we could get AV1 ENCODE acceleration, that'd be sweet...
Not in this GPU generation, probably.
Though, do you actually plan to use AV1 for something?
