Before someone jumps all over me and tells me that these aren't optimal, I know. I'm just a regular computer user, not a video engineer. But I guarantee some TPU participant will dissect every single thing I've written with the goal of brutally tearing it to shreds. Enjoy the fodder!
I have been several years since I researched this topic. Like you my goal was to save storage space on existing content that didn't require ultra quality video. I do most of my Handbrake transcoding on my Mac mini 2018 these days.
Here are the comments I can offer.
Right now the settings i am using are this:
1) same resolution as original
2) HEVC 10-bit H.265 .mp4
Yes to both.
I currently use the H.265 10-bit (VideoToolbox) encoder setting which harnesses the hardware encoder in the Mac's T2 Security Chip.
I also use the .mp4 container since so they can be synced to my iPhone and iPad if desired; the native media players don't accept .mkv container files. My Kodi media player (a Raspberry Pi 4) can handle .mkv though.
I know there are hardware encoders for Intel CPUs (Quicksync), Nvidia (NVEnc), and AMD (I forgot what it's called, VCE/VCN I think). I did identical transcodes on all four (three Windows PCs plus the Mac's VideoToolbox encoder) and I thought they all gave acceptable output. I figure some video engineer could pick apart each and say what's good/bad about each but for my mundane purposes, all four hardware encoding technologies were adequate at least when I tried this sometime in 2020.
Back when Handbrake didn't offer the VideoToolbox Mac support, I used software encoding (H.265 10-bit) and did 2-Pass Encoding, Turbo First Pass. Two full passes simply took too much time for the tiny amount of image quality improvement.
When I researched this, I believe I read something that said that anything above RF 20 was a waste of time for mundane content and transcoding projects. My memory is hazy but I ended up with RF 19 or RF 20. If I recall correctly, transcoding animation source files results in better output if the RF isn't so high.
Today, I don't set quality with RF or CQ (Constant Quality) numbers.
I just set a target bitrate, Average Bitrate (kbps) that depends on output resolution. For 480p output, I generally set it around 1500-2000 kbps. For 1080p output, I generally set around 4000 kbps. If I care more about image quality for some content, I might bump 1080p transcodes up to 8000 kbps, especially if I am using 4K source material. If it's widescreen content that will end up 1920 x 800p or so, I sometimes use 6000 kbps.
I picked these values based on test encodes over the years.
I use Encoder Preset MEDIUM if available like software transcoding. For the Mac's VideoToolbox setting, there are two settings: Speed and Quality. I generally select Quality since the bitrate is going to squash image quality anyhow. I don't need super fast encodes anyhow.
Under Video > Encoder Options > Additional Options I add this line:
strong-intra-smoothing=0:rect=0
I don't remember what it does. Someone seemingly knowledgeable mentioned it on some long forgotten Q&A forum so I've kept it there.
5) original fps - variable (changing this to variable 25 fps now since otherwise at some vids it crashes. Could i go even lower, like 23 fps to save space, without noticing much ?)
I keep Framerate (FPS): Same as source, Variable Framerate.
I avoid transcoding or downmixing audio. I select AAC or E-AC3 passthrough when available. Transcoding audio isn't going to save you much in the way of disk space, you might as well keep the original audio whether it's good (multi-channel Dolby/DTS) or bad.
A long time ago I used to downmix to 2-channel stereo so I could hear them on my iPhone or iPad. At some point, Apple started supporting multi-channel audio formats in iOS and iPadOS and downmixing during playback so now I can just leave them as is.
I tend to batch load the whole folder to save time. Other settings i leave at default since i dont know their effect. It mostly does the job done, though there is always room for improvement. Occasionaly (but rarely) i get a file that is larger than the original, which is strange since the original is like .avi, H.264, .mpeg or something different. Doesn't make much sense. What else can i change/modify to improve space/time/quality ?
My guess is that the RF 25 setting will increase some low-quality files to a higher size. This might have been why I decided to move away from RF/CQ as the way to set output quality.
Ultimately I ended up with these setting based on what my eyes told me was acceptable final quality. I transcoded the same source material (a chapter or short clip) with different settings and let my eyes decide. It's best to look for artifacts, etc. not just where the main action is happening but also in the background patterns, etc. But ultimately if you're watching some superhero fighting a villain, you're probably not looking at the power line on a utility pole in the background.
Anyhow, I suggest you try to transcode content you actually care about and see what settings you come up with. Then apply those settings to your larger batches (possibly dialing down quality if desired).
In the end, it's up to you to decide what works, sort of like cooking. You're the one who has to eat it.