@ecopsorn Are you just using the NT-H1 that came with the D15? It doesn't sound like case airflow is the culprit here. The thing about idle temperatures on Matisse is that a idle peak of 50-60C is just fine, because of how Zen 2 cores respond quickly to load, but
idling around 50C and never dipping below is somewhat concerning on a top of the line air cooler like the D15. I idle around 30-40C and peak to 50C once in a blue moon on just a U9S, a cooler that's only 3/4 the height and just over 1/2 the heatsink mass of the D15.
As for load, the CPU itself starts taking off performance when it starts getting too warm. Given your idle temperatures, it very much feels like the 3800X is holding itself back (ie. load temps would be much higher if Matisse wasn't as smart as it is regarding thermal management), giving you the terrible performance you're seeing. It definitely sounds like there's a contact issue. I'd check the cooler mounting, redo the paste application, and/or check your fan speeds in BIOS.
Secufirm can be a little bit fiddly sometimes with regards to tightening down both of the spring-loaded screws. If you fully tighten one before engaging the other's threads, the other won't want to tighten. Align the cooler, then tighten a little bit at a time until both screws have barely engaged their threads, then tighten them down until they stop, no overtightening.
Honestly, it sounds like you should just try a better kit of RAM. And if you're lucky, the retailer you buy from has a 14-day or 1 month hassle free return window where you can just return it for your money back if you find a way to live with the Vengeance RGB.
Vengeance RGB has a large heatspreader. If I'm not mistaken, it's a bit bigger than all G.skill Trident models.
Have you stepped up the SoC voltage yet to around 1.08-1.09V to see if that does the trick?
