If I had to guess, games can swap assets on the fly to minimize the perf hit when there's not enough VRAM available in rasterized titles, but with RT enabled, the GPU needs access to all the data all at once to calculate the light bounces. And pulling from system memory slows things to an absolute craaawl.
Upscaling can help all the crucial data fit into a smaller framebuffer.
Well let's examine the premise that RT is the root cause of the 3070 FPS issues. Let's take a look at RT in Shadow Of The Tomb Raider, here is the memory usage profile of the game: (Source Guru3D, note that these figures are max settings without Ray Tracing enabled as I was not able to find a chart with RT. TPU didn't start doing it's VRAM charts with RT until more recently. Figure below is maximum VRAM usage throughout 3 runs):
Now a 2060 6GB should immediately cause issues if your theory is correct as given the chart we know the game utilizes more than the card has even without Ray Tracing enabled. Looking at performance though, we can see it's fine:
Cards here including the 2060 are scaling according to their power and as you can see the dip is consistent between all cards when comparing RT ultra vs RT off.
Even if we assume that RT needs access to all it's associated data on the VRAM, it doesn't explain why the game wouldn't just put that high priority data on the VRAM and move low priority data to the main system memory, particularly when you are talking about a mere 35 MB of data in the case of Far Cry 6. Again to me it seems like a bug or extreme oversight by Ubisoft because there's no logical reason for it. You would not put your highest priority data on the main system memory when you can just boot lower priority textures down instead, of which you can just stream in without much of a performance impact. I would normally present more examples but it's surprisingly difficult to find VRAM usage for games with RT enabled while them also including cards with too little VRAM like the 2060 in the RT results. Even TPU's own charts tend to exclude the lower VRAM cards when it comes to their RT charts.
Those reviews all have RT disabled though.
It was to demonstrate the fact that you can go over your VRAM limit without dropping to 1/5th of your FPS. If we are testing the hypothesis that VRAM is the cause, the same game should exhibit the same behavior on other GPUs, particularly one's that are more deficient.
