There are a few more issues in sharing RAM between CPUs which seem to be really ignored, but I'll concentrate on one (I guess it should be obvious for physicists / electronic engineers):
Speed of electric signal is finite.
We are used to the idea that it is so large we don't have to think about it.
But there really is a reason why RAM slots are so close to the CPU, that we are getting coolers that block a RAM slot.
Just to give you an example on some very rough numbers, lets assume that:
- the distance in wiring between CPU and the "other" RAM is 20cm (possible on a large dual-CPU board),
- the speed of signal is 2.8 * 10^8 m/s,
- there are no additional slow downs.
The signal would need around 1.5ns. That would give our new and shiny DDR4-3000 latencies of DDR2-533. Meditate on that for a while.