I have exactly the same problem with two different vram size 760 cards, but I found a way to resolve it:
I haven't use this program yet, just used the coolbits (set in register with value 18), it worked well in Windows 7 64bit.
- Change OS to Windows 7
- Use latest driver 347.09
- Connect the monitor to the card which has fewer memory
It spend more than 30 seconds during enabling SLI in console panel and I want to try this program in Windows 8.
I tried Coolbits and I could get the cards to SLi in the desktop environment, but as soon as I wanted to play a game it would freeze at the launch stage.