My first thought was Nvidia, but only because I usually build AMD powered PC's for people, and probably 1 in 2 people running AMD drivers has run in some kind of problems related to drivers, while I dont know many people with Nvidia cards with problems(but also not many people with Nvidia cards in the first place).
The most annoying problems while using AMD included:
Catalyst not wanting to run untill I completely wiped the drivers, and then manually deleted the registry keys.
Flash randomly crashing the PC(or in the case of the 7950 giving a grey flickering screen which requires a reboot) because of the weird clock profile that flash uses which lowers GPU memory clocks after the GPU memory got OC-ed.
I cannot vouch for the quality of Nvidia drivers though, but in the short time I used the Nvidia drivers they bluescreened my PC in LoL when a champion performed a specific move(very weird, Lux with light binding in this case), and randomly when I tried to watch a movie or use flash while using a 3d application(most annoying aswell).
In the end i'm not so sure, they both do not work perfectly. I wish there was a poll option "They all suck"
, but I guess that would defeat the purpose of the poll.