The most frequently used ways to check if your overclock is stable are:
CPU/Memory overclock:
Run
Prime95 in torture mode (Options -> Torture Test) and leave it running for a few hours (or longer)...
If you have a multiprocessor or HT-enabled system you'll have to run two instances of Prime95 for proper testing. Launch the second Prime95 instance with command-line parameter -A1.
Don't worry if you get a screen like this - your hardware isn't damaged. Most probably your overclock is just too high.
It's also possible that your system will just crash without Prime95 detecting the error.
Another good test is the
SiSoftware Sandra CPU Burn in test. But be aware that it wont be able to detect errors like Prime95 does.
Memory:
Run
memtest86 or
memtest86+ for several hours at least.
It runs tests of memory reads, writes, copies, and moves in different patterns and block sizes and tests the whole memory range independently of the OS and HDD, PCI devices, etc.
Video Card:
Run
3DMark in an endless loop for a few hours. Signs of instability are artifacts and/or system hangs. Artifacts can be anything from distorted polygons, colored dots and chessboard patterns - everything that looks different from what it's supposed to look like is considered an artifact.
An alternative to testing all-around system stability is just playing your favourite game. It may not be the ultimate in stress testing but you will have a lot of fun while testing.