Dithering is simply adding a low level white noise to the sound before it's converted (from floating-point) to the output resolution (8/16/24/32-bit). Adding noise may seem silly, but it actually effectively increases the resolution because it allows very low level signals to appear in the output that would not do otherwise (eg. they could have been rounded down to 0) - even though the noise level is higher than the low level signal, the brain can pick out the signal. Here's a nice/readable explanation (with examples) of dither...
http://www.mtsu.edu/~dsmitche/rim420/reading/rim420_Dither.html
Noise shaping moves the noise to frequencies that the human ear is less sensitive to, so that it becomes less audible. But even without noise shaping, dither noise is practically inaudible at 16-bit and above anyway
To hear the effects quite clearly, set the output to 8-bit, play an mp3/ogg/mod, and try playing with these options, particularly during quiet passages
