It has to do with which light frequencies or colors are absorbed vs. which are reflected.
I can't give an exact definition of how it works (it's been a while since I read up on it). If I recall properly (which I'm not positive I do), white (or light colors) reflects too broad a spectrum of visible light, so it basically gives too much information for your eye to process. Darker colors absorb more light, but reflect back a range of color that though less than what white reflets, provides more clarity to the human eye.
At least that's what my memory says, I read up on these things as part of a photograpy project several years ago.