People are lazy. If you wanted a friend to meet me at the bar, you'd no doubt say,
"he's that big bald guy over there". Notice how you didn't include the word "white" in there. If I was black the sentence would have no doubt been, "he's that big bald black guy there". Now, no harm is meant with a sentence like that, but it kind of illustrates my point that we're never going to be truly colorblind.