No, it's complicated! And strangely difficult to articulate well. :-/

I think I get what you're saying here - the racial overtones are not obvious on the face of the expression, because there's a whole other web of cultural references for the word "white". I'm trying to remember where I first heard this, and if I had any confusion about it, but I think it was too long ago for me to really remember well.