later I would understand that seeing me as white was the only way someone like him could ever accept a minority as a friend.

I wonder about this. Without condoning racist jokes or belittlement dressed as a compliment, isn't part of acceptance being able to see yourself in another?