If a black man says "I'm proud to be a black man" it seems reasonable to most people. If a white man says "I'm proud to be a white man" people think he's part of a prison gang.
I agree that that is unacceptable. Neither stance makes sense to me, but you can be proud of a heritage or culture, but having a particular color of skin doesn't make you automatically part of any culture...it has to do with upbringing or adoption may be, but birth race really has nothing to do with what is a social phenomenon.
If you are proud of being part of a culture, that seems completely reasonable. May be we don't call white culture "white culture," some refer to it as "American culture." Perhaps that's what you want?
I think you are oversimplifying. Look at Black Lives Matter. It is an example of, If a black person says "Black Lives Matter", many white people say "you're racist."