Internal Racism: Btw Africans and Black (African) Americans.......?

Here's the situation. I want to know if I am wrong for being "neutral" and not belonging to "clicks" at the work place. I am a Nigerian-born who grew up in the United Kingdom before moving to Atlanta 13 years ago. I would consider myself an open-minded person. Before moving to the states, I never had the opportunity to meet alot of african americans. So I was excited about moving here and meeting my long lost brothers and sisters (or so I thought). However I noticed from my years of being in different work places that African Americans were quick to tell me to "go the h#ll back to Africa" at the drop of a heart beat and say many degrading things about Africans. I have experienced this alot, but ironically when a racial issue breaks out at work the slogan becomes "all black people are the same." Consequently, I remain neutral wherever I work. When I see a grp of black people forming a little click, I back off. I am sometimes called anti-social bcos I mind my bizness, am I wrong 4 this?