You own the place, but you are biased due to your emotional response to trans discourse and the people you hang out with, you are far from neutral, the fact you are so emotionally invested is what makes your judgment hard and innacurate.
Trans people NEVER infiltrated the community, they were there from the start, whoever thinks otherwise is a tourist and you don't lose anything banning them, people hating on others for their race and gender is deserving of a permanent ban in this specific imageboard.
On the other hand, you fail to realize that some people in that thread, me included, were seeing the OP as a legitimate troll pulling some typical race baiting discourse, disagree? lock the thread as it's too controversial because of its wording and ongoing debate, NOT BECAUSE OF THE CONTENT, I am all for nonwhite characters, but I suppose you just assume bad faith for all cases that don't go exactly with the way you think. To be fair with you, people acting like /pol/ blending on my side of the argument doesn't help you.
Don't get me started on the recent NEETpride thread, that lacked moderation too on all sides.
You need to be cold to manage one of those sites, that's the harsh truth, I've been here for over a decade so I obviously like the site, but if it's unhealthy for you, pull the plug until people with cold blood help you or become numb to it's insanity.
If we disagree or not it doesn't matter, what does matter is that your health needs to be first.
I know all of this is harsh of me, so I will give you nicer advice.
Make it more obvious where the imageboard stands in terms of ideology and userbase to discourage bigoted people from trying to blend in.
You can achieve this by just adding a short but visible description in the homepage right where "Don't touch the lights!" is going like "We are a Yume Nikki fan site blah blah, we have been around for a while blah blah, we gather NEET/Hiki culture and encourage mental health and are accepting of LGBT"
This is not gonna be a magic solution, but if it saves you from one /pol/ tourist, it does the job.