They fear brown people taking over (aka the ‘great replacement theory’) and doing to them what they do to brown people.

They also see PoC as lesser and therefore needing to be lower on the totem pole; if they get rights, or even (gasp!) high-level jobs then they move up the totem pole, which is a violation of the ‘natural order’. And they believe that if the natural order is violated, society itself will collapse.