The way the world works, culturally-white black people are punished due to bad behavior by their group, while white scumbags are able to slide by acting like an upstanding member of society. The classic example is Eddie Haskell from Leave It To Beaver.
White privilege is like how black people speak differently around other blacks. White people behave differently around each other. It's not our fault that other people want our privileges, but our group built them and maintains them through strong policing. When whites enforce white codes of conduct on minorities, they are called racist. You cannot win.
What the progressive SJWs call "white privilege" is simply white culture. It is impossible to oppose white culture without being a racist, nor to eliminate it without removing white people. Restricting white's ability to act like white people will only cause whites to behave more insularly. Most whites in America will extend white privilege to white-presenting minorities. If society turns hostile to white culture, whites will hide their culture the same way black people hide their manner of speech in mixed company.
The answer to complaints about white privilege is to say, "Act more like white people."
When Whites Get a Free Pass
With more than 1,500 observations, the study uncovered substantial, statistically significant race discrimination. Bus drivers were twice as willing to let white testers ride free as black testers (72 percent versus 36 percent of the time). Bus drivers showed some relative favoritism toward testers who shared their own race, but even black drivers still favored white testers over black testers (allowing free rides 83 percent versus 68 percent of the time).
The study also found that racial disparities persisted when the testers wore business attire or dressed in army uniforms. For example, testers wearing army uniforms were allowed to ride free 97 percent of the time if they were white, but only 77 percent of the time if they were black.