Chait says yes:

The most fevered opponents of civil rights in the 1950s and 1960s – and, for that matter, the most fervent defenders of slavery a century before – also usually made their case in in process terms rather than racist ones. They stood for the rights of the individual, or the rights of the states, against the federal Goliath. I am sure Paul’s motives derive from ideological fervor rather than a conscious desire to oppress minorities. But the relationship between the abstract principles of his worldview and the ugly racism with which it has so frequently been expressed is hardly coincidental.

If the connection is "hardly coincidental" then how are Ron Paul's motives not racist? Jon wants to be fair to libertarians and libertarian conservatives, but in his gut he knows we're all bigots. By Chait's reasoning, for example, I am profoundly homophobic. I oppose hate crime laws on the basis of freedom of thought and speech. In Virtually Normal, I even argue in principle against employment discrimination laws for gays. Does that mean I cheer on bigoted employers and gay-bashers?

Yes, says Chait:

[Libertarians] genuinely see racism as a belief system that expresses itself only in the form of coercive government power. In Paul’s world, state-enforced discrimination is the only kind of discrimination. A libertarian by definition opposes discrimination because libertarians oppose the state. He cannot imagine social power exerting itself through any other form.

I can. The social power of homophobia and hetero-sexism in a free culture is crushing. I oppose it; and recognize it. I have spent a great deal of my life pushing back culturally and intellectually and morally against it. But I do not want to compel it into submission. I want to persuade it into toleration. And that is the core difference between power exerted by the state and power exerted by non-state actors: the former is ultimately backed by physical force deployed by the government; the latter by public opinion, economic and social power, and the willingness of minorities to buy into the ideology of their oppressors or haters.

Because I believe in civil equality as guaranteed by the Constitution, I insist on absolute neutrality by the government, which is supposed to represent all citizens, but I would allow for private hostility, bigotry, hatred and even discrimination in civil society. Because our laws are so riddled with identity politics, I find it practically impossible to oppose employment discrimination laws for gays, when every other minority is included in them. But I stick by my principles in Virtually Normal, and I do not believe it makes me a gay-hater or enabler of gay hatred.

Process matters in a liberal society (though not, of course, in a leftist or theocon one). It is not a  bad faith argument, even if you differ. And yet the accusation of bad faith creeps in:

[I]n the absence of government protection, white males, acting solely through their exercise of freedom of contract and association, have historically proven quite capable of erecting what any sane observer would recognize as actual impediments to the freedom of minorities and women.

Since I am a privileged white male, is my alleged dedication to the principles of liberty merely a mask for my desire to get and keep mine and screw the rest? That's Corey Robin's core argument about conservatives in general: in the end, all it is is the maintenance of power.

But what when this country is more multi-colored than white? Or when women outnumber men in the workplace? If I change my position then, and start using the government to ensure white male supremacy, Chait and Robin will have a point. But I won't. Since my principles do not make distinctions between citizens as members of variously oppressed groups, they endure through a future far more culturally and morally complex than the one we have now. Libertarianism anticipates this complicated future and understands that the state will have to become lighter not heavier if the social compact is to survive. 

Noah Smith counters:

The modern American libertarian ideology does not deal with the issue of local bullies. In the world envisioned by Nozick, Hayek, Rand, and other foundational thinkers of the movement, there are only two levels to society – the government (the "big bully") and the individual. If your freedom is not being taken away by the biggest bully that exists, your freedom is not being taken away at all … this gigantic loophole has made modern American libertarianism the favorite philosophy of a vast array of local bullies, who want to keep the big bully (government) off their backs so they can bully to their hearts' content.

There is no question that this is the case. Bullies love freedom. But so do rebels and prophets, who would be silenced without it. And the right response to bullying is to stand up to it, while the government diligently ensures that the free market is not dominated by monopolies or rentier classes; and that core political liberties, such as freedom of speech, are protected everywhere.

Freedom means evil, as well as good, will flourish. And a conservative, properly understood, is someone who understands that evil is eternal, but that evil backed by a monopoly of physical force is the only one that can be restrained by a political order, without undermining the freedom it requires to breathe and grow and think for itself.