In a modern democracy, not only can a libertarian be elitist; a libertarian has to be elitist. To be a libertarian in a modern democracy is to say that nearly 300 million Americans are wrong, and a handful of nay-sayers are right.
Fair enough. But the converse would be, "to be elitist is to be libertarian," and that is not true. Many important elites, in academia and journalism for example, are supremely confident in the ability of government to "fix health care," "manage the distribution of income," and so forth.
Libertarians are a tiny portion of X, whether X is the general population or the elite. Given that, the question is whether populism or elitism are more dangerous. One can argue, as Bryan does, that populism is more dangerous because the people are really, really ignorant. However, my counter-argument would be that non-libertarian elites may be more dangerous, because their additional incremental knowledge is exceeded by their incremental arrogance. The gap between what one knows and what one thinks one knows may be higher in the ranks of the elite. The result is supposedly-clever government interventions, introduced with excessive confidence, leading to disastrous results.
Just off the top of my head, a good example would be wage-price controls, which the elite were enamored of in the 1960's and finally got to experiment with in the 1970's. Bryan may be a bit young to have as vivid a memory as I do of how that played out.