To call anarcho-capitalism a controversial proposal is gross understatement. But the most common objection voiced by the economically literate is that anarcho-capitalism would quickly decay into monopoly, whether through war or merger. And it's a short step from a monopoly defense firm back to government.
The standard anarcho-capitalist rebuttal is to compare the scale economies in the market for defense services to demand, and see how many firms the market has room for. If there is only room for three firms, it's plausible that they might merge to monopoly and become the new government. If there is room for ten thousand firms, it's totally implausible. As David Friedman wrote in The Machinery of Freedom:
If there are only two or three agencies in the entire area now covered by the United States, a conspiracy among them may be practical. If there are 10,000, then when any group of them start acting like a government, their customers will hire someone else to protect them against their protectors.
...My own guess is that the number will be nearer 10,000 than 3. If the performance of present-day police forces is any indication, a protection agency protecting as many as one million people is far above optimum size.
But there is a simple rejoinder to Friedman, which one of my best students hit upon: If scale economies are really this weak, why have states emerged and remained stable for thousands of years?
It's an awkward question, but I've got an answer:
Thousands of years ago, demand for defense services was very small relative to scale economies. In any locality, the market only had room for one defense firm. The result was just what the skeptics would predict: Private monopolies quickly turned into governments.
As economic growth progressed, of course, the market for defense services got bigger, making room for more and more firms. The problem, however, is that if you've got government in an area, it has the power and the incentive to prevent new entry by competing defense firms. Thus, if market conditions initially favor monopoly, monopoly can easily endure due to "lock-in," or "path-dependence."
This remains true even if current conditions are totally different from the initial conditions. Initially, there was room for one firm; now, for 10,000 firms. But the one firm that got on top isn't going to let the market respond to changing conditions. It's going to use its advantageous starting point to terrify potential entrants away.
Thus, my answer to the student's challenge is that scale economies are weak now, but things used to be very different. States emerged at a time when markets were too small to sustain more firms. Over time, the economic rationale for monopoly has grown weaker and weaker. Competition could work now, if you gave it a chance. But the state doesn't care about economic rationales. As long as it can credibly threaten to put new entrants in jail, its monopoly endures.