[Caplan] said: "But there are literally billions of lower-skilled workers who would love to move to the First World"
There are hobos who would love to move into your basement.
No doubt there are. And if the hobos were willing to pay me high enough rent, I would be happy to have them as tenants, and U.S. law would not object. In contrast, if a low-skilled foreigner offers me a suitable rent for my basement, and I accept his offer, U.S. law still refuses to let my willing tenant move in.
Now you might say that I'm just being difficult. Of course immigrants aren't going to move into people's basements without their consent; the point is that Americans shouldn't have to live in the same country with people they don't like.
If that's your point, though, I'm just going to be more difficult. It's reasonable to insist that people get your permission to come to your home. It's absurd to insist that people get your permission to live in your neighbor's house* - much less than people get your permission to live in a hundred-mile radius of you. That's on par with the schoolyard bully's grievance that "You're breathing my air." We should see it for what it is - a flimsy pretext for naked aggression.