I think the political externalities of immigration are greatly overblown.  This piece by Chicago Public Policy Ph.D. student Tino Sanandaji presents the other side more effectively than anyone else I’ve read.  I’m too busy to reply for a while, but enjoy.  The heart of the piece:

…Berggren and other libertarians and liberals who
rely on the Alesina-Glaeser theory are substantively wrong. Ethnic
diversity overall tends to expand the welfare state, not reduce it.
While the research only focuses on one effect of unskilled immigration
(reduced fellowship), there are at least three effects that go the
other way. Here are the main effects of increasing the share of low
income minorities:

1. Solidarity
is diminished and social ties are wakened, so that the majority
population becomes less willing to pay taxes to help “the other”. This
limits the size of government. The
ethnic-diversity-and-redistribution-literature has almost entirely
focused on this sole effect.

2. Increasing
the share of low income individuals increases the welfare state through
a mechanic effect. This means even if you don’t vote any changes to the
welfare state, the use of preexisting welfare programs such as
unemployment insurance and public health care increases…

3. More disadvantaged citizens increases the need for
a welfare state. To the extent that the welfare state reflects a desire
to reduce social problems, having more deprived individuals increases
the demand for more government to solve problems…

4. Though
ignored by proponents of the ethnic-diversity-and-redistribution,
minorities also get to vote, and they vote overwhelmingly for the left.
This effect is dominant when we are discussing free migration, because
with open borders in a world where 700 million people have told Gallup they
would like to migrate right now, sooner or later the immigrants will
become the majority of voters and make the political preferences of the
natives irrelevant.

[…]

The proponents of the Alesina-Glaeser theory tend to focus entirely on point one and ignore points 2, 3 and 4.

HT: Dan Klein