A little checking turned up the following information:
1. Ellis Island was only established as a federal immigration point--the first such--in 1890.
2. The first federal restrictions on immigration were passed in 1875; they excluded criminals, prostitutes, and Chinese contract laborers. Congress passed the first general law restricting immigration in 1882, banning immigration from China. In 1917 the restriction was extended to immigrants from other Asian Pacific countries. Numerical immigration quotas only came in in 1921, but did not apply to immigrants from Latin America until 1965.
3. While it is hard to be certain, it does not sound as though there was any effective mechanism for enforcing restrictions in the early period. That was obviously true for immigration across land borders, and I do not think there was any enforcement mechanism covering all ports that would have prevented someone from simply walking off a ship and blending into the local population--easy to do anywhere with a significant group of the immigrant's ethnicity.
David points this out to address a commenter who wrote, ""Having open borders renders the entire concept of a country meaningless." After pointing out these facts, David writes:
My conclusion is that if the concept of a country is meaningless without at least nominal restrictions on immigration, the U.S. only came into existence about 1875.
Correction: That Ellis Island was the United States' first immigration station seems to be inaccurate--the first one seems to have been Castle Garden (also called Castle Clinton). HT: Alex Nowrasteh and Vipul Naik.