So everyone agrees that we heroically help some, and leave others to die. We only disagree on who falls into which category.
What most non-libertarians favor is some form of collectivist policy at a national level. National socialism, to coin a phrase.
People are naturally collectivist at the level of family and others in their immediate vicinity. And that's fine. But to libertarians, national socialism is a mistake. The way to behave ethically with distant strangers is to trade honestly with them and don't steal their stuff. There is no reason to treat distant strangers that live within one political boundary differently from distant strangers that live inside another boundary.
Now, it might be good for some of us to be particularly generous to some distant strangers. But libertarians would argue that this should be done voluntarily, and that it does not justify employing the coercive apparatus of the state.