I resent my tax money insuring banks that make sub-prime mortgage loans, or risky construction loans. I'd like to ban FDIC-insured banks from making housing loans with less than at least 20% down. I am not opposed to allowing sub-prime loans, just not with FDIC-insured money. If some unregulated financial intermediary wants to make such loans, that's fine with me.
There is a classic proposition in economics called the theory of the second best. One way to think of this theory is that it suggests that when one economic distortion exists, removing a second distortion can make the economy worse off. If the second distortion helps mitigate the first distortion, getting rid of the second distortion can be a bad idea.
I think that is what is going on in Sumner's post. He suggests that easing restrictions on banks might be a first-best policy if there were no deposit insurance. However, taking the distorted incentives of deposit insurance, easing restrictions is not a good second-best policy.
For me, it is easier to understand his argument in these terms. Instead, he tries to play with the semantics of what is "regulation" and "deregulation," and I think this is confusing.
My belief that government should break up large banks is also a second-best argument. If there were another way for government to commit credibly not to give bailouts or other political favors to large banks, I would be for that. But given that the existence of large banks is an invitation for politicians to provide them with favors, I would rather see bank size restricted.