I also find the notion that charters siphon the best students away to be a false argument. These are students that would have been in private school or enrolled in district transfers to another district. They aren't a loss to a poor performing district that never had them in the first place.
This must vary by area. There are a LOT of charters in my city. I know a LOT of kids in them. None of these kids would have gone to privates or "other districts" otherwise. Zip. I know hmm, I guess two families who use private school. It isn't the done thing in my circle.
There are 100+ students who are supposed to be enrolled in our zoned elementary school of 350ish kids (which is bad, bad, bad) who are not going there, including my own child. This does NOT include kids who are in private schools or in another district (unheard of here--our district is very big and to go to the next one over would never be an option--very rural). No, all these kids are in charters or magnets or their parents have somehow finagled a zoning exemption (rare). They are the kids of parents who have decided to fight for their kids' education. They would probably make a big difference to that school if they went to it.
