I would withdraw him if it's a bad experience. Just because every other kid loves it doesn't mean your kid will love it, or that it's the right environment for him. We have had some really up and down camp experiences.
It's important to keep in mind that a lot of camps are staffed by high school or college kids who need a summer job. They may have no experience with kids like yours. They are rarely teachers, and a most camps aren't going to run like a classroom. So you aren't really going to teach your kid how to survive in elementary school by sending them to camp.
When DD was younger we had a lot of trouble with camps being too loud/busy/chaotic and with counselors who expected that a kid who could talk like an adult ought to be able to behave with maturity, too. She isn't an introvert but has a lot of sensory processing issues which get pounded on in loud, busy places.
I've found it easiest to pick the "right" camp by calling up whoever's in charge and asking them a lot of questions about who they hire to staff the camp, how many kids are in a "class", what the schedule and environment are like, and so on.
