People with extroverted kids definitely have a tougher row to hoe with getting their kids the social time they need. But it can certainly be done. If you chose to go that route, you might just need to develop a weekly (or more!) playgroup that meets for "recess." 4-6 kids and someone's playroom or back yard or a park might be all you need. If you can find someone to trade time watching your kids with or find a sitter, that might give you some quiet time. Life with an extrovert is rarely quiet, and if you need quiet, you'd have to find a way to get it.
I'm not saying you *should* homeschool more fully. I can't answer that question. I think you have to decide what would work best for your child. It may well be possible to advocate and get what your child needs from the school. It seems like they accept that your child is smart, but that they just don't currently agree with you about what his needs are. That's potentially a good place to start advocacy if they'll listen to you (or to test scores!).
The math teacher is the one who worries me: conceptual understanding of math and speed are two very different skills that often develop asynchronously. (I have seen this with my son--he handles pre-algebra but does his times tables as slowly as any 7yo.) For a GT child, the speed often comes organically out of greater conceptual challenge. They then have a reason to want to work fast. With a highly conceptual/low speed kid, drilling for speed is worse than drilling teeth! Finding more creative ways to cement the learning is often useful.
I think I'd try advocacy with the school first if you think that might work. You can always homeschool fully later. If things are going south with the advocacy, you can start putting together a playgroup ahead of time so that you have things already in place for when you pull him out.
