Do you have a copy of the test results from the charter school? If not, I'd get a copy of that if at all possible, to include with your records. Even though he didn't test as gifted, the reason you want a copy is to know what specific test was administered, and how he scored. There can be many nuances to testing - most schools don't actually administer IQ tests for admissions, but instead either a type of test which is categorized as "learned ability" or an achievement test. IQ tests are referred to as "innate ability" tests - they assess skills/abilities you are essentially "born with", whereas "learned ability" tests depend to a certain extent on the environment a student has been in and learning they've been exposed to. One example of a "learned ability" test is the CogAT (I doubt that's the test your ds had for kindergarten admissions, but just using it here as an example). One of the questions my ds was asked on the CogAT referred to determining what a metric measure was based on a different metric measure. At the time he took the test, ds hadn't learned anything at all about metric measurement. He made a very intelligent guess about what the answer was using logic and reason, but it was the wrong answer. A less "higher ability" kid might not have had the ability to use logic and reasoning in the same way, and might have guessed at the answer and gotten it correct. Again, this is just one example question on one example test, but the idea in general is, an innate ability test should, in theory, give you the better estimate of your child's true intellectual ability.
The other thing to think through - you've mentioned your ds isn't very interested in conversational language. When a child is "interviewed" for school admissions (granted, young children aren't directly interviewed, but if there was a situation where he was in the room with a person from the school etc)... some people will make inferences from conversation re whether or not a child is gifted. Not talking or talking too much or talking too deep or talking about fluff - none of that is really a reliable indicator of true giftedness (outside of conversations that reveal depth of thought). All children are different and children also behave differently in different situations. So - it's perhaps possible that there was an element of the previous school assuming your ds wasn't gifted based on conversation with him.
The third thing you need to do is gather evidence of the work he's capable of doing - math problems, lists of what he's reading, whatever you can briefly use to show someone else his current academic levels.
If you haven't already done so, I'd research all the options available for education in your area - and that includes schools that aren't necessarily labelled "gifted". Oftentimes the best solution is a school that is willing to work with you and your ds, irrespective of whether or not the school is public/private/charter and whether or not it is specifically set up as a school for gifted students.
Last thing, I'll second/third the recommendations for IQ testing. You might be able to get the testing through your public school - but again, I'd check first to be sure what type of test they administer. If you can't get a full IQ test through the school district, then I'd seek one out privately. If you have to go private for an eval, find out what types of tests your school district and/or other schools you are interested in will accept for admissions to gifted programming.
And get ready to advocate. Chances are you'll be doing a bit of that, particularly in the early elementary years. One thing that kept me going a long time ago was running into a young person working at the kids' area in the health club I was working out at. When I went in to pick up my children I had a book about gifted kids in my hand, and she asked me why I was reading it. I mentioned something to her, and she responded to keep on advocating and stand up for my kids at school - she said she'd been a gifted kid herself and that the school she was in was not providing what she needed, but her mom fought fought fought for her to get an appropriate education - and that not only helped get her what she needed at school, it also gave her a tremendous respect for her mom seeing her advocate for her to other adults who were in a position of power. It can be tough in the trenches, but definitely worth hanging in there!
Best wishes,
polarbear
