What exactly is he doing now in terms of speech? Does he answer questions like "Tell me something you enjoyed aobut school today?" Does he ask you questions? How many words can he put together into a sentence? Is his speech mostly grammatically correct? You can look up developmental milestones on this site:
http://www.pbs.org/parents/child-development/You can click on language and his age.
I think it is common for many total normal kids to be late talkers, but from what I have read, if a child is still having difficulties in kindergarten, it's less likely they are just going to "grow out of it."
DS had the huge gap between non-verbal ability and verbal ability at age 3, and there was still a gap on the most recent testing (cogat) but he was in the superior range (I think) in both verbal and non-verbal. The bigger the discrepancy, the more likely there is a 2e issue. I do think that people can be mathematically gifted and not gifted (or as gifted) in other areas, but it's a matter of degree whether it's normal or not. FWIW our district now requires kids to be gifted in both verbal and mathematical ability (they don't seem to care about non-verbal), because those are the kids that do the best in school gifted programs (doesn't mean they are more gifted, they are just more likely to be the high achievers in terms of school).
