My DSalmost7 was also extremely verbal from a very young age, but he wasn't what I would consider a "mathy" kid. He knew how to count really high early on and could read a calendar early (both by about 3), but nothing amazing, just what I would consider "regular gifted". When we brought him home from Kindergarten half way through the year last March, we started with 1st grade math. In literally a single lesson, he went from being able to sort of add single digit numbers to adding any size numbers he wanted, including carrying. In 6 months, he blew through 3 years of curriculum. His calculations are still a bit slow, but he gets there. He's probably conceptually working even higher, but until he can calculate more quickly (which for him, I think is a developmental thing) we aren't moving forward to find out.
He's definitely good at math, he's just not self-taught and it's not really his passion. If we had only gone by what the school was doing, we never would have seen what he could do.
There are some kids who get the stuff very intuitively early on, there are others that take a concept and run with it once they're shown. If he enjoys it, work on it with him and see what he does- he might surprise you!