Your son sounds so much like mine did early in his education (he's 12 and in middle school now).
I wish I had known then what I know now...that my son is 2e (twice exceptional). He is gifted, but he has specific learning disabilities that mask his giftedness (and vice versa).
Although he was doing 3rd grade math when he was in 1st grade, he started running into more and more difficulties (with behaviors and his auditory processing disorder, working memory problems, etc) that prevented him from achieving what he could.
Now that we've pinpointed his troubles, we're able to work with the school on an IEP (Individualized Educational Program) so that he gets the accommodations he needs to succeed. We missed so many opportunities! He's now in a G&T program that, while accelerated and difficult, meets academic needs while we (with help from the teachers and his IEP) address his LD.
I would highly recommend that you ask the school psychologist to do testing (WISC IV) that will show his areas of strength and weakness so that you'll have a better overall picture of his learning abilities.
Good Luck!
