I think a little chat with teacher to 'keep them in the loop' is a good idea. I would also spend time observing a 4th grade gifted program and see if it's anything that would challenge and interest your son, that will prevent you from putting all your eggs in the wrong basket.
How old are the siblings? Having a close enough in age older sib is a great comfort to 'outside the norm' kids.
He is now in second grade and recently shared with me that school is getting in the way of his way of learning.His reason for going back everyday his 4 best friend since kindergarden he would miss them to much.
I'll bet you a dollar that the 4 best friends are also in need of further accommodation and perhaps it can be arranged for all of them to spend time in the gifted 4th grade math or reading room for part of the day.
I would try requesting, in writing, an IQ test, now, for your son. If you school has a gifted program, then you probably also have a gifted coordinator. Try do convince her or him to spend some time with your son and see what is going on. If that doesn't work, I would look around for a tester who is familiar with kids 'who teach themselves to read 'such and such' at age 3' ((what constitutes reading varies quite a bit from person to person))and pay for a test out of pocket, as insurance. It might be worthwhile for you to find out how your school accepts kids into the gifted program, to let the tester know.
I'm glad that your son is having fun and behaving, but that doesn't last forever in SOME kids, and a little insurance goes a long way.
Best Wishes,
Grinity