First of all, realize that whatever decision you make now will not "make or break" his academics for the rest of his life. I worry about lots of stuff (just ask my kids), and I recall worrying about a number of things when they were little. As I look back now, for most issues, the long term success or failure of the kid was not affected by one decision when they were small.
Second, don't feel that you have failed. There will be lots of other opportunities to advocate for him down the road (we've almost made it through the latest one re: AP French, I hope).
If he seems happy, I wouldn't sweat it too much. I think elementary school is easy for many kids in our district. However, most seem happy and well-adjusted. While the kids may be capable of more than what is asked of them, they have opportunities for acceleration down the road (starting in 5th in math for our district). I haven't seen that it hurts these kids long term. The very bright kids end up doing research with university professors in HS, placing nationally in academic competitions, go on to elite colleges, etc.
I have a little one too. She turned 10 this summer and started 5th grade today. She has been accelerated into 6th grade math, but she doesn't really seem too concerned with acceleration, testing for the gifted program (we haven't tested her, older two were tested by school in 7th grade), etc. She explores those academic areas that interest her on her own time, and that is fine with me. We might advocate for more accelerated/advanced courses once she hits HS, but I think it will turn out fine. Eldest is a happy college sophomore and middle kid just started 11th. They are fine despite DH and I not advocating much until MS/HS.
Good luck with whatever decision you make - it will be fine.
