What does your DS have to say about it? Are you seeing any issues developing in him?
In general, he likes to blend in, which may not be good for him long-term. When he is not challenged, he gradually starts to act out.
It’s hard for me to foresee how my actions or inactions will affect a 6-year-old in the future, especially he does well in school.
It is, as you know, impossible to predict the future! We have found it helpful to research best practices for educating gifted children (based on evidence, wherever possible). In our son's case, we also saw the downside of inaction. In first grade, he was forced (there is no other good word) to sit quietly while the rest of the class learned what he already knew or had grasped very quickly. He became the class clown, was constantly in trouble and started hiding from me when it was time to go to school. So we knew we had to act. We now know, based on testing, how highly gifted he is and wish we had acted decisively at the beginning of that school year (we tried, were rejected by the principal, but should have pulled him from there posthaste).
I wouldn't say that we are obsessing over trying to get him to reach his full potential, but rather, that we are trying to balance things out so he has at least some learning that happens at school.
It is unfortunate that our school system is so focused on age, rather than letting children move through subjects as they master them. This forces us to do things that are against the norm or try to find another way to allow actual learning to take place (isn't it interesting that schools claim social over academic progress as their mantra?!). As an alternative example, my mother and her siblings attended a mixed-grade (K-8)rural school (as in one large room). They were able to progress based on mastery because the teacher was constantly grouping and regrouping students to cover different topics at their educational levels. That seems so much better.
