Been there, been accused.  By now people know the elder son (10) well enough to know that he couldn't really be hothoused without his consent; too difficult.  Still get this regarding the younger.
I think this typically arises because people worry that their kid isn't doing X yet if they see a kid who is doing X precociously.  I try to remember that it's about them.  Not about us.
But to the OP:  that the SCHOOL STAFF would say this, after approving a skip... I'm aghast.  It is, among other things, a violation of FERPA for them to discuss your child's placement and the reasons for that placement with anybody.
DeeDee