Easy answer to 'what are you worried about' is - how is he supposed to learn a good work ethic if he never has a chance to work at his readiness level? Isn't the chance to develop a good work ethic one of the most important things we can give a child?
OK here's a Comic that I saw flash in my mind this morning:
Parent is visiting a school, sees a 3rd grader in the hallway writing 'X's on the kindergarden door with Chalk.
Visiting Parent: Why do you keep writing Xs on the door to the first grade classroom?
3rd grader: My mom says that it's Obscene that my Brother is spending hours a day practicing phonics when at home he reads 'Harry Potter.' So I thought since it's Obscene, I should rate this classroom, triple-X.
Visiting Parent: ((Blank look))
