What I don't know is whether he has all the problem solving skills that would be taught (and skills in "showing your work" and so forth). And there may be other gaps I haven't envisioned.
I believe it's reasonable to expect that a significant portion of his 5th grade classmates, who will have been shown those skills, will not have mastered them so well that they don't need review. If your DS can pick up new things after seeing them once or twice, I wouldn't think there would be a problem there.
There's a lot more written work, even in math, in the higher grades, though. I know for my DD, the increased writing volume is keeping her from being bored, even though the material isn't necessarily challenging to her. I think that a second year's acceleration would be too much writing for her, although she's a year younger than your DS, and a year makes a big difference at that age.
