I too say worry about the college later and look what you want for him right now. If it's challenging environment then it looks like you either need to go with a grade skip or homeschool. The skip to the 3rd grade sounds like a good idea, like Dottie says you still want to have your child in the top 10% of his new class. You can always move him to the 4th later on.
What does DS think about it? Does he want to go back? Does he have friends in school he would miss? What does he think about the other school? Would he like to have more time for himself?
Kriston has a good point about summer and the kids wanting and needing a break.
Good luck with your decision.
