What about cyberschooling? That could be a good way to transition to a brick-and-mortar school. Your son would have other teachers besides you, assessments and grade levels would be documented and more likely accepted by another school if you decide to transfer later.

You could still hang out with the homeschool group. My guess is he would disappointed with school if he expects to "do really hard stuff" or meet kids like him.