You have him with you during the day, but there must be activities in the afternoon and evening.
I would not feel bad about enrolling him in after-school activities where you can drop him off. Find something that lasts for more than an hour and parents don't have to attend.
I wouldn't even worry that much about it being something he loved. Its good to get use to different situations. Library volunteer, chess club, a sport, tutoring center where he can tutor other kids, youth groups, etc.
Then you can have some time to yourself in the afternoon and evening.
