We do independent study and Wolf (5yrs) is normally with 4th graders (11 yrs) on our field trips and other than the size you couldn't tell the difference. It doesn't matter at all to the boys, but that is one of the bonuses I've found with homeschoolers of any type, age just doesn't seem to be as big of an issue for the kids. I've seen teen boys honestly playing with elementary kids and having a ball.
He's also in Cub Scouts with boys 1-1.5 years older than him and once again, no difference. These are conventional school kids, all ND, and other than a comment or two about the little guy knowing more than them once or twice there haven't been any issues.
I say see what he thinks, talk to the teachers, see if they would be a good fit and understanding of the difficulties he may have. If academically 5th is a better fit, odds are he'll do well there socially as well, although there might be some hiccups at first.
