I can see the points about him possibly getting hurt in sports with larger kids and possibly not needing to be exposed to the "older" ideas that the kids in his class discuss. However, my DS7 has always had a hard time with kids his own age because they simply don't understand him and they are mean, as little kids can be. I would look forward to getting him into play situations with the older class, in your place, for that reason. He gets along much better with kids at least a couple years older.
Some sports are more size-neutral than others. Soccer was one sport I played with much older classmates where we were pretty level once my fitness caught up after a few games. I played pickup soccer every day during lunch as well as during gym time. I did get hurt, but I never got injured if you can distinguish the difference.
