There seem to be two issues here: 1) He's small 2) He's not extremely athletic.

He could be average-sized and still not all that athletic due to the skip. Or he could be small and athletic. (DD has a male friend who fits this description. It's very possible.)

He will be small no matter what, sounds like. So, is this year difference in athleticism worthy of holding him back?