I would second the suggestion in getting his outdoor/physical time in some manner that doesn't require you to engage in this dynamic. Whether it is a sport, reg ed PE, or a homeschool PE trade, I would think it gives you a much needed break while giving him one as well.