Yes I think you guys are right. He does seem to disturb and bother others when he is not medicated. He gets into time out more and is generally causing us as a family to have more arguments because of his behavior. I don't think the medication is too high of a dose because he still acts impulsive and loud, but just not as bad. The medication tones him down but he is still not a kid with great behavior. He still gets in trouble at school at least once a month, but much better than once a week when off medication.
Another reason we try to give him medication breaks is his lack of appetite and he is already very thin. But with medication he can't sit still enough to finish a meal. He will constantly get up and get distracted by something.
Thank you everyone who responded, I really appreciate all the different perspectives.