If your son is gifted, the reality is likely that he can miss that much fourth grade math and writing time without it affecting his grades. If the school has decided the trade off is worth it, I wouldn't worry about the missed class time. They have certainly considered this and decided it isn't an issue.
It isn't too early to work with your son on focusing on the joy of learning, discovering new things and not on being perfect. He will pick up on your worries about this, so be positive about it and he will adjust. If you notice he needs extra work on writing or math, that can be covered easily at this level in a short time at home.
As parents, we need to be careful, imo, about not labeling our children with traits such as "perfectionism," because such labels can become self-fulfilling prophecies.
