I can't imagine a 3 or 4 year old who wouldn't fidget or act out when bored. Personally, I think a 4 year old would be better off playing without any instruction than sitting through lessons that aren't teaching him anything anyway.

Is there a way to pull him out and wait until he is more mature before you attempt school? (Of course, that'll mean that he's even further ahead by the time he's in school, but if they're willing to accelerate if he is mature, why can't you just pull him out, wait for him to mature, and then put him back in and accelerate him?)
ETA: I know you said you can't afford to homeschool, but is a play-based daycare an option?