Hi Junior,
four is very young for a child to have to attend school, specially if he doesn't enjoy it. My own daughter didn't start school until she was six (refused to attend K). There is nothing he would gain from a stressful experience at that age except for a great dislike of school.
You say he is already at 2nd grade in reading / math, so clearly his academic needs are being met. You could always find some classes (art, music, Karate) where he can interact with other kids his age in an environment he likes.
Just my thoughts.
