I had a very similar experience with my son (now 5). His music class only had a handful of kids, lasted about the same length of time, and had a wonderful and very understanding teacher. My son (he was three at the time) used to through fits about going, but loved it when he was there. Like yours, he likes predictable routines, and there were rarely all of the students in the class each week.
I can tell you that as it was the first thing we did where he was away from family, I would encourage him to go and reward him if he went and stayed for the class. I told him he could leave, and I would take him home, but then there would be no reward. We left without the reward a couple of times (and he was upset, and wanted to go back, but I didn't want him to feel that he could come and go as he pleased in the class). Sometimes it was absolute heck (and I felt guilty, but I also knew if he had no exposure to being away from us, school would be horrid for him), but I persisted, and the day came where he went in happily.
A lot of sensitive kids have hard times transitioning from one activity to another, and especially when they will go into an environment where it is unpredictable. Even though it is very laid back from an adult perspective, for a child who is sensitive, it can be a bit overwhelming. That said, unless he is totally miserable (and as you said, he enjoys it when he's there - he's probably experincing a type of anticipatory anxiety rather than disliking the actual class), learning to deal with this may be a good thing.
