I can't really sympathize. We don't really get out much. We're going to a ranch to watch them shave cows tommorrow. Yeah! We got something to do !!! We do have story time at the library, but only for two weeks in the summer. Ha! The first year he wouldn't sit with the other kids unless I sat with him. Last year (at 3) he sat with the other kids, but I had to keep "shushing" him and telling him to listen to the teacher. We have one more summer story time to practice then this fall he's off to school. Yep, we're screwed too. He's mastered the art of getting other kids to play with him and that's what he thinks kids are there for.
If u were me, I'd just work with my toddler on the development I think he's at, enjoy the activities, smile at the ladies, and assume the best- that they see you as a mother actively parenting her toddler the best way you see. If they really think anything else, well, what do you want me to tell you? You ARE teaching him about circle time and paying attention just by taking him there regular. Baby step.
Maybe you want to talk to him about the patterns. You've said he talks to you a lot. Even if nobody else understands what he's saying they'll see there's a conversation happening. Try to coax him into describing the coloring he's doing. You don't have to tell him to follow their pattern, if you're talking to him about his coloring you're still doing the lesson, right?