We needed some structure to our days, just because it's how we work best (not because I think all 2 year olds do!) so as of his 2nd birthday, DS and I had a theme of the week. It wasn't fancy, but it kept us in library books for a year or so. By the time he was 3 a week wasn't long enough for a topic and we started sticking with one for a month or three (LOL) and got a little more involved with each one. That summer the theme was rockets... and my dad got involved, which meant we basically played with toy rockets ALL SUMMER. Lots of fun though!

We also got some workbooks... He was 2 1/2 when my MIL sent a little "activity book" for him to play with, and he was hooked... so I figured I'd get some real workbooks from real curriculum publishers in hopes that they would last longer. They didn't. But he enjoyed them all and I started realizing that we were going to be just a wee bit out of the Normal Box...

Our first curriculum purchases were the Explode the Code (phonics) Primers, Singapore Earlybird Math, Building Thinking Skills Primary level, and Handwriting Without Tears. They were all good. And I read aloud to him for as long as my voice would hold out. He would have had me read every waking minute had it been up to him!
The only thing I would caution is that little ones can be fickle. Sometimes he was into something and sometimes he wasn't. At 2, 3 and 4 it didn't bother me at all, and I think letting him have that space to control all that kept it fun. I'm much more particular now (he's nine), and there haven't been any ill effects from letting things go back then, or from ramping them up since. Which is to say, don't worry about it. Keep it fun and interesting and let it go if it's frustrating either one of you.