Heather, have you done any reading about unschooling? It might give you the confidence to start majorly messing with curricula. Our school kind of does it (on a group level), and while it's a huge amount of work for the teachers it's wonderful for the children. You can take his interests and dive in deep, teaching things like grammar and writing and math in the context of whatever you're focussing on.
At the end of the day, he's not going to end up unable to read or do math, but what you could do is teach him the joy of questioning and discovery, and how to really think about stuff. Your kids are a standard deviation above mine, and mine respond so well to this style of teaching. Of course, maybe all kids do and they just don't have the time and money to do it in schools.
I don't know about other children, but for mine third+ grade work as presented in curricula and other resources is dull. It's designed for eight and nine year olds whose brains work differently to four/five/six year olds. Now that my oldest is the right age for third grade I could see her sitting down and focussing without pictures and stuff, but until now it's been completely inappropriate.
So, yeah, that's the long way of saying I think you should take the leap and just start exploring the world together.
