You should do what our Independent Study teacher did with us when I was worried we weren't doing enough "school."
She sat down with me and made me list EVERYTHING we did each day that was educational. That includes all the discussions we have, the educational TV and computer games he plays, free reading time, physical activity, social interactions with other kids (like Cub Scouts and play dates), reading to him at bedtime,even imaginative play time like legos or pretend play, etc...
Once you actually get it all down on paper you will probably be astounded at how much time your kid is actually spending learning or doing "school" related activities each day. I know I was!!! We sit down and "do school" for about an hour or two each day. That includes math, root words, phonics, Mad Libs, logic, spelling and some science and social studies. Art happens when he's interested, music is in the car, typing is when he wants to and Spanish is when I have the initiative. He also reads specifically for school a little each day. I thought we were missing something and just not doing enough "school." When we wrote EVERYTHING down it turned out that he was spending between six and eight hours a day doing school related things, well over the four required hours.
The reading and exploration your kid is doing more than likely count as school. Some of the playtime probably does as well. Take some time and really list everything educational/school related he does. It should be eye-opening.