We "unschool" all the time. Our DS6's interest in particularly in math and what he does at home (for fun) is far beyond what they do at school. Math comes naturally to our DS too. We let his interests drive what he wants to learn. Truth be told, summers are more productive that the school year

When our DS was in 1st we eventually worked out an agreement with the teacher that DS would do homework (for math) that we assigned him. We just went to a local learning/educational center and got a 3rd and 4th grade math workbook. We'd give him a sheet or two from the workbook instead of assigned homework. We're still working on addressing this year BTW. I think the one difference is that our son is very slllllooooowwww with everything he does. Even the easy stuff. The drawback is that no one is really teaching him anything. He was reading the instructions from the worksheet and just doing them. I'd rather he actually be taught.
Some things I recommend:
*
http://www.khanacademy.org/ excellent short math videos
* TimezAttack an excellent, fun math multiplication game
* Murderous Maths books from Ray @
http://www.horriblebooks.com/ total homerun for the mathy kid that can read.
When, if ever, would afterschooling not be appropriate
JB