One of the things my youngest and I enjoyed (found it purely by accident) was metal detecting. We'd look for possible interesting sites together, always fun seeing what you can dig up of course....but it's what we dug up that was the academic part. Figuring out what it was a part of or the history behind what we dug up, looking up the value of coins, etc. This might be a good activity for that "neglected" child (our second one was in some respects the same) and the together time is of course highly valued as well.
I've also read about some 1-2 week programs of archaeology for kids programs that either just the child or the whole family can be involved in. There are a lot of local geology clubs as well.
If you're looking for home challenge math or simply to explore math in greater depth my recommendation for elementary age are books by Edward Zaccaro. He presents math in a fun way with good practical application as well.
