Difficult childhoods in what respect?
Constant exposure to violence or just a lot of obstacles to overcome? Are the actions employed leading measurably to a positive result or do they lead to personal destruction?
This time is a gift for him. He will not always be around you and his dad. He will not always a carefree summer stretching away to forever. He will not have the time to read, to think, to wonder.
And in the future, when life drops a brace on him from nowhere, when the world turns its back, he will be capable of stepping forward and assuming that load without blinking a eye. He won't stumble or fall or collapse. He will assume that huge load - with full knowledge of what it entails and still be able to function. There will be very, very few of his generation who will be able to follow.
If I had a lot of time on my hands, I would begin to write again. Now might be a good time to capture the goofy world he lives in now so that when he leaves it, he will have it clearly set.