Quote
then he goes off by himself for 45 minutes or so when we come home. He is an only child, though, so that may make a difference?

This is exactly like DD6, not an only child. I was like this as well, as a child. I have 2 siblings and TONS of extended family.