He sounds a little like my dad. He probably assumes your son is normal and all the other kids are deficient or got no stimulation as infants.

My family believes my dad, and several other relatives, have Asperger's. He truly doesn't realize he's supposed to show an interest in my kids, or how to go about doing it. My stepmother has basically put him in intensive social therapy for the last 20 years and he now makes an effort, but it always seems forced.

I obviously don't know your dad, but it might make you feel better to think along those lines. Some folks are heartless, but some just don't know how to connect with others.