Ours happened at age 3.5 too.
My father died suddenly. He said good-bye to DD in the typical way he did each week but then never came back - he collapsed on a subway platform a couple of days later. DD was d-e-v-a-s-t-a-t-e-d. No one could tell me what to do for her because nothing about her response was typical for her age. She *got* the whole thing and understood the permanence in a way no one who encountered her had ever seen in a child her age. It was awful. And then 6 weeks later, as we were beginning to see some small signs of improvement, his dog - who DD considered one of her own pets - was hit by a car and killed while we were away on vacation. That was it - all the emotion went away and she just shut down. It was clearly too much for her to deal with.
We enrolled her in a grieving children's program even though they thought she might be too young. Her super high comprehension and verbal skills made this "talking about it" type of program really good for her. She talked about grandpa and Whiskey there. She brought pictures of them and told stories about what they did together and was able to express how much she missed them. They read books about death and it was all discussed very concretely. No heaven, reincarnation, etc. It was very factual - when people die they stop breathing. They live on in your memory but their bodies are no longer here with us. It is ok to miss them. It is ok to be sad. It is ok to be happy when you remember them. Near the end of the program they took the kids to a funeral home so that the ones who wanted to see the technical aspects of what happens when you die had the opportunity to. We did not bring DD on that trip but I heard from other parents that it was done very well and it was the key to helping some of the kids get a better handle on the whole situation.
More than 4 years later DD is still very affected by these deaths. She has lost a house full of pets in the intervening years (DH and I were married 17 years when DD was born so we had a bunch of spoiled rotten, elderly pets who have all since passed...) but none of them affected her the same way. Each one was awful to lose but more for us than for her. None of them died suddenly and she was able to see the deterioration happen as their ends approached. She is 7.5 now and still uses "I miss grandpa" as her go to when she feels really sad or overwhelmed. I think the suddenness of his death and her age combined with her gifted intellect to create a perfect storm.
I don't know if any of this will help since your son is asking more theoretically than in response to losing someone he loves. Maybe speaking concretely about the elderly dog - while it may be difficult for you - could help him to understand better. DD helped take care of our dying cat the last month or so of his life and that helped a lot. She and DH also held a funeral for a dead squirrel that they found in the backyard, Over the next few months she kept sneaking over to the burial site, though, and digging it up. I don't know - maybe that was her way of answering your DS's question of "what happens when they die". HTH
