To be totally honest it wasn't until I saw test results that I finally admitted it.  DH and I both relocated for work after university and 99% of our friends were found through work (everyone has some sort of BSc+) so we're almost always surrounded by crazy smart people.  Coincidentally enough most of them have pretty smart kids so we live in a bit of a bubble.  Admittedly, DS7 stands out even among them but we still never really thought that much of it until school was a complete disaster so we got testing to prove them wrong and it was a bit eye opener even for us.  In retrospect there were a lot of indicators but really it didn't matter until we entered the school system. 
DD5, I still don't know.  She is completely different than DS so I guess we'll have to wait and see.  We'll likely get her tested eventually but are hopefully holding off for a bit.