I'm planning to have a long conversation with my parents who are coming to visit in a few weeks about this topic, as I figured out I need to sort out some of my own "gifted" baggage before subjecting my little ones to anything.
In kindergarten, I remember being able to read and baffled about why the other kids could not. I read anything I could get my hands on and raced ahead. I went to a magnet school with a mixed group of students (i.e. poor neighborhood kids and bussed in non-neighborhood kids). My school's windows frequently sported bullet holes but I never once felt unsafe during school hours on school grounds. I wouldn't have walked anywhere else...
I do know that starting around 3rd grade I got to leave my classroom a few times a week, to go to the gifted program. I enjoyed this but can't recall anything specific we did. I also got to see the ??school psychologist? where I did things like play puzzle games and look at ink blots. My parents were concerned about me socially. I was pretty introverted and preferred books to people. Despite this I usually had good friends to play with in elementary. Outside of the pull out times, it was boring. At a certain point there wasn't much in the library I hadn't read. I remember during these years my parents got me an encyclopedia and I often would sneak a volume into bed at night and read from it.
Middle school I had a very tough time socially, got teased etc. I had some academic challenges too perhaps tied to the social changes (in the advanced reading group I quit doing my work for a while, which spiraled into a bad grade, trouble with my parents etc). What I remember I loved about middle school was the 6 person advanced humanities group. We studied Latin, Shakespeare, etc. So it wasn't ALL horrible, but I struggled. I know my parents were worried I would not get into the HA/GT HS where most but not all students skipped a grade upon entry. I did get in, but only after not skipping. Thankfully my parents persisted with me and/or the HS as I got in eventually.
I loved the HS. It was a very unique environment (public, by the way). I did not have to worry about being weird as there were a lot of kids far stranger than I was. Everyone was expected to go to college, and a decent one. Work was challenging but for the most part very achievable. I did have to learn study skills there. I had a few minor academic bumps but ended up shocking the heck out of everyone and even myself senior year as I got some prestigious awards that year - until then I had been seen as a middle of the smart/gifted pack kid. I also took my first college class while in HS. After graduating, I spent a year in France before going to university on delayed entry (already admitted so no academic benefit whatsoever). I wanted to do the year abroad but not miss a year of school, and it helped my parents financially a bit too.
College seemed more a continuation of HS academically versus a new level. My university had a lot of kids that went to special public HSs, private or prep schools, like I had. Some were brighter than others, but we all were within a few stripes of each other so it was a great environment. I loved it, graduated w/Honors. Many classmates went on to grad/med/law school right away. I did not.
Now many years later, I'm finding my way to career #2 and presently in grad school. I guess it was inevitable.

But there are things I want to understand, so if at all possible I can head off any challenges with my own kids who are very young still. Like some others have mentioned I guess I shouldn't be surprised to be wondering if they are gifted but by the same token, it puts a whole new spin on what I think they may experience in school.