ultramarina, this is undoubtedly a really complex situation with a whole lotta stuff going on. What really jumps out at me, though, is:

(a) she says she's not good enough for the highest math track

vs.

(b) she was in the top 5 MathCounts scores.

Whatever is going on, it feels like inability to handle the class is more excuse than issue. Random possibilities might include...lack of confidence. perfectionism. underlying hidden learning issue. social issues and self-image, in general and/ or around the kinds of kids in higher level math. It's hard to tell if this is normal teen angst, or at least typical gifted issues, or something a little more hidden. But from what you'e said, she likes thinking about math. She's very good at it. And she's avoiding it. What is she actually avoiding?