Unless students were randomised to attend ALEKS or MOEMS sessions - and the results analysed on the basis of where students were randomised to, regardless of whether they actually attended - this comparison has no value. Were they?
( My intuition is that you're basically right, *but* that there will be significant numbers of students, including some with huge potential, not reached by any optional competition programme. My first step would be to make actually competing in the competitions optional, and advertise that it's perfectly fine to come to sessions to work on problems and never actually compete.)
You're right that the self-selective nature of the groups is problematic. However, since she could find no evidence of impact on students' outcomes from ALEKS, she couldn't justify writing another grant proposal to continue the program.
The students are welcome to attend sessions without competing. Most choose to compete. A few need encouragement to try at least one of the five contests. Of those, most of them continue to take the contests, but a few don't. They simply attend the practice sessions and skip the session on days we have a contest.
One thing I thought of when I asked the question a few posts up was, it may sometimes happen that someone that didn't think of themselves as super strong in math, may do really well in a competition, realize they've underestimated themselves, and improve a lot. (I.e. the competition could "discover" unrecognized/undeveloped talent, which is subsequently developed.)
Some students are surprised by how well they do, and the contests do build their confidence. Many math competitions are touted as being good for discovering math talent. However, the opposite can also be the case, especially with gifted and perfectionist students. We are careful to explain the these are contests and they are different from tests. We tell them that whereas the tests they take in class cover material they already know, math contests are meant to challenge even the brightest and strongest math students. Students who are used to routinely scoring 100% on tests can be discouraged when faced with problems they can't solve, so we try to prepare them for the challenge and encourage them to keep trying if they are less successful than they expected at first. We also allow students who do really well to compete up when the rules allow, so our strongest students are always challenged.
