Also, assuming that they will know what education the kids have received this year is false since any parent who realizes their kid needs something additional will supplement outside of school, where it will be entirely uncontrolled and unaccounted for in the study.
Welcome to the world of edumacation studies. You do not want to know how many of these studies fail to include appropriate control groups and how many fail to ensure that the methods they're testing are reproducible and valid.
In this situation, I'd ask who the control group is and how they've ensured that each control group gets the same instruction. If they're just using "students who weren't taught by our methods," they'll end up comparing a zillion different teaching methods with their own. Sounds like very noisy "control" data to me.
How are they controlling for academic activities outside school or lack thereof?
Importantly, how will they measure outcomes? Are all the kids going to take the same tests (eg every participating 3rd grader around the country will take one math test, including control groups)?
If not, how will they compare outcomes? They can't use standardized tests written by states because each state has a different test. They can't compare this year's kids with last year's kids because the kids are different (and presumably the test, too). Even if they claim that the high numbers of students normalize the data, many of the teachers will have changed and maybe textbooks and homework too. How will they know they're comparing apples with apples?
I have no idea how any of these parameters have been set up in this project; I'm only throwing out questions to help you judge the validity of the study.
FWIW, I looked at some guidelines written for projects funded by the NSF, and their educational studies
seem to be exempt from institutional review and consent. Don't know how this applies here, as the project isn't NSF-funded
My cynical side is thinking, "Lack of institutional review is a factor in the low quality of education studies."
Val
