Basically you're looking to have the child working in the "zone of proximal development." This is the difference between what a child can do with help and what he can do without help.
With DD6 I know she's in the ZPD when she can do most of the problems on her own, misses a few and then the light bulb goes on when I explain it to her. If she misses a few just because of careless mistakes and no lightbulbs come on, this is a sign it might be too easy. If she misses a bunch and the lightbulbs aren't coming on when I explain, it's too hard.
A major factor in this is DD6's attitude and I've discussed with her what an important role attitude plays. When she shuts down, the ZPD drops to a much lower level. With a bad attitude, she can't even solve problems with my assistance that she can do independently when her attitude is good. I tell her the most important thing is her attitude and give lots of positive feedback when she keeps a good attitude in the face of challenging work.
I couldn't find any articles to support my personal experience but I hope this helps.
