In the case of my DD, who hasn't been challenged much in life, you can tell she's working hard when she is emotionally engaged and excited, perhaps even sometimes on the verge of tears, but also persisting with the task and ultimately very happy and satisifed upon completion....
The sight of her actually working is a beautiful one for me.
I have seen DD have to work hard a couple of times recently. It was awesome.
1) AOPS. Some of those questions are really tricky. She might go up a blind alley, and have to start again. I could see her getting a little panic-y every once in a while but then she would renew her focus and start again. I would say this was working "on the edge of her competency"
2) For the first time this year, she's had some abstract concepts in English (for example) that she's had to really mull over to completely understand. It's been good to watch her stretch.