He was having a hard, confrontational time in school before you pulled him out, wasn't he? I just wonder whether something a bit on the routine side might be better for him for a bit. Would he get on better with "sheet of sums" style maths daily, with LoF left as an optional supplement? You're in it for the long game; you obviously need to avoid giving him only things he can already do, but it would be OK if progress was slow this year and the challenges he got were all of his choosing. Working on hard problems in maths is super-important, but it's only going to happen if and when he's willing. I think, given that he's doing recreational maths and hasn't been out of a bad school situation that long, I'd be going for peaceful for now, with a calendar note to reconsider in a year's time.
I would expect that the memory issues were most likely due to the anger issue. Does he forget things he chose to do?
