I wonder about this:
I hope that I've found enough of a middle ground to only comfort him when he's past the point of being manipulative.
could be backfiring? If he doesn't have the skills to calm himself down when he's upset, could it be that the realisation that you're not going to help until or unless he's extremely upset is itself part of what's getting him upset, in that moment? I should say that I'm coming from DS-now-7 having been a really rather reasonable toddler, with only a fairly brief period of getting very upset over incomprehensibly tiny things, so take it fwiw, but: it seems to me that it's not only possible but wise to separate out whether you allow yourself to be forced into changing your mind by a child being upset (I agree that one shouldn't, unless one has actually made a mistake in which case it's good to admit it) and whether you are sympathetic to the child's upset. I think you can be sympathetic to the upsetness without any suggestion of giving in over the decision, and that there's no need to wait until the upsetness is clearly out of control. It's only manipulation if you're manipulated by it :-)
I think "We can't make a smoothie yet, but you can help me clean the broccoli and put it in the steamer." is great and did a lot of that kind of thing. Another kind of thing that worked well for us was more like "Tomorrow we'll have a smoothie, but today we haven't got [time before the next meal/the ingredients/whatever the reason is]. What do you think the best smoothie in the ENTIRE UNIVERSE would have in it?"
