I think (based on having one, having been one, and a bit of reading) that mathy kids need two kinds of stretching:
1) a steady flow of new concepts to grapple with, AND
2) plenty of work solving novel problems which can be solved using only concepts they already know.
In the end, (2) is probably more important (in terms of developing real mathematical talent), but it's often the hardest for very young children to get to grips with. (1) is what you get out of subject acceleration, the kind where your DC just gets put with older children and taught alongside them.
It sounds as though your DS is currently only getting (2) and not really getting any support with it, and it's not surprising he's finding that hard. Looking at my DS and then generalizing wildly :-) I think it's important to have both together for young children because (1) actually contributes to (2). The process of understanding a new concept is itself a kind of puzzle, in that they have to deal with looking at some maths and not understanding it immediately. It's a rather tame kind of puzzle as there will be a path laid out for them into the new concept, but this can actually be helpful if novelty itself seems a bit scary.
My DS6 is in what is probably the ideal situation for him: he's in a small class with a very willing and capable teacher, and basically all his maths is individualised for him. He does a wide range of different things in class, including some problem solving (nothing very way out, I think, mostly word problems) and some new material; at home he does Descartes' Cove, Martin Gardner puzzles and anything else we think of, and we keep in touch with his teacher about what he's doing. The main advantage this has over acceleration is that he's still going at his own pace, which is much faster than standard, whereas if he had been simply put with a higher year I'd have been concerned that he'd have been bored and unchallenged again almost immediately. The main downside is that he's not getting to do maths with other children at his own level. I am pleased to see that he still has a pretty good level of perseverance (for a 6yo!) when faced with things he can't see how to do straight away.
I think in summary, no, I definitely don't think you should forget about acceleration (1) because of the difficulty your DS has with (2). Acceleration may well be a step towards dealing with that difficulty. I do think that, if and when you get your acceleration, you shouldn't assume that that will be all that's needed: if your DS is to realise his mathematical talent, he will also need support and encouragement with (2).