I can ABSOLUTELY understand the approach of not making this an issue right now and taking the stress level down - this MAY be the right approach for your DS.

DD started complaining about math in 2nd grade and it turns out it was because she was not quick at her math facts. We solved the problem by having DD practice (she does NOT suffer from anxiety). She has a weaker WM than reasoning-ability, so this actually took some work on her part - but it was great for her. Learning her math facts quickly/automatically actually GAVE her math confidence. This might not work in your situation, but DD has gone from "I hate math" to being one of the very top math students in her grade (and possibly very high a grade or two up). I hope that you find a good solution for your DS.