I think it's extremely helpful to get to the point where you don't need to count on fingers or struggle to recall answers. But depending on if there is a learning challenge it might not be possible. I don't think I'd give up at this point, just find ways to work on it without emphasizing speed or putting on pressure. Like have him work on math fact problems for just a couple min. per day (not timed), then move onto more "fun" or interesting math.