Seems odd. So if you gave him a sheet of math facts like 8+6, 7-4 etc (without a timer but telling him to work quickly), would he be able to answer each question within a couple seconds and stay focused on it?

If he's not fidgeting with stuff on his desk but actually playing with it, I think it makes sense to remove as much as possible from his work space.