Welcome!
The most important two pieces of context for your early testing are your report that you were not fully testable (apparently for emotional/behavioral reasons) at the time, and that you were young. Both of these (but especially the first) suggest that not too much weight should be put on your lower scores on the WISC-IV (principally in the PRI). It is possible that you have relative weaknesses in visual spatial domains (as throwing the blocks may well have been a chicken-and-egg situation, where the behavior was an outgrowth of the difficulty of the task, rather than the task having been spoiled purely by the behavior), but that does not preclude mathematical strengths (the best predictors of math ability were either not administered or didn't yet exist on the WISC-IV).
FWIW, it is possible that your true VSI/PRI is closer to the 120s than the 100s, but also possibly not. Depending on what you were given for the later nonverbal test, I suspect that instrument was more parallel to Matrix Reasoning than to BD, and likely was a measure of fluid reasoning rather than visual spatial thinking. Of course, FR is even better correlated with mathematics reasoning than VS/PR is. Some areas of physics (e.g., not classical) are likely less dependent on VS than they are on FR.
But generally speaking, I find that present performance is a better predictor of future performance than past performance is. Especially where the data that causes you to doubt your potential in a field of interest to you is already questionable and inconsistent, I would focus more on present measures that demonstrate you are capable of physics achievement, as predictors of future physics achievement.
IOW, I see no reason for you to give up on your dreams, just because a test given seven or eight years ago, when you were in a challenging place in your life, doesn't match typical patterns.
Also, goals can grow and change, as you experience and learn more about various fields of study. Pursuing an academic physics career, as appealing as it currently appears, may lead you into areas that you find even more fascinating and rewarding.
