I did this at that age, and I assumed everyone did this associated the same colors with numbers that I did. I was surprised to learn that others not only didn't have the same colors, but didn't do this at all, when they'd say things like "What do you mean 3 is orange. What are you talking about. Numbers aren't colors." I still remember my number-color correspondence, but I stopped associating them at about age 9 or 10 when I realized others didn't see it that way, but also realized that the correspondence didn't have significance mathematically.

I don't think it's truly a case of Synesthesia, maybe a very mild form.