I did/do this and so do my kids.
Once my older son said to me, "I know why I can't remember the difference between Saturday and Sunday. They're both red!"
The older one seems to have it more strongly than the younger one. The older one has dyslexia, and I've heard that it's more prevalent in dyslexics. As you can see from the above example, it's not just with numbers, but also with letters, days of the week, months, years, decades, even words. None of us actually sees colors, it's just a really strong association.
