Dd does this and I did, too. We agree on some but not all. I figured it might come from posters in our classrooms or favorite books etc that imprinted a color on a number or letter, or common examples of them (T is green, perhaps because they always show turtles in our abc books... And it starts the word three, which is also green!). But maybe it is some synesthesia.