Hi Jenafur,
Yes, it is exciting and scary when you see your toddler doing things that most kids do at 4 or 5.
In the early stages, it is not too hard especially with all the educational preschool games and toys available. Once he masters something then simply introduce him to the next level of material. For example, once he masters recognizing capital letters, then introduce him to lower case letters, then letter sounds. Similarly with numbers, once he masters counting, then help him to recognize the numbers, and eventually introduce simple addition. I was fortunate to have a good friend that was a kindergarten teacher to help me when my daughter was a toddler. I got a little stuck between teaching the letter sounds and teaching to read. There are some auditory things like hearing beginning and ending sounds and rhyming that fill in that gap.
Introducing your child to material is not pushing, especially when your child has an interest in learning. You will run into people that say you are pushing your child because he demonstrates a high level of knowledge. I frequently had people ask me if I worked with my daughter a lot. My answer was always no. I just introduced the material. Actually, I often felt guilty that I did not work with her more.
One thing my daughter really loved to do as a toddler was check out children's encyclopedia books about animals from the library. She would memorize the names of the animals in the books, she especially loved sea creatures. It was always fun to see people's faces when she would name creatures like nautilus and gulper eel at two years old. Find something that your son is interested in like animals, dinosaurs, insects, or cars. Don't be afraid to get higher level books if he shows an interest. If he is not physically ready to handle the books by himself (still tears pages, puts them in his mouth, etc) you can look at them with him.
Also, I recommend that you look at Hoagies Gifted Education Page.
http://www.hoagiesgifted.org/ Enjoy the adventure.
Summer