This is a topic near and dear to my hear, since my kids started Suzuki at 4 (daughter, violin) and 5 (son, piano). There are plenty of kids that start violin at our music school at age 3 (even some a little younger, piano tends to be a little later). The difference between starting at 3 and 4 is the willingness of the parent to practice with a very little one. We were doing kindermusik style classes at that age. My daughter at 5 is the only one in her group lesson that started so "late". So she's at the exact same place the other kids are (or even beyond in some cases). You just move slower through the initial phase the younger you start, which is painful at any age! It can take months and months and even over a year to be able to play a whole song on the violin. A lot of that is dependent on small motor skills, willingness of the child, and the enthusiasm and commitment to daily "practice". Much of the first year is just building up a practice habit. Sometimes those practices are 5 minutes and involve holding the violin and playing a few games and being done. It's not for every family for sure. But it has been wonderful for both my kids (and me - I took Suzuki violin growing up too). They're able to crank along at their own pace now that they're well established. It has taught them a lot about working at something over time and working with a mentor. Lessons that go well beyond music. But kids can learn these types of lessons in a lot of ways.
As far as where to go, we love our large well established Suzuki program with many other kids and group lessons, but we know families having an equally good experience in a small private studio. I would observe teachers and group lessons. It's great if you can get a good teacher fit initially (my kids are on their 2nd and 3rd teachers for various reasons!). A lot of it is about someone who "gets" your child. When people ask me about Suzuki I tell them it is a HUGE parent commitment. There's nothing magical about how my kids play. They practice 6 days a week and I'm usually helping them. They've taken off faster than others, but it's still hard work for me and for them. My 5 year old can usually be done in 15-20 minutes. I would expect that is the kind of practice you can expect until you reach late book 1. My 9 year old piano guy never practices less than 30 minutes and it can reach up to an hour when we are preparing for something (averaging 45).
I wouldn't worry at all what other people think! If you go to an established Suzuki program or teacher young kids are the norm. And so many kids in our program are obviously GT++. It's more I think a parent decision about the level of commitment you have to music lessons.