Same here, Kim, though we don't have the 3 families so close. That's lovely!
But the neighbors who aren't homeschooling seem quite accepting of what we're doing. My kids have *far* busier social schedules than I had at their age, and most of their buddies are public school kids, so they don't seem to be pariahs in our neighborhood, even among people who aren't homeschooling.
We also see lots of crossover between secular homeschoolers and religious homeschoolers, and thus have lots more opportunities. It seems like religious homeschoolers in our area are pretty accepting of inclusivity because of all the benefits that come from having greater numbers.