I really don't like the idea of consciously making myself sound dumber or less confident or less straightforward about what I'm saying.
Neither do I. I think if you adjust your approach a little bit, you would actually sound more confident, more straightforward, and maybe even a little bit smarter. But you could very well interact with others in a different way from what you've presented here, and almost certainly do speak differently, so it may not be that important. It seems unimportant to dissect the sources of my opinion; I don't mean to be insulting, and you seem fine with the way you present yourself.