My DD made friends with an older bully that lived across the street when she was 3. There weren't really any other options for her if she wanted to play with a peer. It wasn't as bad as you describe... the other kid could be engaging and fun, but she just always had to have her own way, was very controlling, etc. I suppose it has something to do with gender, because bullying in boys and girls manifests differently.
DD would come to us with tears in her eyes and tell us what her friend had done lately, and sometimes we'd advise her to not play with her friend the next time she called, and explain it was because of the given behavior. Then her friend would call, DD's resolve would evaporate, and they'd be playing that afternoon, as DW and I shook our heads. Some of the problems involved the other child never sharing toys at her house, always seeking DD's favorite toys at our house, never inviting DD over to join when other playmates had come over (including ones DD was already friendly with), etc.
In a way, it was a good thing, because it gave DD exposure to bad behavior, and gave her a start on how to deal with difficult people. But she no longer has any play dates with the child in question, because while my daughter was learning to stand up for herself, the bully's mom decided to call my wife and try to bully her. She failed, and now this woman is stuck at home with a bored child far more often than she'd previously enjoyed.
I'd say don't assume it's not bothering your son, because he might just be bottling it all up. My DD did at first... until she snapped and tried to shove her friend off the top of the slide (only 2 feet high, but still). She slugged her once, too. This is not a violent child, either... but she could only take so much.
