I believe if you go back to how they met, the Man With the Yellow Hat went on a safari. He and George 'met'. George was very curious and snuck into the Man's luggage in his typical oops kind of way. The Man found him in his luggage when he got home. I believe the Man tried to return him, but George kept hanging around. That was when the Man adopted George.
Scary that I know this.
That is the updated, kid-friendly version shown in the movie, etc. The original story reads like this:
The hat covered George's head. He couldn't see. The man picked him up quickly and popped him into a bag. George was caught.
(pictures of a scared George trapped in a sack.)
The man with the big yellow hat put George in a little boat, and a sailor rowed them both across the water to a big ship. George was sad, but he was still a little curious.
Umm, how awful is that to read to young children?!
Given the circumstances of his abduction, Aquinas, I do not think that his silence conveys consent. And the man with the yellow hat is clearly dangerous!

That said, I actually really like the PBS George cartoon!