If you approach homeschooling as a choice you are making to serve his needs, and you do NOT let him always have his way, he won't feel that he can do what he wants. I am more strict with my son than any teacher would be, but because he does work that is at the correct level for him, he is much happier and more cooperative than he was.
It's all about how you approach the subject with him, I think. I use the words "better fit" or "good fit" a lot to explain our school choices. I work very hard NOT to blame the school or the teacher, nor to blame him. We're just trying to find a good fit for him.
The fact is, your son knows that the current situation is not working better than anyone. I daresay that's much of his acting out. Responding in a way that could help solve the problem is not "giving in" on your part. So don't approach it as if it were and you'll be fine.
Rules are key! Homeschooling is still school! It should be more fun than what he's going through, certainly, but it should be fun because the work is more challenging.
I think of it this way: a kid who is hungry acts up because his needs aren't being met. We don't keep food from him because "that might teach him that he's the boss"--we feed him because that's what he needs! A kid who is hungry to learn is exactly the same thing.
Meet his needs. That's not giving in!
