That's how we started out with our first. No TV until he was three. Then we let him watch certain things, mostly educational, and that's pretty much all he'd ever want to watch anyway. Then we started using it as a way for him to relax because he gets kind of intense, but when our second was born it was difficult to keep her separate. So now, they both pick one show after school and that's their TV time for the day. On weekends they might watch a bit more.
Movies are impossible because they won't sit for that long, which is probably a good thing.

I'd like to cut back on TV, so I think I'll take advantage of the busy start to the school year and just leave it off during the week. Like you, we go to a museum almost every weekend, so a little TV mixed in with that makes for some needed rest time for the parents.