I don't have a problem with the AP tests counting for college credit. My issue is with the idea that a student taking AP courses is unequivocally demonstrating an ability to do college work.
There is a huge difference between how AP (and IB) classes are taught and how real college classes are taught, and not just the pacing. AP/IB classes work like regular high school classes in that they have lots of assignments (and therefore lots of stuff to grade). Generally AP/IB teachers are scaffolding instruction more and giving students more input on studying the material (hence all those grades). In a college class, frequently the only grades are the midterm and the final. There are no assignments that are intended to help the students study the material (such as "read the section, answer the questions at the end, and hand them in"). The students are expected to do whatever it takes to learn the material.
I agree that high school has its own set of challenges with juggling 6-7 courses and that having 3-4 of those be AP courses can be very difficult, but it is still not the same thing as dealing with an actual college course.
