I think this problem is due as much to the way that physics is taught than an inherent super-high level of difficulty. I struggled through first-year physics and really understood very little of it, in spite of doing okay-ish on exams.
That's how I remember intro college physics. There were 3 tracks, depending on how much calculus background you had, and I was in the middle one, meant for science majors who weren't physics majors. A bunch of us in Mandarin classes were also taking Physics, and we'd all meet at the science tutoring sessions 3 times a week, and get each other as far as any of us could get on the problem sets, then ask the science tutor to explain when we got stuck. In a typical session, we might be able to solve 1 problem between ourselves, out of 10 assigned. (We went to discussion sessions, too, where a grad student attempted to teach us the stuff the professor had been unable to. Unfortunately, the discussion session topics were a level easier than the lecture, and the problem sets were a level higher than the lecture.)
All of us were passing with Bs at semester break, and none of us enrolled for the spring.