I'm not sure of the cause of your difficulties with algebra specifically but the problem is not yours alone. There is a lot of debate and research into struggles with math and algebra in particular. Moving from concrete representations of numbers to abstraction involved in algebra is frequently mentioned but that would appear to be a symptom and not a cause. Why some people struggle with abstraction is a bigger question.

http://hepg.org/hel-home/issues/28_3/helarticle/the-algebra-problem_533

http://www.nctm.org/news/content.aspx?id=12332

http://usablealgebra.landmark.edu/instructor-training/math-deficits-why-do-students-struggle/