I agree that you've received a lot of good advice on this thread!
I think that our society has developed an idea being a perfect parent involves always being with your child and loving every minute of it. Not only do I disagree with this idea, I also believe that because nobody is perfect, trying to force yourself into a perfection mold will only result in disappointment.
I think that a lot of women (and some men) feel the same way that you do, and I expect that most or all feel like they're letting their families down because they're experiencing the exact same feelings that you described in your OP. If they hide these feelings, a myth gets perpetuated.