I have been (and to some extend still are) where you are. My DSs dad and I divorced right before he turned one. He does go there regularly but I don't think that his dad sees what I does. It's funny because sometimes even a year after he's doing something (reading or addition for example) his dad will call to excitedly tell me and I have to point out that he's been doing that thing forever.
Until recently he didn't want to hear that DS was gifted and he didn't want me testing him. He didn't want him in gifted classes because they would make him weird and even recently told me that I don't need to push for him to move up in reading because reading isn't really that important in life.
It's frustrating when it seems like you're fighting a battle against the schools and then there's another person that you're having to battle too..and it's the person that should want as much for your child as you do.
I don't want to post too much about my situation on a public board

but it's been tough at times to say the least.
I decided at some point to do what I think is best. I'm the one that's most involved in school and that does homework with him and sees what he can do. I am the one that calls the school and the one that the school calls when there are issues. Since I decided to do that I still keep him up to date and he's been much easier to deal with. He's even started to agree with me on some things

Good luck and do what you know is best for your son. There will always be people in your life and his that think you're doing things wrong (whether it be about gifted issues or sports or discipline, etc) but you have to trust yourself to make the right decisions for your son.