My husband is a mathematician. I asked him this question and he also recommends starting with Wolfram Alpha because it's "free, online, has support for arbitrarily large numbers and is good for symbolic math (Matlab and R are not)." The only other alternative he can think of is sympy, which can be found at www.sympy.org