I have spent many years surrounded by math. I think some brains are better built for it than others. It has always come natural to me. Even at the highest levels. It's harder for me to understand people than to understand math. Numbers are not emotional and are always the same. People now, that's a different story.
We depend on our dominant brain half as far as I am aware. Or we simply need to become better at thinking abstract. Goes for human understanding too. They are not that far away from each other. ;-)