Weren't all slave owners racists? How could you own people and treat them like property otherwise?

I don't think they saw themselves as "racist". The term is modern and skin colour was not so important at the time of enslavement. It became so only later as attitudes to the slaves lingered after abolition of slavery. The race was an identity useful to those callous enough to be determined to keep those attitudes alive.  

Rather, the slave-owners regarded the people they enslaved as backward in culture, religion and society, lacking NE European standards of education, so fit only for manual labour. Also the slave trade relied very heavily on African and Arabic help.  

Yes it was a nasty, cruel attitude born of a mixture of ignorance and feelings of cultural rather than biological superiority; and the campaigns against slavery fought the cruelty, not the racial aspects, seeing the slaves as people beyond their skin colour.

I think you are right, Rosie, unfortunately. Human nature won't change, and feelings of superiority and need to find scapegoats won't go away.