Not to mention Latino expansion from the sixteenth century onwards. The Brits actually were the first major imperialist power to abandon slavery. The growth of the European (ex) colonies was primarily driven by poor European settlers be that in English, Spanish, or Porgtugese speaking areas. It turned out that for modern capitalism, the legal and economic strictures of a market economy are far more efficient at generating wealth for the elites than chattel slavery. Those areas that had been particularly dependent on the early modern system of slave labour tended to fall behind (US South, Caribbean, northern Brazil). The same was largely true in Europe as well with a pretty good correlation between the persistence of serfdom and relative poverty and economic underdevelopment.