Just from the numbers, I would guess at two possibilities, oppression of Serbs in Prizren or better opportunities for Serbs (but not Albanians) elsewhere.

This being the 19th century, opportunities could be migration to America (a very large proportion of the Swedish population at that time emigrated and generally that happened cluster by cluster). Or it could have been possibilities of employment in new industrial sectors in other regions or countries.

(Btw, if you use Firefox, I really recomend TribExt, it makes it really simple to cut and paste while keeping formats and links.)

Sweden's finest (and perhaps only) collaborative, leftist e-newspaper Synapze.se