they still ended up essentially trying to force non-Slovenians out
Well, some non-Slovenians. Not that plain racism would be better (it's worse), but the situation of ethnic minorities other than Roma and Serb is much better than elsewhere. Say, language use in official situations, inluding village name tables in different languages has been an issue all across the region, even in Austria, but not in Slovenia.
On the referendum: I did remember the issue vaguely, but I was a bit incredulous and had to check. I find (1) the 3 April 2004 referendum was non-binding, (2) participation was a mere 31%, (3) that after opponents called for a boycott. (Still, sadly I suspect they could well have had a - narrow - majority had there been no boycott, I found reports of late 2003 - early 2004 polls on the issue.) *Lunatic*, n. One whose delusions are out of fashion.