The SP indeed became a larger party only after the referendum. But its base had already expanded at that time.

To cherry pick on my side a bit, here's a short quote from this evaluation (pdf) of the Dutch referendum:

Looking at results per municipality, it shows that a majority voted no in 9 out of 10 municipalities. The municipalities which voted yes in majority are the richest municipalities of the country: Rozendaal (Gld.), Laren, Bloemendaal, Heemstede, Wassenaar, and the rich communities in Eindhoven. Municipalities which had a high percentage of no-voters were the fiercely Protestant localities, the leftwing Socialist localities and those that also voted in high numbers for Pim Fortuyn (`protest localities').