It's a pretty weak argument to claim that people who refuse to get vaccinated should not be restricted against social activities. That argument suggests, for example, that people who refuse to obtain a civil engineering license should not be restricted against designing bridges.