The swap rate is hedgeable, not neutral. Well, it is "risk-neutral". And the assumption is that the market will evolve towards the forward rates, which is not observed and would lead to absurdly high interest rates if it did happen.

En un viejo país ineficiente, algo así como España entre dos guerras civiles, poseer una casa y poca hacienda y memoria ninguna. -- Gil de Biedma