The issue is that being driving wheels, where there has to be a direct mechanical linkage between the wheel and the driving force (cylinder assembly), there is a considerable unsprung weight impacting on the track. The "hammer blow" resulting from the combination of the weight and the vertical force from the cylinders can exceed  20 tons per wheel. It is that which wears the track.

I don't dispute that it's a problem tho'.

The issue of the discomfort of working on steam engines is largely moot. Much of that was from a victorian attitude towards worker comfort and safety that was hardly restricted to railways. And anyway, even nowadays you'll find many drivers (mostly now retired) who worked on both steam and other traction who would gladly endure the discomforts of steam for the professional pride of working such beasts.

keep to the Fen Causeway

Oh, no problem there, there were special solutions :-) There were of course the steam-electrics -- and the steam-motor locomotives. For example the German 19 1001, built during WWII,  ended up in the USA where it was scrapped:

_{*Lunatic*, n.
One whose delusions are out of fashion.}