And this is the central idiocy of Smith-ite economics.

Firstly, this is a purely ad hoc example. It's not based on a general sampling of different economies. This is hand-waving and supposition, not evidence-based economics.

Secondly, what sets worker wages is business practice. In the real world this means the extent to which shareholders and investors are willing to, or sometimes forced to, share financial profits with the people who do the manual and/or intellectual labour that create those profits. That includes elements of regulation, supply and demand, and other factors. But GDP seems like a relatively small influence on that - as you can in individual companies which are booming but still pay their workers the minimum they can get away with.

Unless I'm missing some direct evidence for the divine intervention of the Invisible Hand, that's the only factor that I can see that sets worker wages.

GDP is irrelevant. Even in a steady state economy, as long as there's profit and profit distribution, investors will be compensated and workers won't starve.

So why growth? Here's a different questions:

Is it possible for an economy to be sustainable and also innovative with zero GDP growth?

Heretically, I suspect it is. It's never been done, but I don't see why it couldn't be possible. The key difference would be in the finer points of the trading relationships, and in what is being traded. If it's intellectual or artistic property, education, shared experience, or other items with little or no ecological overhead, then it should be very possible.