Adam smith was denouncing the fact that this asymmetry was taking for granted, and that it was bad for the common good. At least that's my reading of him.

Whenever the legislature attempts to regulate the differences between masters and their workmen, its counsellors are always the masters. When the regulation, therefore, is in favour of the workmen, it is always just and equitable; but it is sometimes otherwise when in favour of the masters.

Hiw view of "the combinations of the masters" was rather hostile:

People of the same trade seldom meet together, even for merriment and diversion, but the conversation ends in a conspiracy against the public, or in some contrivance to raise prices.

Smith does read as strongly pro-labour throughout, although his view of the "corporations" (guilds) would have made him frown on large, powerful unions.

The pretence that corporations are necessary for the better government of the trade, is without any foundation. The real and effectual discipline which is exercised over a workman, is not that of his corporation, but that of his customers. It is the fear of losing their employment which restrains his frauds and corrects his negligence. An exclusive corporation necessarily weakens the force of this discipline. A particular set of workmen must then be employed, let them behave well or ill. It is upon this account that, in many large incorporated towns, no tolerable workmen are to be found, even in some of the most necessary trades. If you would have your work tolerably executed, it must be done in the suburbs, where the workmen, having no exclusive privilege, have nothing but their character to depend upon, and you must then smuggle it into the town as well as you can.

The Wealth of Nations makes for some jaw-dropping reading (just like Macchiavelli's The Prince, by the way) because of how strongly and frequently it is misrepresented in the public discourse. For instace:

Such regulations may, no doubt, be considered as in some respect a violation of natural liberty. But those exertions of the natural liberty of a few individuals, which might endanger the security of the whole society, are, and ought to be, restrained by the laws of all governments; of the most free, as well as or the most despotical.

There you have a clear endorsement of government regulation of powerful business interests. (And I am not taking this out of context: this is smack in the middle of his discussion of the banking system — Smith would have liked the Tobin tax)

A society committed to the notion that government is always bad will have bad government. And it doesn't have to be that way. — Paul Krugman