Supply and demand are in balance anyway, with the total price (including non-monetary) including the inconvenience, additional fuel consumption, etc., caused by the congestion. I think that the argument is better staged in terms of differing negative externalities of driving at different times, noting that differential monetary pricing is a non-destructive transfer of tokens, while congestion destroys actual fuel and slices of human lifetime.

It is often argued that X should have a lower price because the poor will be more affected, but this applies with similar force to all non-luxury goods X. In all instances, to act on this would distort prices and incentives. This suggests that it is far better to address inequalities more directly, on the income side.

Words and ideas I offer here may be used freely and without attribution.