Market forces say that when demand is growing, prices will go up, which will encourage new supply to be provided, and some demand to be discouraged. But oil is a very atypical market right now

Take the wikipedia article on Supply and Demand, which sports the "representative" diagram to the right. It does include "diminishing returns" in that the price increases faster than linearly with the supply, but the diagram contains no hint that supply might be effectively bounded regardless of the price. If your mental picture of the price elasticity of supply doesn't have a limit to the supply volume, you won't be able to understand what's happening.

In one of my earliest diaries I put together the following chart (the axes are switched with respect to the WP one, unfortunately)

That supply curve displays both economies of scale and diminishing returns as well as a maximum possible rate of supply (it's a logistic curve). With a curve like that you can see that if demand gets too high, then supply becomes nearly independent of price.

Now, that may spurn new investment in production capacity but investment doesn't instantaneaously result in new production. That may be one of the reasons why the 1970's oil crunch was followed by a supply glut and a collapse of prices in the 1980's: it took about 10 years for both demand to adjust to high prices and for supply to increase.

When the capital development of a country becomes a by-product of the activities of a casino, the job is likely to be ill-done. — John M. Keynes