Ore grades will never degrade below 4ppm uranium and 12ppm thorium.  That's the concentration in granite, and there's plenty of that.  A 1GW fast breeder[1] will need about 3 tonnes of fuel per year, which at the above concentration amounts to about 500 tonnes of granite per day.  Let's say 1000, allowing for inefficient extraction and some losses.

A coal plant of the same size requires 10 times that amount of coal, leaving a hole in the ground 10 times as large.  This environmental impact is not considered larger than that of radwaste, so the same should go for mining rocks.

"He who refuses to do arithmetic is doomed to talk nonsense." -- John McCarthy

[1] Yes, a breeder is needed.  An LMFBR would work, a molten fluoride or molten chloride reactor might, too.  It is completely clear that nobody would burn such low grade ore in a light water reactor.