There is slowness from lower priority related stops, but the latter are not due to the number of passenger trains, but the speed differential -- and that more vs. expresses than stopping trains. In that sense, replacing expresses with high-speed trains on dedicated lines would indeed be the way to go. I note though that due to noise issues, currently ever more countries return to the (IMO bad) idea to put freight trains on the high-speed lines (because those are avoiding lines), and hope that advanced signalling will improve capacity.

Meanwhile, European railfreight is also slow due to a low top speed of trains, inefficient switching when trains are re-arranged, and (considering the distances at which rail is most competitive) above all borders (which are often technological borders). There are improvements in each field, though.

_{*Lunatic*, n.
One whose delusions are out of fashion.}