A better answer is volume 1 of Feller's "Introduction to Probability Theory and its Applications", which uses entirely elementary techniques.

Epistemologically, all statistics is finite, and infinites and continuity only appear in the limit (via a process of closure useful because completeness simplifies many proofs, but not essential as sophisticated but straighforward proofs by closure can be turned into involved proofs using elementary techniques). Also, stochastic processes are equivalent iff all their finite-dimensional distributions are equivalent, so even there things can be a lot smaller than they are made to be by professional mathematicians.

My actual point is that the fixation with Calculus as the gateway to higher mathematics is misplaced. There is nothing more useless that what Americans call "AP Calculus" or "Freshman Calculus", especially for people in the humanities and social sciences (who, often, take a single term of Calculus as their only exposure to 'higher math'). I would much rather teach people "finite mathematics".

Bush is a symptom, not the disease.