There's a few ways to go about this. As you've mentioned, a core drill will perform well, but it's probably unjustifiably expensive.
Likewise, as Alex suggested the Bosch Speedcore will perform the task well, if a trifle expensively. The huge advantage of the Speedcore is that it uses a pilot bit for guidance. Once the pilot punches through the other side, if you turn the paver over to finish the hole you'll minimise breakout. Your Milwaukee cordless SDS should be darn near perfect for this type of bit.
Those diamond holesaws mentioned seem to be pilotless, & you'd need extremely accurate measurement on both sides of the paver to core through the whole paver's thickness from both sides effectively "blind". In addition, these tools' shanks are a standard 3/8 hex or round, & won't work well even with an accessory SDS+ adapter chuck, tending to slip in the chuck (if round) due the high torque required to drive such a large diameter.
You shouldn't be using any impact at all with gritted holesaws. Meaning that any quality lower-speed high-torque drill will work with these. I've had good results with a (corded) Bosch GBH 4 (which has interchangeable chucks & drilling/hammering/chipping modes) with large diameter holesaws in a variety of materials.
I also think Alex's suggested "budget" method of multiple holes within the circumference of your required diameter would work, albeit slowly. You could use any bit that you already have, although smaller diameters will drill faster & minimise breakout on the bottom. However I wouldn't recommend using a rasp to clean up the circumference. Concrete is too hard, esp. the high MPa concrere used to make pavers. I doubt anything less than diamond would abrade this type of high strength, brittle concrete.
Better yet would be a stone grinding disc fitted to a mini-grinder, which will make messy but fairly short work of any imperfections in your hole. Had to do this myself to "square off" the overlaps of 3 x 300 mm cores through a concrete tilt panel. Appropriate PPE is a must: silicates in cement are dangerous. You'd have to use the grinder from both sides.
A bit of clearance around the hole is advisable when using softer PVC plumbing fittings through hard or abrasive materials to allow for thermal expansion, movement & the possibility of replacement. Just a few MM would suffice: say a 4 1/4" hole for a 4" pipe? A bit more if you have to feed a female flared end through the paver.