Dicky0331: I have been using the wonderful 204 Ruger cartridge since it first became available in factory Rifles!
I now shoot 5 Varminters in this outstanding caliber.
Each and every one of my factory Rifles has shot noticeably well since each was brand new!
I have to surmise that the 204 Ruger is an inherently accurate cartridge - and is easy to load for!
Each of my 5 Varminters in 204 Ruger caliber even shot factory ammunition VERY well!
I assume your intention is to attain accuracy as opposed to all out speed or top notch lethality on Varmints - so I will direct my comments to that end as you did not specify what you are seeking specifically.
All 5 of my Varminters shot the 35 grain Berger bullets the best - barely edging out though the 32 grain Sierra BlitzKings accuracy wise!
Having said that I have now re-adjusted all my Rifles to shoot the less expensive and more readily available Sierra 32 grain BlitzKings.
Back to your inquiry and my recommendation. I suggest you start out with the 35 grain Bergers along with Federal 205 Match primers and H 4895 powder.
I use Remington, Winchester, Hornady and Nosler brass to keep the brass from mixing among my various Rifles - all brands of brass have held up well and like I've mentioned provides great accuracy.
I use the Sinclair Bullet Seating Depth Tool to adjust my seating depths. I wish I had kept better notes on this end of the accuracy equation but I didn't.
If my memory serves me correctly my 2 Remingtons (XR-100 and a 700 VLS) along with my 3 Rugers (2 Model 77 V/T's and a #1-V SS) all had LONG factory throats - in other words my bullets jump a variety of distances to the leades of the rifling!
Best of luck with your 204 Ruger.
Hold into the wind
VarmintGuy