SSV1761982: I feel somewhat "qualified" to answer the question you pose.
I have now brought online 4 different Varmint Rifles in caliber 204 Ruger. I have shot both 32, 35 and 40 gr. bullets in all three of them.
I broke all four 204's in with 32 gr. Hornady factory ammo - with the first three Rifles there was no factory brass to begin with and I bought a large supply of 32 gr. factory ammo then went to 40 gr. factory ammo then 35 gr. handloads and then to 32 gr. handloads for those 3 Rifles.
My latest 204 Ruger got broke in and initially sighted in with some left over 32 gr. factory ammo then I tried the 40 gr. factory then my 32 gr. Sierra handloads.
So, "I been around" with different bullet weights in the 204 Ruger.
2.5" difference in P.O.I. between bullets that are 20 to 25% different in weights is nothing!
I am going from memory here and visualizing my "official BR targets that I prefer to do all my load testing and sight in's on - and I am sure several of my load changes resulted in P.O.I. shifts where 4" differences were the norm. I also note that these P.O.I. shifts were not just vertical in nature but were horizontal to some lesser extent as well. IIRC. I did not make note of the differences in P.O.I. shift with different bullet weights as I have never felt it important to do so.
Over the past 5 decades of shooting I have shot numerous Rifles that have printed the SAME weight bullet (different manufacturers of bullets and/or slightly different velocities/loadings) that have had far greater dispersals of their P.O.I.'s than you have enumerated!
I would not concern yourself with these small (or any!) differences in P.O.I. between different weights of bullets. Its of no real consequence in real world shooting. Once your Rifle is sighted in with its best (most accurate?) loading/bullet then forget about the P.O.I. of previous loads tested.
A whole HOST of issues determining P.O.I.'s come in to play when you change bullets - things like barrel harmonics, recoil, length of time the bullet is in the barrel etc etc etc.
Frankly I think 2.5" difference in P.O.I. is a rather minor difference in bullets that are 20% different in weight, that have 10%+ differences in velocities and maybe about 10% difference in length.
Anyway glad your 204 is shooting well and I wish you continued good luck with this fine cartridge.
Long live the wonderful 204 Ruger cartridge!
Hold into the wind
VarmintGuy