Fascia board install - unlevel rafter tails


I’m a first timer in the process of replacing fascia board. House has a hip roof with 2x4 rafters square cut to 2x4 fascia board. First 18" of plywood is taken off all the way around. I took the old fascia board off and ran into an issue - rafter ends are not level horizontally (up to 1/2"). I think it’s mostly due to shims (between top of the wall and rafter) not being of a consistent thickness as well as several rafters having some axial twisting.

My question is how to choose the best level to start from? My first thought was to put up a first 2x4, try to average out how much rafter ends are not flush with the top of the 2x4 with the same amount of other (lower rafters) below the bottom and find a median. With this in mind I installed first board at the corner, but after turning the corner and nailing 2nd board level with the first one, I ended with all rafter ends (that 2nd board is sitting against) higher that the board, i.e. my original “centerline” was chosen pretty low. I was wondering if there is a better way of doing this? Should I run a string line around the whole house to get a better level idea of all rafter tails? Should I base my level of the lowest rafter end and let plywood follow the rafters as they are instead of installing fascia board higher than some low rafter ends and forcing plywood end to sit of the fascia board with a gap to those rafters? That way I will have some gaps between plywood an 2x4 fascia, which will be covered by drip edge anyways.
Pardon my long write up.
Your input is greatly appreciated.

Like putting a bandaid on a bullet wound!

Post some pics, and we can give better advice.

Yes giper, youve got it figured out. Just try not to set it too low.
I wouldnt want it all messed up because of one or two stray too low rafter ends.