Is it normal to get annoyed by the sidewalk shuffle?

That's because no-one in civic government (in many places) had established clear-cut rules for high-volume pedestrian traffic areas, that everyone understands and follows.

Here would be an interesting rule:
That everyone walks towards the inside when proceeding generally in two cardinal directions, and everyone walks towards the outside when proceeding generally in the two opposite cardinal directions.
In this way, you could be walking either on the right-hand side of the sidewalk, or on the left-hand side of the sidewalk; while proceeding in the same general direction, depending on which side of the street you are on (where there are sidewalks on both sides of the street).
Better than mundane right-side, left-side vehicular-type rules; and it could be applied universally, as long as municipalities agreed on a standard of which cardinal directions to use.

I think inside Northerly and Easterly; and outside Southerly and Westerly, would be easiest for everyone to remember (clockwise on a compass dial).