You’re the asshole in that situation. Don’t expect that any parking place is yours until it’s empty, especially when you have no idea when the people actually parked in the space are leaving. You should never be blocking traffic waiting to park in an occupied space because you assume they’re leaving.