“But the audiences, the opinion panels, the surveys”
It was a Starwars movie, it was a guaranteed success at the box. The viewers can handle a little disappointment for a better story.
I could handle a non sensical character for a long stretch if at the peak of my rage that character became a such pivotal one, with a twist so unexpected and deep applied to it, it forced me to go back and review the entire story with new eyes.