In episode 5, he beats Luke easily but didn’t kill him because he wanted Luke to join him to overthrow Palpatine.
In episode 6, Luke is no master but also no slouch, and Vader’s conviction isn’t as strong. Plus Luke uses the dark side to beat him while Vader didn’t have the same emotional drive to beat him, so Luke might have had a stronger connection to the dark side in that moment and surprised his father who was used to dominating any conflicts he had. Gotta keep in mind that in that moment, while Vader was using the knowledge of Leia to taunt Luke, he also just learned he also had a daughter. And one of the main themes of the OT was that even a villain like Vader wasn’t completely beyond redemption, even though the Jedi themselves believed the dark side was a black and white thing that one couldn’t come back from.
Not to defend the franchise overall (honestly, fuck them for not giving the guy that brought Vader to life a cent for Return of the Jedi), but I don’t think Luke beating Vader is one of the problems with it.