The movie is certainly messy and the film gets a bit rushed towards the end. There certainly enough potential for a fourth film to be found. Things like Leia's mother having been Jedi who was killed when she was a little girl by Vader should have been a cool moment rather than something Yoda tells Luke halfway into Act 2. It should have been a bigger part, instead, it's just sort of an excuse for Leia to learn the force and get a lightsaber (Fisher always jokes about how the prop they gave her looked like a big purple dildo) and fight Vader to avoid having Luke try and kill his father. Still, Speilberg did bring his standard "big emotional payoff" touch to the whole thing, so by the time, Leia refuses to kill Vader in revenge to pay homage to her mother most people tend to forget that a lot of that payoff isn't really earned. Especially when Vader decides to save Luke at the expense of his own life because of it. The twin lines from Leia and Vader ("No I won't kill you, I'm a Jedi, like my mother before me" and when asked why he would stop the Emperor from killing Luke Vader's response of "Because I am Anakin Skywalker, a Jedi Knight") are pretty much up there with "may the Force be with you" as lines people who haven't even seen the films at least sort of know. The whole Luke and Leia hook up just after we hear as an audience that Han is dead sort of feels gross, and it would have been nice to see them get justice on this Jabba person, but like I said the film is pretty packed as it is and they likely couldn't find a way to fit it in.
I think at a certain point Star Wars fatigue had set in. The film isn't exactly a disaster, it did plenty well at the box office, but even Lucas seemed to recognize that the story was getting repetitive. There's a reason that the property has been more or less dead since outside of a few licensed video games and a couple of comics that he has no hand in and are basically just contract riders that he sold decades ago. I know HBO keeps saying that they're planning on doing a prequel series about the rise of the Empire and the Clone Wars that barely get mentioned but they haven't even found actors to play the main cast of Obi-Wan, Anakin, or Leia's mother Padme, much less anything else. Lucas says he plans to write for it, but pretty much everyone agrees that this just means that he'll write the basic outline and world-building and leave the rest up to the showrunners. That's been his MO pretty much since he retired from actual directing in 1999.
