The following represent my own personal point of view (haha) with regard to where I believe George Lucas intended to take the Star Wars saga in Episodes VII and beyond. I have not read every scrap of Expanded Universe, Canon, New Canon, etc. literature so you are welcome to argue that I am right or wrong based on your own point of view.

It has been said many times that the current run of Star Wars films (Episodes VII through IX) were informed to some extent by notes George Lucas commited to Disney and Kathleen Kennedy. Like any writer would, each film's script team created their own story, borrowing (heavily or lightly, who can say) from those notes. We also have an interview with Geoge Lucas in which he stated his intention was to explore the realm of the Whills, which would delve more deeply into the nature of the Force. So where do our familiar heroes and villains enter into such a story?

Lessons Learned

Throughout the original and prequel trilogies a common theme is learning a lesson. Invariably these lessons involve a personal choice with wide-reaching consequences, often painful. Anakin rushes Dooku and loses an arm. Luke rushes to confront Vader and....loses a hand. Obi-wan defeats Maul with a move that later costs Anakin the rest of his limbs (since Obi-wan well knows how to defend against that attack). Yoda presumes that he is the only Jedi strong enough to defeat the new Emperor Palpatine, and has his belief in the dominance of the Light Side shattered, sending him (pouting?) into exile.

So exiting Episode VI, what has Luke learned? Yoda taught that once someone has turned to the Dark Side they are beyond saving — a pro-Light Side philosophy that hinges on the Dark Side being equated with evil. But as Chancellor Palapatine (and later Obi-wan) points out, good and bad, Light and Dark are relative concepts. Just as falling to the Dark Side is a failure of the Light, returining to the Light Side is a failure of the Dark. But what Luke came to realize is that Vader's redemption was not a failure at all: it was a triumph of the one thing stronger than the Force, free will.

Vader's saving Luke demonstrated that no matter how powerful the Light or Dark Side, free will is not subservient to it. In retrospect, so many of the critical changes wrought by characters revolve around free will and ignoring the mandates of logic, duty, or dogma. And each time such a choice is made, the consequences are far reaching.

The Wheel of Time

In the prequel trilogy we are shown a galaxy ruled by the Light Side, where the ponderous machinery of governance by majority appears to be maintaining the status quo. The Jedi, with a rigorously-dogmatic philosophy, are impotent to effect change or influence events. The galaxy is not moving forward or backward, merely maintaining its existence.

Next, the Dark Side takes over and implements an entirely different form of governance. In stark contrast to the representative government that preceded it, the Empire is the imposition of a single will upon the galaxy. The Rebellion, a civil war, is an abundantly clear roadblock to any forward momentum in that agenda. Thus, the galaxy is again unmoving under the rule of Palpatine.

Assuming the meaning of existence is betterment of the sentient beings' (human) condition, neither form of government seems to work toward that end. But the strories center on drastic change as a consequence of individual choice. The union of Anakin and Padme yields the foundation of the Empire as well as Luke and Leia, the destructors of the Empire. Luke's flight to Cloud City provides him the contradictory, humanizing information (and doubt) that eventually leads to Vader's redemption rather than destruction. Jar Jar Binks, seeking the recognition any ignorant, downtrodden being would desire, ushers Palpatine into office. Individual choice under free will influences every turning point in history.

Light versus Dark

The nature of the Force is presented in black and white terms in the original and prequel trilogies. The Light Side eschews personal desire and intention, making the Jedi a vessel expressing action as controlled by the Force. The Dark Side is diametrically opposed to this philosophy, with Sith seeking to control the Force by means of personal desire and intent. However, once all adjectives have been established to be relative, the Force itself can have no absolute point of reference and in its purest form is free from all such interpretations. Its nature is personally established by each individual's experience.

The Force is not Light or Dark, good or bad: people are. And if free will is stronger than the Force, then free will must be the most pure expression of influence on the galaxy. Thus, Vader's redemption indicated to Luke that the Force can be a powerful aide in individual choice, but never a full replacement for choice, the same way that a wise old man like Obi-wan can help guide someone toward his destiny without outright leading him there. Likewise, Palpatine's reaction to Luke's refusal to turn to the Dark Side shows that he was truly surprised by that turn of events, despite everything's proceeding as he had foreseen up to that point. If the future is always in motion as Yoda told Luke, then something prevents the Force from showing a singular path ahead, and it must be outside the influence and control of the Force. The choices made by people control the future, not the Force.

All of this leaves Luke conflicted. Yoda's suggestion that he pass on what he has learned and Leia's desire to restore the galaxy to the ways of the Old Republic both suggest that by need Luke must create an Order founded on the principals of the Jedi (the Light/good) but which explores the integration of personal choice and free will (the Dark/evil).

The Force Awakens

Luke naturally would feel some trepidation about departing down a new path of interpretation of and teaching of the Force. With the galaxy needing some kind of order and rule, he would need to quickly respond in concert with Leia's efforts to establish governance. Reestablishing the ideals of the Republic and the Jedi is, in essence, the quickest way to achieve those goals. Ironic, since Luke knows the Dark side to also be the quicker, easier path.

Unfortunately for both twins, in a very short period of time both the New Republic and the new Jedi Order are under siege. The remanants of the Empire seek to reestablish their rule over the galaxy, and the Jedi Order is subverted by a single wayward pupil. Both have failed just as their predecessor did, and in a ridiculously short period of time.

Luke realizes that polarizing factions will always exist and will always clash due to their conflicting points-of-view. All previous approaches to establishing peace and order in the galaxy have failed, so they were doomed to fail. This is the final piece of evidence needed to convince Luke that a new philosophy is necessary, one that does not embody right-and-wrong or Light-and-Dark. Into exile he must go, to contemplate such change and how it can be effected and sustained. This is a parallel with Yoda's own exile, but where Yoda seemingly disappeared to await the return of the Jedi, Luke is instead disappearing to contemplate the evolution of the Jedi.

The key plot points presented by The Force Awakens are the spectacular failure of old concepts of rule and order; the struggle that inevitably occurs when polar opposites are present; and the need to find a new path forward using the lessons of the past.

The Last Jedi

When Luke is eventually discovered by Rey, she assumes that he will train her to be a Jedi. This could not be farther from the truth, since Luke has spent years coming to a single conclusion: to evolve beyond past failures, neither Jedi nor Sith factions can be allowed to control the galaxy. Thus, the Jedi must end; by extension, so must the Sith.

What Luke learned from his father was that free will has dominion over all. Pure Jedi do not allow free will to dictate their actions, Sith allow free will total control. Luke himself took a step down the road toward the Dark side as he concluded his final duel with Vader: he understands just how alluring the Dark side can be. In that circumstance the redemption of his father was his motivation, though, and that helped him resist the urge to use the Force as a Sith would. So what holds back Luke's new philosophy of free will in temperment with the Force is, coincidentally, free will. Without Vader's redemption on the line, would he have been able to resist the power he felt?

Rey's insistence that Luke train her to be a Jedi only strengthens the true nature of the problem: people are inherently comfortable with diametric binary pairs. A new philosophy will inevitably itself create such a pair: those who subscribe to and adhere to it, and those whose do not. Luke cannot alter the nature of people. He feels utterly lost because he realizes the path he seeking will have the same consequences, thanks to free will. Yoda appears to him in this desperate hour, and burns the library tree to the ground, seeming to indicate that Luke must let the past go, or destroy it if it holds him back. This mirrors the philosophy of Ben/Kylo, who says that Rey must "let the past die. Kill it, if you have to." (As pointed out in this online video, in exile Yoda's own philosophy of the Force seems to have changed slightly since Jedi Order days while Obi-wan's has seemed to remain unchanged — making a visit from Yoda apropos of Luke's path.)

The Last Jedi shows Rey and Kylo — the Light and Dark — working together, symbolic of the combining of Light and Dark that Luke has been trying to create. Their partnership is just as short lived as the New Republic and Luke's new Jedi Order, though, when they attempt to turn each other to their respective ideals. Naturally, they fail. We also find that Rey has taken the Jedi tomes from the tree prior to Yoda's burning it: she continues to look to the past. Kylo wishes to destroy the past, which he acts out violently against Luke: he has the right idea, but lets his own will and desire corrupt his intentions.

Enter the Whylls

George Lucas stated in an interview that post-Return of the Jedi he wanted to delve into the Whylls, the underlying nature of the Force. The Force is described as having a voice that speaks to those who are deeply connected with it. Sith believe themselves to control the Force, corrupting it to their own desires. But if the Force is agnostic to Light and Dark, then it's equally possible that both philosophies are simply serving the designs of the Force — the plans of the Whylls.

The Whylls are presented as mystical, immaterial beings in the concluding episodes of the Clone Wars cartoons. When Luke becomes one with the Force, he enters the realm in which the Whylls exist. Armed with the knowledge he gained in life — free will is stronger than the Force, but is also the primary problem that sentient beings must overcome to evolve — Luke has one path forward. He's seen that Light versus Dark provide powerful opposing pairs that constantly leave the galaxy in ruin. The Whylls' agenda is to keep the wheel of time moving in that same cycle of shifting power, destruction and creation. To escape that cycle, the influence of the Whylls must be removed. That the Whylls' own agenda influences events in the galaxy implies that free will can be strongly influenced by external factors. Removing their influence serves to elevate free will, perhaps even purify it.

Assuming Luke is successful, the Force would cease to influence the galaxy. The people of the galaxy would become wholly responsible for their present and future, no longer being pushed by the Whylls on a ceaseless and useless cycle of existence. Without the Force, there would be no Jedi or Sith, either. They would fade into the past, existing only as epic stories of a bygone time rife with magic, until eventually the tales were a reality a long time ago. And perhaps those stories would be told by their descendents, who spread from that fabled galaxy to other galaxies, far, far away.

In Summary

The character arc that I believe Luke's life was taking was one of discovering that free will is the most powerful aspect of human nature. When he said that the Jedi must end, he was recognizing that:

• classic Jedi dogma allows free will to be subverted by the Force
• by contrast, Sith use the Force to impose their own free will upon others

Neither point of view is useful to moving humanity forward because the duality necessitates conflict. Unifying the two points of view to allow the Force to be used helpfully in promoting free will suffers one major glitch: it still implies assisted promulgation of the ethics, views, or desires of the individual or group. Thus, the only true balance in the Force is complete and utter disconnection from it.