The name “Saxony” can be applied for a variety of things, which leads into bewildering amounts of confusion. The various varieties of Saxonies therefore deserve a lengthy explanation here:
The original Saxon people emerged in Late Antiquity in what is now northwestern Germany (approximately the area of modern Lower Saxony, Westphalia and parts of Schleswig-Holstein). Part of them migrated, along with the Angles and the Jutes, to Britain, whereas the mainland Saxony was conquered by the Franks after a prolonged military campaign in the late 8th century and absorbed into their realm. Subsequently, the “stem duchy” of Saxony was established, which became part of East Francia after the Treaty of Verdun (843) and later the Holy Roman Empire. In the late medieval ages, Saxony fragmented, resulting in the bewildering ambiguity we know today.
The term “Lower Saxony” was actually applied first to the Lower Saxon circle, an Imperial Circle (Reichskreis) established as an administrational unit of the HRE in the early 16th century, in order to distinguish it from the “Upper Saxon” circle (covering the modern states of Brandenburg, Saxony and Thuringia). The modern German state was created after World War II by the British military administration as a merger of the states of Brunswick, Oldenburg and Schaumburg-Lippe with the former Prussian province of Hanover. As a result, Lower Saxony is the only modern entity bearing the name “Saxony” which actually is located largely inside the area of Old Saxony. Also, the local Low Saxon dialects are actually derived from the Old Saxon language, unlike the other Saxon dialect (see below).
The final entity to be called “Saxony” is the modern German state of the same name and its predecessors. It is however “the” Saxony that will come into the mind of most Germans when they hear the word. By ethnicity and language, the state is utterly unrelated with old Saxony. The original population of the area (the Margraviate of Meissen, also known as the Thuringian March) was actually West Slavic. To a small degree, it still is today, in the shape of the Sorbian minority. The distinct Saxon dialect (which is something the state is infamous for, and ridiculed on by the rest of Germany) is actually a highly aberrant Middle German dialect, derived from Frankish and not from Old Saxon. This schizophrenia comes from the fact that “Elector of Saxony” sounds more funky than “Margrave of Meissen”. Under Napoleon, the Electorate was elevated to the status of Kingdom, which it remained until monarchy was abolished in Germany in 1918, when it became the Free State of Saxony.
The Prussian province of Saxony was established after the Napoleonic Wars and incorporated territory previously part of the Kingdom of Saxony, hence the name. After World War II, Provincial Saxony (minus the areas that were given to Thuringia) was merged with the state on Anhalt to form Saxony-Anhalt.
Because it is so awesome to have “Duke of Saxony” in your title, a bewildering number of entities had the phrase “Saxe-” in their name, which applies in particular for the bewildering number of Thuringian states. The best-known example, of course, is Saxe-Coburg-Gotha, which managed to churn out an insane number of monarchs across Europe.