TLDR Warning, lots of text and brainstorming. If you want my basic thoughts just go down the the third to last paragraph for my starting conclusion.
I do not believe Europe was destined to be as divided as it is today, not by geography alone, but rather the climate at the time. Between France, Benelux, Central Europe, Eastern Europe, the Carpathian Basin and the Balkans there is no clear natural division and there are countless rivers to connect it, non-Mediterranean Europe is basically a River orgy. Today the area is also very fertile, but that was not always the case. The Channel, Oresund, the Pyrenees and the Alps are really the only natural obstacles I can see. The Carpathians or the Sudetes really are not all that formidable barriers, and even the big obstacles like the Pyrenees and the Alps are not impassable, they can be circumvented.
Before I start brainstorming, I just want to point out that I did not forget Rome. Rome or a Roman successor state could've come to dominate most of Europe. It's just that I personally believe a periphery, be it Italy, Iberia, Britain or Scandinavia are disadvantaged at conquering/holding Europe. Also for a state with its core territory in Southern Europe it becomes harder and harder to dominate Europe the closer we get to today unless it can successfully integrate and assimilate most of it. Southern Europe had a huge advantage early on due to climate and proximity to the near east where western civilization first started and spread from. But the closer to today we get the rest of Europe gets an increasingly warmer and milder climate, and thus the center of power moves away from the Mediterranean, as its agricultural and population advantage diminishes. Rome can not predict the future, I see them focusing and prioritizing the Mediterranean and the Near East, it only makes sense. I could see them form a Mediterranean state similar to China. A Roman Europe would be harder, that would be if they survived as a united Empire until central Europe became profitable to conquer. If they somehow conquer and Romanize central Europe then I can definitely see a more Roman Europe surviving, be it though the empire or successor states.
Personally I believe the lack of a large historical European state depends more on that central/western Europe started to develop much later than the Mediterranean did and was heavily shaped by the politics of Mediterranean states, Rome especially. Europe's first major civilizations were all around the Mediterranean, often on the periphery. The rest of Europe for lack of better words was unsuitable for major development early on. Specifically in the critical time in history when agriculture and civilization begun to rise, spread and develop. The distance to Mesopotamia/Egypt did not help.
For Europe what this means is that there was no major civilization that early on in history could become the main occupant of the European plains+Central Highland, or since the terms are flimsy, the area west/north/northeast of the Alps, roughly modern France to modern Poland/Czechia/the Carpathian plains. The area that today I would argue is the core of Europe, and the core of any type of theoretical alt-historic pan European state that is not Rome or a successor of Rome. (1) So there was no early state/civilization to grow and cement a sense of unity or even tradition of unity in the area. Yes Rome did conquer Gaul and did a halfhearted attempt at western Germania. But the fact that they did not succeed meant that now northern/western Europe had a fairly major Roman/German linguistically and political divide. Later made even more prominent with Slavic settlement in Poland. This hurt possible unity more than it aided it. An area which otherwise would've been perfect as a core of a stable, fertile, easily connected Central/West European state instead ended up divided between 3 different cultural/linguistic groups.
Had Central/Western Europe been warmer and thus been developed/deforested earlier then I can see having a successful civilization develop along the river orgy in France/Germany/Benelux early. From there on it's not that far fetched to have such a civilization spread to the Alps and Pyrenees, into Poland, Austria, Czechia and possibly Lombardy fairly easily. Then you got the basis for a state controlling most of Europe, it already controls the center, it's fertile, got a fairly mild climate and the rivers help make the state interconnected. But this was not the case, Central/Western Europe when the Romans showed up had been sparsely populated throughout most of pre-history, was heavily forested and had no large unified states (2). There was a short lasting Carolingian Empire, but that failed to remain united and was divided pretty close to the Roman-German cultural and linguistic line.
So in my opinion, Europe's failed unity has its base in that its early civilizations were Mediterranean based and that's where their focus understandably was. Northern Europe's climate/ice ages made the area unsuitable for early development or major population centers. Later on the disparity in development, the Roman/German divide and the political situation made unity... hard. Central Europe, NE Plains and the Carpathian basin, the large area in Europe that lacked natural borders and was interconnected by all the large rivers had been divided between a multitude of competing ethnicities, languages and states. From this point unity becomes very hard. Compare that to China, where the core territory of modern China was united very early on, and a tradition and sense of belonging to the same group of people took grip early on. This is the opposite of what happened in Europe, and I believe it has to do with the core territory of modern China having favorable climate to develop early on. While the core territory of modern Europe developed very late. In Europe the periphery developed first, and due to climate and geography it had a hard time spreading into the area that is today the European plains, all it got was Gaul, Dacia + minor parts east of the Rhine and North of the Danube, which just created another divide.
So in conclusion, I believe for Europe to be as united as China historically you need western and central Europe to develop much earlier, either to create a native civilization or to make it more tempting for a Mediterranean based empire to conquer. But that is hard due to the climate and lack of incentive, but possible. Other than that the only way I see a "European China" is to somehow change when the first major civilizations centers started to develop in the rest of the world, to make it coincide with one of the warmer period in Europe, when it would be more heavily populated and thus more easily farmed and settled. But the butterflies would be the size of moon, and the flapping of their wings would displace cities, entire nations and cultures.
1. I do not mean to insinuate global importance here, Iberia, the British Isles, Italy, Greece and Scandinavia all make up important parts of Europe and several held massive sway in the world until very recently. But in the theoretical state I am talking about they, while all important parts of it, would at the same time also a periphery and not necessary for the survival of the state. Unlike western mainland Europe and Central Europe which I view as absolutely critical to such a large European state, both positional and agriculturally.
2. Yes Gaul was developing and had smaller states/major tribes, but compared to the Mediterranean world it was too late. Had the Gauls been 500 years earlier to develop then I could see the region having the time to develop a dominant state in Gaul, creating a more organized state. Which could have acted as en embryo for the theoretical Central/Western Europe state I am theorizing here.