Empire was the standard form of Government around the world for millennia and prior to the rise of nationalism nobody doubted its legitimacy as a form of government very much. Nations (ie groups of people sharing language, religion etc) would happily throw off the yoke of Empire over them and then immediately conquer their neighbors to make their own empires without a trace of irony.
This requires a pre-1400 POD if you mean European transcontinental imperialism. It could be a cultural or religious turning inward like in China after Zheng He. But if this comes to happen sooner or later some culture with no such inhibition would fill the same role.
It really depends on the definition of imperialism. If you mean that groups don't conquer other groups then it is pretty much ASB, as humanity has been conquering each other since the formation of tribes. It's probably ingrained in us genetically.
If imperialism never happened in the communist sense of the word, mankind probably progresses slower but fairer sort of like of like the better half of native American tribes.