Failing to adopt European political,economic, and military systems
That's hard to do at least economically, Japan did have economic theorists that believed in bulilonism, and wanted to conserve their precious metals like some Europeans, even if might been from seperate conclusions. Politically and military you had something of mix of institutions and weapons that would not be far from Europe, unless you mean Modern Europe 1700-1800's and beyond. If this PC is really Japan fails to westernize then, it cannot be achieved with an isolationist Japan. The Tokugawa Shogunate would be interested in new weapons and technologies, even if it is for domestic uses, especially a railroad system since most of the country is mountainous.
That would ironically enough no isolationism is the only way this challenge works if we are talking about failing to "modernize" to mean actually fail to westernize. Japan would have to develop its own institutions to compete with Europe if that counts but not have adopted European models, which is possible with an earlier timeline.
Mind you regardless of what happens Japan was never destined to be expansionist but in order to play the European game of imperialism that is how they turned out. Without a "Westernized" Japan, more racism is hard to pinpoint, on one hand, no way to really expand the Yellow Peril without Imperial Japan. On the other hand, no Imperial Japan to stand up to the white European powers in Asia, So I'd assume colonialism lasts longer with no threats to European dominance in the East, depending on the timeframe of the POD, no Russo-Japanese war in sight, so the Tsardom of Russia lasts longer. Asian decolonization might take a much longer time to happen with no disruptions from Japan.
It can't really be overstated how critical the Meiji institutions were to the speed of Japan's modernization. Without the restoration, Japan will not develop at anything like its historical trajectory from the 1870s.
A POD in the Second Choshu Expedition or the Boshin War could prevent it. An earlier, more forceful French presence in Japan could easily tip the balance of power to the Shogunate forces against its enemies. It's possible different events in China or the American Civil War could trigger that.
Then this is possible short of a straight up invasion by another power, keeping Japan back somehow. Both sides in the Boshin War used modern weaponry, and the war was not such much about progressive Imperialists against the backward Shogunate, but more who would take power in the new Japan. It would not prevent changes to Japan, it would only impact how it happens, and who oversees it. The Tosa domain, Imperial loyalists wanted Tokugawa Yoshinobu to govern a council of daimyos, but not as Shogun before the Boshin War broke out and he accepted. However, the Choshu and Satsuma domains would not accept it, and that led to the Boshin War in the first place.