In mid-1940 a pro-peace section of the Conservatives move to replace him with Halifax.
Attlee, Simon and the King still wanting to continue the war, but the King is unwilling to ask Churchill's rump War Conservatives to form government, even though Labor plus Liberals plus War Conservatives look likely to have the support of the House.
Attlee then invites Churchill to join the Labor Party, moves the motion that he is elected as leader, and the King asks Labor to form a government.
Churchill is returned as Prime Minister of what he calls the National Government ...
Churchill can't be replaced as Prime Minister unless he resigns and instructs the King to appoint Halifax. Which he won't do.
What you'd need to happen is the King sacking Churchill on his own accord, and installing Halifax. Who would then be subject to a vote of no-confidence in the House. If the vote succeeds, under normal circumstances you'd have an election - but in the middle of the War, I think Halifax then advises the King to recall Churchill. No change of party required.
The King's authority is, of course, terminally weakened. Another abdication might be on the cards (14 year old Queen Elizabeth II?).