This is not the era of the Divine Right of Kings. One "bad" king out of how many? Not such a big thing. It isn't like Ernest can just order people executed, a la Henry VII/VIII, burned for heresy a la Mary I, this is a mostly symbolic office that has a set of rules that he will follow in public (now, you couldn't pay me enough to work for him, but that's a whole other can of worms), the nation will survive the fifteen or so years he has left to live (he'll probably die sooner than 1851 with the additional responsibilities, the hard work seems to work for the gals but not the guys).
The only way I see any sort of consideration of changing the succession is if the upcoming candidate for kingship has Down's Syndrome, unpredictable seizures, is so crazy he is talking to elves during the opening of Parliament or during public occasions. But "we don't like him" doesn't cut. For one thing, "we" consists solely of nobility, Parliament and the servants. Most of the populace only see the King ceremonially, so rumors of his bad temper or the fact that he tries to molest all the maids, will be just that - rumors. He gets crowned, they're going to support him.