Not at all. His succession is in accordance with the law. He didn't kill Nixon or poison Ike. Furthermore, the same electorate that voted for Ike in '56 also voted for a Democrat majority in the House.
Rayburn would operate as a pure caretaker President. He'd retain all of Ike's cabinet secretaries, and try to follow Ike's general policies where he could.
I quite agree with this. In the event of the scenario described, President Samuel Rayburn would take office legitimately, in accordance with the law, and most people would recognize that fact. The first major decision to make in his case is whether to stay on as President, or to step aside.
If he chooses to remain in office, then he is in the awkward position of being a President from the Democratic Party, overseeing an Administration established by a Republican. Even so, he might give it a try. He will most likely keep Eisenhower's appointments to the Cabinet, excepting those who resign because they are unwilling to serve in Cabinet under a Democrat. In that situation he is in a difficult position. If he nominates Democrats to the new holes in his Cabinet, the Republicans accuse him of hijacking the government, but if he nominates Republicans, then he faces attacks from within his own party, not to mention he would need to get Republicans who were both reasonably suitable to the jobs,
and willing to serve.
He would likely try to stand by President Eisenhower's policy positions on most issues, even when they were positions he did not necessarily agree with, but he would face pressure to "moderate" things in some cases, and the level of criticism on both sides in any situation would likely be almost constant, regardless of the decisions he made.
In the end, I think it likely that a President Rayburn would choose to push for a new Speaker to be elected in the House, and he would resign quickly once the man had been confirmed.