From p. 52 of The Hammer and Bayonet: A History of the Communist Party of the Soviet Union Until the End of the Second Great War by Colin Morford, 1948
...However, the turning point in the war between Trotsky and Stalin came on 20 April 1923, at the Twelfth Party Congress in Moscow. By this time Stalin had already begun to use his position as General Secretary to his advantage, having appointed many of his followers to local delegate positions. However, most of the other voting delegates were still unaligned with either Trotsky or Stalin, and were undecided as to whom they preferred. Furthermore, even Stalin's supporters were largely unaware of the tension between Stalin and Trotsky, and so, for the most part, they still held Trotsky in high regard.
On the 22nd, hours before Trotsky was due to deliver his speech, a rumour spread through the Congress that Lenin had arrived to speak. Some initially disbelieved this--Lenin was not listed anywhere on the agenda, and as far as anyone knew he was still recovering from his latest bout of fever at the Gorki Estate--but, sure enough, in the late hours of the afternoon, a somewhat shaken Stalin, who was about to give a speech denouncing the localised imperialism of the Georgians, called for the audience to turn their attention to Comrade Lenin, who was about to deliver an unexpected oration. Any buzz that had been stirring in the hall was immediately silenced as the delegates leant in to listen to their respected leader.
Lenin's words found Stalin completely off his guard. Though they began tamely enough--a typical call for unity in the Party, a reminder not to forget the purpose and legacy of the October Revolution--they soon evolved into an attack on Stalin's increasing greed for power. Through his post as General Secretary, Lenin warned, Comrade Stalin "[had] unlimited authority concentrated in his hands", and he could not be expected to "[use] that authority with sufficient caution", as shown by the fact that his "appointments to the Central Committee [had seemed] to be based on loyalty rather than merit or dedication to the efficient progress of socialism". He was therefore "unfit to serve the position of General Secretary", and he should be replaced posthaste with "a man more tolerant, loyal, considerate, and less capricious" to serve the vital role. Comrade Trotsky, conversely, possessed "the outstanding ability to embody the traits of a proponent of the Communist cause", and was "undoubtedly the most able man in the Central Committee".* Though Lenin did not explicitly recommend a man to replace Stalin as General Secretary, the implication was quite clear.
Despite Lenin's coughing (One delegate wrote in his journal that it was quite obvious that Comrade Lenin was ailing), the words rang clear in the minds of the delegates. By the time Lenin finished his subdued attack, Comrade Stalin was red with anxiety. If it had been Trotsky or Zinoviev who had dared to speak out against him, he might been able to persuade the delegates in the seats that the attacker was not to be trusted, and that such an address was a blatant attempt to destabilise the Party by an enemy of the Revolution. But no one was more respected than Lenin; to the Bolsheviks, he was the wisest, purest source of Revolutionary perfection. To attack him as a traitor to the Revolution would be to hasten his own demise. Lenin had already turned much of the Central Committee constituency against Stalin in one short announcement.
The worst, however, was still to come. To Stalin's heightened horror, Trotsky's speech was not the expected, inoffensive request for party democracy, but an impassioned tirade on the dangers of the new "autocratic despotism" of the existing order and the increasing "bureaucratisation" of the Party, which, he argued, had allowed "enemies of progress" to infiltrate the Party ranks and "sabotage" the Party's economic plans. By the end, Trotsky's every remark was accentuated by raucous, fervouristic agreement. One Bukharan delegate would later write that "near the end of Comrade Trotsky's speech it was nearly impossible to distinguish the words that he was saying amid the noise, and yet it was just as difficult not to join in the shouting and denounce Secretary Stalin at the top of your lungs".
In an apparently spontaneous fashion, Trotsky called for a new referendum for the position of General Secretary, to which the crowds immediately agreed. A vote was hastily arranged; Stalin watched silently as even his carefully-selected followers cast their votes against him in a fit of controlled frenzy and hysteria. After the results came out, Stalin nearly fell out of his chair: by a margin of 268 to 155, he had been ousted from the position of General Secretary. In a single gesture, Trotsky and Lenin had unseated their main rival and swept away much of the power that Stalin had spent years gathering for himself.
Memo drafted after Politburo meeting, April 28, 1923
Effective 1 March 1923, Comrade I. V. Stalin is to be removed from his position as General Secretary and appointed President of the first Policy Council [Sovpol], membership of which will be his prerogative. In order to prevent autocracy and to maintain accountability and central democracy in the Party, the Policy Council's decisions will be reviewed and deliberated by the Politburo. To give his full attention to the issues decided the Policy Council, Comrade Stalin will resign from the Politburo. By will of the people's delegates, and thereby of the Soviet people, Comrade L. D. Trotsky will assume the role of General Secretary.
*This speech, known as "Lenin's Testament", was written by Lenin in OTL (albeit in a slightly different form) after he saw through Stalin's lust for power. But he had been unable to come to the Congress to deliver it due to a stroke, which was brought on by his wounds from the shooting in 1918. Without Lenin's support, Trotsky did not bring up the issue of Stalin's consolidation of power, and he missed his chance to finally turn the Party against Stalin before it was too late.