Byrnes left the Supreme Court to head Roosevelt's Economic Stabilization Office, which dealt with the vitally important issues of prices and taxes. How powerful the new office would become depended entirely on Byrnes's political skills, and Washington insiders soon reported he was in full charge. In May 1943 he also became head of the Office of War Mobilization. Thanks to his political experience, his probing intellect, his close friendship with Roosevelt, and in no small part to his own personal charm, Byrnes was soon exerting influence over many facets of the war effort which were not technically under his departmental jurisdiction. Many in Congress and the press began referring to Byrnes as the "Assistant President."
He was a serious possibility for vice president in 1944. However, he was too conservative for the labor unions, big city bosses vetoed any ex-Catholic, and blacks were wary of his opposition to racial integration. The nomination went to Senator Harry S. Truman. Roosevelt brought Byrnes to the Yalta Conference in early 1945, where he seemed to favor Soviet plans. Writing in shorthand, his notes comprise one of the most complete records of the "Big Three" Yalta meetings.