The Secret War: Espionage in WW2
Sun Tzu remarks that “All warfare is based on deception.” It must then be equally true that warfare demands penetrating the veil of deception. While armies of millions clashed with titanic thunder, often the most important battles of the war occurred unseen, and often the full value of their contributions would not be felt until many years after the war.
The UASR’s intelligence services can be seen as scions of the Soviet intelligence services. Much of their personnel began as infiltrators and informants trained in the craft by GRU (Main Intelligence Directorate) or OGPU (Joint State Political Administration) handlers. The network of Soviet residents, moles and informants served a dual purpose, funneling intelligence to two masters, the Soviet government and the Workers’ Party of America. With the outbreak of the civil war, this network became a servant of the Provisional Government. Having thoroughly penetrated many sectors of industry and the bourgeois state, the well trained network of spies kept revolutionary forces one step ahead of MacArthur. The compromise of the Generalissimo’s operational plans gave Patton all the leverage he needed to engineer a crushing victory in the Battle of Washington, effectively winning the civil war in a single stroke.
With the establishment of the revolutionary government, the American secret services established themselves on a global scale. The military Main Intelligence Directorate and the civilian Committee for State Security echoed the division between the Soviet GRU and the OGPU (reorganized in 1934 as the GUGB – Main Directorate for State Security), and the occasional rivalry that such divide entailed. In general, the division of labor was that the MID would focus on military foreign intelligence and “illegals” (case officers operating without diplomatic cover), while the CSS would focus on counterintelligence and legal residents with diplomatic cover, but there were numerous counterexamples.
The conflict and rivalry was manageable only because both organizations ultimately reported to the same parent, the Revolutionary Military Committee. There the top-levels of government ensured coordination and intelligence sharing between the two agencies.
Foreign Intelligence
In the mid-1930s, American intelligence was focused on rapidly expanding and improving its networks. Foster often battled his Soviet counterparts to enforce a principle of “revolutionary coordination,” or the sharing of assets and coordination of efforts between the American and Soviet governments. Part of these closed door, secret negotiations between intelligence chiefs was essentially Stalin pawning off assets he regarded as unreliable to the Americans, including Richard Sorge. When the CSS agreed to take over funding the Sorge ring in Japan from the GRU, Stalin remarked “good, let them pay for the dilettante’s womanizing.”
The CSS in particular was keen on recruiting new assets. Sándor Goldberger, the first head of the CSS, adopted a long-term strategy of subversion, focused on recruiting bright left-wing college students to become moles in the parlance of the trade. The archetype of this would be Kim Philby: born to privileged families, well educated, with a strong attraction to left-wing politics. Once recruited and deemed reliable, they would publicly disavow their leftism as a youthful flirtation while remaining, in secreted, committed communists. By serving as reliable functionaries in their home country’s government or intelligence service, they would rise in stature, emerging like a mole from the ground and re-establishing contact with handlers.
This would be of some use when war broke out in 1940. Some agents were reliably placed in the British and French government, and thus could keep the RMC reasonably apprised on their strategic intentions. Human intelligence sources in Germany or the Axis minors were much more scarce though. The GRU had only made a few nominal penetrations into the Abwehr. The Rote Kapelle (red orchestra) network was more concerned with preserving the KPD than serving as a reliable resistance against the Nazis. The American counterpart, often called Rote Kapelle II or RK2 to distinguish it from the Soviet network, was less well developed, without access to the personnel from the thoroughly Stalinized KPD. Instead, they had to rely upon whatever they could scrounge up from the dissident communist left or the SPD. Many had been drafted into the Wehrmacht, frustrating attempts to get useful intelligence.
Nonetheless, they scored a few early victories. During the fall of Poland, American diplomats secured the defection of a number of Polish military and intelligence officers, including several mathematicians who had worked with the Polish cipher bureau. With them, they brought working examples and a nearly complete mathematical description of the operation of the German Enigma cipher machine. A KAPD mole in the Kriegsmarine would provide a damaged specimen of the naval Enigma that had been slated for disposal, along with an operations manual. These pre-war victories provided the basis for reliable decryption of Enigma cipher traffic during the war.
Signals Intelligence
Enigma had begun life as a commercial cipher machine, best suited to protecting trade secrets from amateur snooping but not from any real professional interest. It was a relatively simple polyalphabetic cipher machine. The operator would press a key on the integrated keyboard, sending a current through three rotors in sequence. Each rotor was wired to change the character to another character. After completing this path, the current went through a “reflector rotor,” sending the current back through the rotors via a different path, directing it to one of 26 lightbulbs on the top of the machine. This would light up a character, yielding the enciphered character.
Each time a key is pressed, the first rotor advances one step. If the operator presses the same key twice in sequence, each time it will be enciphered as a different letter. When the first rotor completes a revolution (i.e., after 26 key presses), the next rotor will advance one step, and so on. To decipher encrypted text, the receiver would set his machine to the same initial settings (distributed usually through monthly codebooks detailing initial settings for each day) and key in the cipher text. This will output the original plain text message. In other words, Enigma encryption is symmetric, unlike most modern encryption like the RSA a computer uses. The same key encrypts and decrypts. Crucially, the machine will never encrypt a letter as itself, which would ultimately prove to be a serious cryptographic flaw. This commercial model had been thoroughly compromised by the Poles, and they completed several electromechanical machines which they called bombe to aid in identifying initial settings.
The models used by the Wehrmacht by 1940 were dramatically more complex. The general service model had added two more rotors with different internal wiring. Operators would select three out of the five rotors based on the initial settings in the codebook. Further, the rotors could be cycled internally by moving a pin, changing the internal wire path. Finally, a plugboard was added to the base of the machine; operators would add another cipher layer by redirecting current from one letter to another. In effect, the plug board added a 4th rotor; it wouldn’t advance but it could reconfigured with the standard 10 leads over 150 trillion different ways.
Because of the staggering number of possible initial settings, it was widely felt in the Wehrmacht that the Engima machine was unbreakable in practical terms. But through a combination of mathematical flaws in the Enigma itself, whatever its configuration, and a number of serious operational flaws in both the doctrine and practice, an eventual thorough cryptanalysis of the Engima was developed through great effort by the MID. Aided by new electromechanical bombe and later by advanced general purpose digital computers, the MID Code and Cipher Cell stationed in Matewan, West Virginia spearheaded Comintern signals intelligence.
The small mining town became a hub of advanced technology, powered by the prodigious hydroelectric power of the nearby Tennessee Valley Authority. By May of 1942, when the Franco-British government were being invited to share some cryptographic information, the facilities at Matewan had fifty working electromechanical bombes of various marks.
Alan Turing and other FBU attachés touring the Matewan facility were quite impressed with the resources being deployed. The bombe were much more sophisticated (and expensive) than anything British industry or the war government would countenance paying for, and much faster than the prototypes that had been developed at Bletchley Park. But the administration had chosen a rather brute force approach to breaking Enigma. Turing was astounded to learn that the planned order for one bombe for every rotor configuration in the new four rotor naval Enigma, or 336 machines.
In the end, a workable wartime arrangement for intelligence sharing was worked out, including the sharing of mathematical techniques in exchange for decryption time on the MID’s bombes. More advanced work, like the Athena machine, the first digital electronic programmable computer, were kept secret from their war time allies for much longer. This machine, later given the backronym Analytical Turing Heuristic Electronic Numerical Arithmetic computer, would be produced in significant quantities during the war to support cryptanalysis, the computation of artillery firing tables, and even the atomic bomb project. The last version of the ATHENA, the Mark IV, entered service in early 1946 and included the ability to store programs as well as greatly increased processing speeds. In sum, the wartime investment into computing for cryptanalysis dramatically jumpstarted the field. With much of the work being swiftly declassified by national security juries after V-J Day, the thousands of individuals who worked on the project would be able publish computational methodology and bring the innovations into civilian economic life.
In the early years of the war, the Enigma intercepts relied upon an elaborate system of covet listening posts in neutral countries, spy ships, and operational intelligence groups in the Soviet Union itself. The intercepts would be collected in forward bases in Iceland and the Canaries, where they would be sent by secure cable to North America for analysis. Intercepts would be retransmitted by their stations using their own electronic cipher machine, Arcana, to avoid tipping the Germans off to their strategic intentions. The creation of the United Nations pact greatly simplified matters, allowing Enigma (and later Lorenz) intercepts to be sent across the Atlantic directly by secure cable.
The breaking of Axis military and diplomatic codes would be a closely guarded secret. German military records establish that operational security was successful. While German intelligence was aware of the theoretical weaknesses of Enigma and the later Lorenz machine, and steadily worked to improve their mechanics as well as operational security in their use, they believed the complexity of the technical challenge to the task made compromising the system impractical. While a few officers were adamant in their suspicions, they were largely dismissed as overly paranoid by their organizations. Only with the beginning of declassification in 1947 did the surviving personnel realize how thoroughly their signals intelligence had been compromised.
Counterintelligence
The suppression of enemy agents was a similarly important task. Within the Soviet Union itself, the GUGB would serve as the Party’s inquisition, directing both military and civilian counterintelligence within the Soviet Union. “Smert Shpionam” (death to spies) became the rallying cry, it would itself be subject a purge in late 1941 to early 1942, removing many of the former NKVD head Lavrentiy Beria’s cronies. After the restructuring, the GUGB adopted organizational practices more in line with the American CSS. Its new head, Commissar-General of State Security Yuri Piatakov, had been the Soviet liaison to the American security services, and likely escaped Stalin’s Great Purge only by his remoteness.
The GUGB would maintain headquarters in each of the front commands, working closely with the army’s political commissars to maintain discipline, improve morale, prevent desertion, and counter enemy agents. Behind the lines, they were often brutal in the maintenance of wartime discipline in the factories, fields and mines, and zealously scrutinized foreign visitors. In the liberation of Axis occupied territory, the GUGB would ferret out collaborators, and harshly suppress anti-Soviet partisans.
While they were often excessive in the commission of their duties, after the Beria purge they were restrained within the limits provided for them by the law and the Soviet constitution. The much amended 1924 Soviet constitution was by no means a liberal document, but it did provide protections to the accused, and it was considered prudent to adhere to them. While being better than their Nazi opponents was not a high hurdle, the measure of mercy they were instructed to give eventually led to the collapse of German collaborator groups like the so-called Russian Liberation Army and Ukrainian nationalist groups.
The death penalty was reserved primarily for instigators, leaders and war criminals. Most collaborators and cowards were sent to penal or labor units, an arduous punishment that claimed many by exposure or pestilence during the lean times of the winter of 1942-43.
The differences in methodologies often hindered Soviet-American cooperation in the early years. But by 1942 the working relationship had been solidified, and rough agreements on tactics and acceptable levels of brutality had been reached.
Within the UASR proper, a more nuanced approach was taken to enemy agents. Within the Soviet Union, enemy agents were likely liquidated fairly quickly. CSS Section 9 took a more insidious approach to its work in North America, neutral nations, and Comintern allies.
Both the Abwehr and the SD would seek to plant agents and develop networks within the UASR and its trade networks. Some of the more delusional types believed America’s large German population was yearning to support the fatherland. Director-General for State Security Luigi Martelli worked diligently to ensure that this delusion would be fostered.
Standard policy was to identify and turn German agents in the UASR, Latin America or in neutral states like Turkey. Many of the spies trained by the Abwehr were poorly trained Eastern Europeans, quickly identified by much more experienced Section 9 counterintelligence officers. One by one, agents were identified, and turned into double agents through a variety of methods. In the UASR, they were discretely arrested and given a carrot or stick choice. Rather than face liquidation, many were eager to cooperate. Some German communists deliberately defected via the Abwehr. In Turkey, which the Comintern remained neutral to for most of the war, more sophisticated methods were required, running the gambit of normal spy strategies, from honeypots to financial leverage.
Turned agents became the conduit to compromising further agents: succeeding groups of infiltrators were often ordered to report to established agents, who the Abwehr or SD did not know had been doubled. The compromised networks were used to prevent meaningful penetration of military secrets and then later as part of large scale military deception plans.
The formation of the FBU and the outbreak of the war in the west made things exceedingly difficult for German overseas spy networks. No longer able to use British or French passports, it became more difficult to effectively transfer communiques to their handlers back in Germany. Much of the traffic would have to flow through the reactionary exile regime in Cuba, a neutral British ally which still permitted some level of freedom for German embassy staff to operate. Thus, the main hunting ground would be Florida, where German spies would have to gather to transmit intelligence to receivers in Cuba. Legendary spy hunters like Michael Easton* would perform their (eventually) famous exploits against the “Cuban Connection.”