As someone who once tried to do just this in a TL, here are my thoughts: the key to civilization is urbanization, and the key to urbanization is excess food. Germania has difficulties in this regard due to its climate. Germania is colder than Mediterranean Europe and has harsher winters, which naturally demands hardier crops. The region also has harder soil and heavy plows would be needed to till much of it; neither the Romans themselves nor Germanic peoples had such implements. The northern coasts, although pleasantly flat and sporting potentially productive soils, is also swampy in places and has been prone to flooding since ancient times, while the south is hilly and rugged. The land as a whole is also thickly forested compared to Gallia or the Mediterranean. All in all, you can't do much with German soil without more advanced technologies. These technologies are unlikely to be developed in a society that is so decentralized and unable to push itself beyond subsistence agriculture. In short, Germania is just a harder place in which to start a civilization from scratch, and in ancient times no one had the means or will to import civilization.
The Rhine Valley is the one place where you might be able to begin a more sophisticated society. Tacitus reports that the valley was home to the only large Germanic settlement, Asciburgium (which is clearly a Romanized Germanic name). Keep this city or one like it intact, add a few more, avoid disruptions, and you'll have a network of city-states that evolve into larger polities. Preventing disruptions is the hard part. Rome would assimilate any native developments along the Rhine, so the conquest of Gallia must be temporary or averted altogether. The fledgling Germanic civilization might then have to be wary of the Gallic peoples. Lastly, there's the risk that tribes from the east migrate the shit out of them before civilization can really get off the ground. It's not the most likely scenario, but unlikely things do happen and are fair game as long as they're well-explained.
As for languages, I think you have it backwards. A fragmented people will speak a fragmented language. Linguistic homogenization in OTL European states is a fairly recent phenomenon, coinciding with increasing centralization and nationalism.