Even if Gaddafi had complete blueprints for a fission weapon, and details of the processes needed to produce the fissile materiel and so forth, you need to build the enrichment infrastructure, and then take the uranium ore through the enrichment process. This requires a significant technically qualified staff to do all of this, not just the scientists but the welders with skills to put together all the complex machinery of the bomb factory and many other examples of "blue collar" but highly skilled folks. Additionally Libya will have to import essentially all components or materials - maraging steel, all of the electronics/switches and so forth as they have no local manufacturing capability for any of this. In 1990 the population of Libya was roughly 4.3 million, of which slightly over 50% were ages 15-64 (workers of any sort). In Libya, women with university education in the sciences or skilled blue collar abilities/jobs were essentially between few and none. This gives you a population of roughly 1.2 million males between 15-64 to draw from. When you look at the reality that there were not enough Libyans to provide skilled labor at all levels for the oil industry and there was significant reliance on foreign labor...
The peak employment in the Manhattan Project was ~125,000. Assuming only 25-30% of that number would be needed to construct and run a Libyan program given the designs already present and no duplication that means roughly 40,000 so employed (and you need for more skilled workers due to people quitting, getting sick/dying, retiring, etc). Maybe just maybe if you shut down everything else that required folks with these skills in Libya you could find this many...maybe. Bringing in this number of foreigners with these skills would be the same as daily broadcasts saying "we're building bombs, we're building bombs". A reaction from one of several quarters will then occur.