I hate to repeat a common stereotype, but the Vichy French forces at the invasion zone were simply not of the same quality as Rommel's Afrika Korps. It should be noted that several Allied commanders praised the bravery and gallantry of the Vichy French troops there.
Now, if there was some sort of intelligence leak and a German armored division or two happened to be there, things might have gone differently.