He doesn't have to destroy the Army of the Potomac, he only has to force it to withdraw. This allows him to march quickly on Philadelphia, and once inside of it, he is able to fight a gruelling street battle. The Confederate troops are much better at fighting in street battle, and this again...