Had he done so the remnant of the ANV you speak of would be James Longstreet's Corps. If Lee resorts to his usual pattern and attacks directly the 70,000 men who drove through the Wilderness his 60,000 men might spank them.....for the other part of Hooker's army under Sedgwick to show up and destroy the ANV. The look on Longstreet's face would be epic to see......
Best case scenario, for a given definition of "best":
Jackson: We've routed Hooker! Its all God's work. Now, these subordinates of mine were trying to let God down...
Stuart: General Lee, General Lee!
Lee: Yes General Stuart?
Stuart: More Yankees coming! Under Sedgewick!
Longstreet: Dear God...
Jackson: Don't take His name in vain.
Lee: Don't worry, we'll destroy them like we destroyed Hooker.
Stuart: They have at least thirty thousand, maybe more.
Lee: Excellent. General Longstreet, move your corps to attack ASAP.
Longstreet:
Lee: Are you arguing with me General Longstreet?
Longstreet: Um...sir...
Jackson: God will take care of them.
Longstreet: I'm surrounded by idiots.
A Sudden Silence Ensues
Longstreet: Did I say that out loud?