Well you would need Thomas Jefferson to have less sway and influence on him.
Also, an interesting butterfly could be the Olive Branch Proposal that Adams proposed Jefferson-giving Jefferson as vice president equal powers to himself. Jefferson was thinking about accepting it, but having Madison constantly in his ear telling him to turn it down made him refuse it.
Jefferson didn't want to accept Washington's offer to be Secretary of State and was inclined to return to his post in Paris as Minister to France. It took some persuading on Washington's part to get him to join the administration. Just have Jefferson stick to his guns and return to France, and then he won't be around to pull Madison away from the orbit of Hamilton.