Vespasian didn't want to do it, and was pissed when some idiot jumped the gun and lit the place on fire anyway. This does not stop future emperors like Hadrian from destroying the temple on purpose to stop Jewish revolts.
Of course resolving the problems with Judea's constant rebellion have both religious and political roots, axing the corrupt governors would have been a good start...