This is an India-1989 scenario: the outgoing government has the most seats and votes, but there is no doubt that they lost the election and the moral authority to govern, plus no one will coalesce with them. In order to remain in coalition she would've had to submit to Shas blackmail and all sorts of budget-busting measures and massive welfare outlays if she wanted to retain the support of the Orthodox parties. The choice was govern as a blackmail victim or go into opposition, and quite rightly she chose the latter. If she chose the former, then as happened in 1977 and 1999, the religious parties would plant a poison pill that she would indignantly refuse, causing a breakup of the coalition and an election which she would lose. Most likely within 2 years of the previous election.
Let's also remember that Kadima was in horrible shape to fight the election: they had just gone through a stalemated war under a lackluster PM, with both PM and President under criminal investigation that would ultimately lead to charges. Livni could not separate herself from the cloud of sleaze that hung over her party, IMO it was a miracle that she was able to salvage what she did.