Apparently, Mr. Babbage actually married without permission and lost access to the Babbage fortune. Had he had his father's permission or not married, he would have had access, and may have had enough money to finish the difference engine.