Actually, President pro tempre Lafeyette S. Foster would have become acting president in this case. The speaker did not become second in the line of succession until the 1880s.
And in any case there was no Speaker in April 1865, as the new HoR had not yet met. Colfax was not chosen until Congress assembled in December.
This raises a point I can't recall seeing in scenarios like this. Had Johnson been killed and Foster become president (actual or Acting) then should anything happen to him he would have no legal successor. So I think we may assume that in such a situation he would call at least one chamber into session, either the HoR to allow a Speaker to be chosen, or the Senate so that in the event of his death it could immediately choose another Pres Pro-Tem.
However, it is at least conceivable that he might go further than that. Can we imagine him summoning both Houses, to allow passage of a new Act extending the line of succession further - most likely to Cabinet officers - to make the situation completely secure.
One possible butterfly. If Congress is back, it might find other things to do. One obvious possibility would be an amendment repudiating the Confederate debt and barring compensation for slaves - IOW Sec 4 of the 14th Amt, but as "stand alone" article. If this happened, would there be votes enough later to pass the rest of the 14th iirc it cleared the Senate with only three votes to spare.
Alternatively, Congress might go for the full 14A a year early - but then it comes up a year early, while the South is still reeling from defeat. , and they probably won't reject it as OTL. Either way, interesting results.