No SRB mean Challenger disaster can't happens. Still a disaster will happen sooner or later if NASA persists trying to ram 24 shuttle flights into a single year.
Even in Saturn mode, the shuttle won't lacks lethal flaws in its design. Boom goes the SSME is an example. Crack the heatshield is another. There are hundreds of possible failure modes hidden in the shuttle.
For example: the SSME are fed from the tank, and the pipping has to go through the heatshield. When the shuttle drops the tank, it needs to fill the holes left by the tank plumbing. Thus there are folding doors that close the heatshield. If for a reason or another one of the door doesn't close, you have a certain STS-107 burn through.
The issue was that the S-IC was not reusable - it couldn't really be dunked into the sea for reuse
(yes, I know, there were studies where they blowed the top of the oxygen tank so that air got trapped inside, making the S-IC floats upside down, with the pack of F-1s safely bobbing out of the water - NASA doesn't seem to have been interested. There was also the matter of flight rates - reusing a standard S-IC made sense only at low flight rates)
The S-IC two options were either a) expendable or b) manned-winged. There was no in-between.
Late September 1971 NASA administrator James Fletcher send his budget request to the Office of Management and Budget. He proposed a Plan A and a Plan B.
Plan A was the Saturn shuttle with the manned/winged S-IC.
Plan B was a big dumb pressure-fed booster, unpiloted, unwinged, that could be recovered at sea.
OMB picked up Plan B, but later changed their mind. See my new space TL for what happened next
Presumably there would be a political impact due to the whole thing about the SRBs being manufactured in Utah, for a start.
I did a little research on this the other day. NASA twice administrator (1971 - 1977, 1986 - 1988) James C. Fletcher was a Mormon and director of the university there.
After the Challenger disaster there was a senator that accused Fletcher of favouring Morton Thiockol back in '72, a company with headquarters set in ... Utah.
The name of the Senator was Al Gore, Jr.
The inquiry led to nothing - Fletcher was cleared rapidly.