I can certainly see short illustrated novels happening more than they did. As a counterpart to the manga-style illustrations, maybe the text could be accompanied by standard American comic book art. It would be in panels and strips, with text and dialog, corresponding to the story. There would probably be a mixture of those illustrating the whole story and those illustrating just selected parts.
Serialization would be more difficult, if that is a defining characteristic. American readers, and even more so American writers, cannot abide the words "to be continued" unless preceded by a cliffhanger. I don't know how long the segments are in the Japanese light novels, but if the story is 50,000 words, there would easily need to be five or ten cliffhanger moments per story. The genre would become shaped by the need to make the reader anxiously anticipate the next installment. Competition between authors in an anthology would become fierce, too. I don't see the US version breaking down stories into more than two or three parts, and each magazine would need to have just two or three stories, with some common thread between them.
Detective or police stories would work well (I'm thinking of the grittier kind, not Nancy Drew Mysteries), with two or three cases per issue. Maybe they'd be staggered so that every issue has a new case, a continuation, and a resolution. Or maybe they'd be synchronized -- that way, you could guarantee higher sales for the issues with the endings, and charge a premium for advertising space. Because there would be ads. It's the American way.