Peel cared because it was about power.
The Ladies of the Bedchamber were appointed by the Monarch but under preceding rulers since the Hannoverians came to the throne a precedent was set more and more that when the Government changed the Monarch would signal his or her support for the incoming government by shuffling their Royal appointees accordingly, picking from a list of suggestions made by the Prime Minister. Under William IV, Victoria's immediate predecessor, this had become almost rubber-stamped by the monarch, largely because William IV didn't really care about the issue.
You've got to remember that although Parliament is ascendant by this time, it still officially governs at the invitation of the monarch. In 1839 Melbourne resigned after his majority eroded and advised the Queen to call on Peel and the Tories, as the procedure went. Peel wanted Victoria to shuffle her Ladies to show that she supported his incoming government - otherwise you'd have had the Monarch surrounded by Liberal Ladies whilst the Government went Tory.
Part of it was symbolism - Peel wanted to be sure that the Queen would publicly signal her support for, and thus the validity of, his government. If not, you could imagine a situation where MPs on the fringe of a potential vote, or the wider public, might begin to wonder whether the Queen disagreed with her Government and whether they should too.
Part of it was about how political power worked in the period. While you are right that the Ladies held no direct power, that isn't how politics works then or now. They were still influential, and most were the wives/sisters/etc of prominent Liberal politicians. You can't separate the informal politics of parties and social events from the formal politics of Westminster in this period. Alliances could, and were, made and broken in drawing rooms and ball rooms in the c19th and the female relatives of prominent politicians were often key players behind the scenes.
And part of it was about gender. Queen Victoria was young at this time, and untested, but also a woman in a period where it was felt women shouldn't hold political power directly. Of course she was the Queen, but part of Peel's request was very much about testing the Queen to see if she intended to be a mere figurehead or more involved in politics. Remember that whilst William IV had let parliament shuffle his Gentlemen of the Bedchamber about, George IV hadn't. In fact in a very similar situation in 1812 he brought down a prospective Government by refusing exactly as Queen Victoria did here. While respecting her as Queen Peel was, to some extent, trying to put her in her political place as a woman nonetheless.