Have the game in 3 parts:
1) The instrument
2) The cartridge that has the engine
3) The music/data chip
The instrument would be obvious, it is the drum, keyboard, or other item that matches the stuff on the screen. it would communicate with the game engine so the engine knows what to display on-screen. If the keyboard has fewer controls than the next version, the engine will limit the number of displays, so it can actually be played (so if you are using the 4-button guitar, and the game requires the 6-button guitar, it will only display 4 of the buttons).
The cartridge has the engine, and will adapt what it shows on screen based on what is being used. A smart developer will have okay visualizations if you are using the controllers only, but get things fancier the more instruments you buy. If the player has an avatar, the engine will use that avatar to form a rough face/outfit for the on-screen character. One idea might be to just select the 'concertgoer' option, and the game plays the music while you do something else.
The music/data chip. This is the part that is sold 2-3 times per year. This contains the songs, cues, etc that the engine uses to create the screen information. This would be plugged into the cartridge itself if using a cartridge, or you would buy a new game CD/DVD otherwise.
This way you have three parts that can be used together, and the people playing it don't have to buy everything brand new each time. You can have one person who only uses the regular controller, and they have an okay experience. The person with keyboard and drums has a better experience, and friends come over as well. The music chip means you can change out the music and play something different when you feel like it, or you change out the game CD/DVD to play something else.
The key is disconnecting the buying plastic guitars/drums from buying the game. If they can manage this then they can probably get more fans. Lower prices on a per-disk basis would be explained as they will be using the same (or nearly so) engine for the next several years, so the cost can be spread out. The changing musical data means people will be buying new discs each time, several times per year.
DLC would be a nice idea, since people can go online and purchase the music they prefer, it will arrive immediately, all of it can go in one folder (instead of hunting through multiple discs for That One Song), and you can have a record of the purchases, in case somethng happens to their system. You'd have it where people would buy the game disc once, and they would purhase the music online from that point on, to make their lives easier. The DLC would have higher profits (no need to make money on the engine), and lower material requirements (no location needed to burn, package, and ship the CD/DVD). The songs would cost more than Amazon since you are including the game cues along with th music itself, but not that much more since it can only be played in the game.
The primary source of repetitive costs would be the license and legal fees for the music, which might eat up most of the profits.