Useful pages about game mechanics on Wikipedia and at Board Game Geek show solutions for board game rules design. Even without sound some of these are not suitable for music games.
Big people
Most important reason why music games cannot benefit from every development of board game design is scale. Comparing to board games music games have much different spatial organization and they usually have to consider the presence of instruments of different sizes or sound equipment. This will be our main criterion of usefulness in this article. Designers always have to take practicalities into consideration.
Of course you can work the size around with establishing Conductors that will play a regular-sized board game, and musicians interpreting board situation from distance. Another way of not troubling musicians with manipulating the small pieces is enlarging any component (and maybe let participants serve function of pawns). Both these ways might let you end up with great music games, nevertheless here we will consider possible games that allow for direct, peer interactions between all participants and operate in medium scale (which makes them not too demanding for domestic production).
Mechanics overview
Now let's review aforementioned lists of game mechanics. Notice first that very popular modern mechanics of tile-laying, worker placement and resource management are usually implemented with small pieces. It's not a problem with obtaining these components. Some of them (especially tiles) are made for specific game title, but often stock game pieces are used and these are available for purchase. But usually the preferable amount of this type of components bound these mechanics to table top, and precision with pile sorting will be difficult while playing an instrument. (Mentioned mechanics sometimes work well with small numbers of items but you have to arrange for it).
You might pay more attention to other classics like capturing or action points which most often need less tokens and might be followed in memory. Typically "spatial" mechanics of capturing and movement might not require space but may be easily projected to tones or even rhythmic patterns (such an abstract space may also mediate for fighting). Mechanics on Wikpedia and BGG lists that are based on traditional components include: rule cards, dice or time-tracking devices - you can get or make these in different sizes for musicians' convenience.
Some standard mechanics on lists are very general, they might give you inspiration in any possible type of game, and you might use them in many different ways. The most popular are: game modes (perfect for adjusting difficulty), rock-paper-scissors (in a general meaning of three-side strength balance), risk-reward (viewed by Jesse Shell in Art of Game Design as a universal all-games mechanic). If you use explicit victory conditions of any kind (to be considered in competitive games) you can always use a catch-up (possible to realize in one of many different ways).
Scale-independent mechanics like voting or bidding cause musical downtime so they tend to be useful for composition games category rather that improvisation games, although you might make it work in both. Interestingly at BGG there is also singing mechanics, unfortunately this category of games at Board Game Geek doesn't cover games that let you create music. Rather, games with that mechanic reward you for suggestive humming of some hit song, assumed to be well known by all players.
Arguably greatest potential lies in turns mechanic, as it is seemingly already present in so much music (that is not supposed to be inspired by games). Best example here would probably be (regular) jazz music where musicians "take turns" for soloing and trading.
In our library
Check out Sound Collage Game Board for a take on turn-based game where board serves as a partiture.
