When a contestant fails to replicate the judge's phrase, I'd suggest that the judge needs to play the phrase again: if the judge can play it again, the contestant is out and play proceeds as outlined in the instructions. If the judge can't play it again, though, play continues as though the contestant had properly replicated the phrase (or, in a stricter version, the judge him/herself is out). This would prevent the judge from playing something unreasonably difficult.
You could also play this in a circle: player 1 plays a phrase and player 2 plays it back, then player 2 plays a phrase and player 3 has to play it back, and so on. A player who can't replicate a phrase is out, and the last player standing is the winner. To make the game last a bit longer, a player who can't replicate a phrase might get a strike against them, and a player who accumulates a set number of strikes is out.