Lose the XP but not the levels. Or, really, don't lose any XP but take on an XP debt. Until you pay it back you have reduced stats. -1 on STR, DEX, CON, INT, WIS, CHA. Obviously, you also can't gain any extra XP to advance a level until the debt is paid.
That could mean that you're unable to do some things you could before, but I think is easier to RP than suddenly dropping 1-3 levels.