I've designed a key layout called qwpr (available for osx and pkl/autohotkey at https://sourceforge.net/p/qwpr/wiki/Home/). It has m=2 modifiers and d=6 deadkeys, plus c=3 static "capslock/numlock" layers (that is "lock", "lock"-altgr, and "lock"-shift/"lock"-shift-altgr), for a total of s=(2^m)*(1+d) + c=31 possible outputs for a given key. (It may sound impossible to remember, but since these are organized logically, it's not as bad as it sounds). A key feature is that the punctuation symbols are available on two different keys: in their qwerty positions with shift+digit, and on the letter keys with altgr (which is available on the regular capslock key). This simply doesn't work using the xmodmap-based deadkey scheme, because after a deadkey, the "(" which is shift-9 must behave differently from the "(" which is altgr-d, but by the time it gets to xmodmap it's just "(".

So in order to come to an understanding with xkb I have to be a little bit more... um... forceful. (OK, it's really just an ugly hack, but I'd rather be playing Queen Beryl here than Gyro Gearloose.)

After reading http://pascal.tsu.ru/en/xkb/ (awesome, despite the broken English), I think I've mostly figured out a way to do this using xkb, though I haven't put it in practice yet. The idea is to use Mod1 as AltGr, but then Mod2-Mod4 as a 3 bit deadkey field, enough to handle the 6 possible deadkeys plus the default non-dead-key state. So With shift and capslock, that's 6 bits, or 64 possible "states". I'm still wading through the xkb docs I can find to try to understand "modifiers", "types", "levels", and "groups" for translating the 64 possibilities down to the 31 outputs. Hopefully I can group the outputs so that the 4 basic shift states (based on the shift and mod1 bits) are together in modular groups. But the end result will be that each key results in a symbol (like 'e') and/or an action (like 'lock mod2 and mod4').

So altgr-e, which is the acute/common/math deadkey, will lock mod4 (that is, mod2-mod4=001, first deadkey). Then ALL the non-modifier keys will, when mod2-mod4=001, reapeat the "lock mod4" action, clearing it.

So if for some reason you wanted to type the characters "é₪" here's how it would work:

mod2-mod4=000
press altgr-e (flips mod4)
mod2-mod4=001
press e (outputs é, flips mod4 as do all keys in this state)
mod2-mod4=000
press shift-altgr-h (deadkey #5, flips mod2 and mod4)
mod2-mod4=101
press d (outputs ₪, flips mod2 and mod4)
mod2-mod4=000

The only problems with this plan are:
   - if the user presses a dead key D, then key X, then before releasing X presses Y, then Z, all three of X Y and Z will output using deadkey state D (that's just how the modifier locking action works in xkb).
   - the above problem would get even spookier if in state D, key X were another dead key... that is probably best just avoided, and never happens in my layout.
   - A key like enter which was not separately programmed for each dead key state would just leave you in the same dead key state.

Does anyone have any comments on this plan? Is it doomed for some reason I don't see? Mostly, I'm just posting here to help me work it out in my head, but any comments or advice are welcome.

Thanks,
Jameson