Disclaimer: I don't recommend using this method for dual-role modifiers, since it is very buggy. It is better to use dedicated programs like xcape for it. However, it is an implementation in just XKB.
In XKB, dead keys are implemented using a Compose file, which contains entries like
<dead_acute> <space> : "'" apostrophe
<dead_acute> <A> : "Á" Aacute
<dead_acute> <a> : "á" aacute
The part before the colon is a list of keysyms, which must be typed to activate the part after the colon. This is a string and/or a keysym. See `man Compose` for more details.
Compose sequences are also implemented this way. For example, with `<Multi_key> <minus> <greater> : "→"` an arrow → is printed when "Compose - >" is typed.
Since the list before the colon consists of arbitrary keysyms, we can abuse the format and use modifier keysyms. By trying this, the resulting string and/or keysym is only activated when the modifier is tapped on its own, not when it is used with a different key. This is exactly a dual-role modifier! However, there are some limitations:
The modifier is still active when it is released. This is no problem when a string is needed as output, but it does affect the behavior of keys like BackSpace and movement keys.
Not all applications recognize the keysym in the action. For example, movement keys do work in xterm, but don't work in Firefox. This can partially be solved by using non-printable ASCII codes for some actions (e.g. BackSpace as 0x08, Return as 0x0d and Escape as 0x1b), but those aren't recognized very well as well. (Also, the escaping should be done it in octal as \123 instead of hexadecimal as \0x123a, as the hexadecimal version doesn't seem to work.)
Despite these limitations, it is still possible to implement useful dual-role modifiers. For example, to use the left control (mapped to the CapsLock position) as escape when tapped and Extend (mapped to the space bar) as space when tapped, one could use:
<Control_L> : "\033" Escape
<ISO_Level5_Shift> : " " Space
(The Escape and Space parts aren't necessary.) The escape might not be recognized by all applications. However, VIM recognizes it, which is the most important application for this binding.
To use custom compose sequences, you should add your custom entries to ~/.XCompose. This file should start with
include "%L"
to include the default sequences. Otherwise, the dead keys and the compose key stop working. Furthermore, changes in this file become active in an application when the application is restarted. Also, GTK apps use a hard coded Compose file by default. To disable this, you should add
export GTK_IM_MODULE="xim"
to ~/.profile. This will disable the Control+Shift+u method to input Unicode characters.
Create advanced keyboard layouts in various formats using my Keyboard Layout Files Creator!