I've been working on my qwpr layout ( https://sourceforge.net/p/qwpr/wiki/Home/ ). This is a layout that pushes the boundaries in almost all respects: modifier remapping, backspace and tab remapping, modified arrow key outputs for inputs with different modifiers, custom dead keys, complex enough to need a cheat sheet for learning. Getting this to work multiplatform is thus not easy. I'm not finished, but I thought it would be worth sharing what I've learned so far. I'll try to keep this post up-to-date as I learn more.
Basic layout editing and file formats:
OSX: Ukelele rocks. It creates .keylayout files which are in a very clear and highly flexible xml format. These can be dropped into /Library/Keyboard layouts/ (as long as you have admin privileges) and you're good to go.
Windows: The only free option is Microsoft Keyboard Layout Creator. There are two paid options: http://www.kbdedit.com/ and https://www.klm32.com/index.html . Both seem to have various crippled demo versions. As far as I can see, klm "2000 edition" looks like the best, but I haven't tried it out. None of the file formats are properly documented, but the KLC format at least is not hard to figure out. But the problem is that it is far more limited in features than the compiled .dll format that windows actually uses.
Windows: There's also Portable Keyboard Layout. This is a compiled autohotkey script which stores layouts in an .ini format which is quite easy to understand. It's open-source, too, which means it should not be too hard to make a version which fixes some of the problems I mention below, if you take the time to learn the proprietary AHK scripting language. However, it seems that AutoHotKey is not 100% reliable; for instance, the support for alt-tab is finicky at best.
Linux: keyboards can be either xkb format or xmodmap format. XKB is more "native" in that most distros have graphical control panels and widgets for picking your format. However, it will always take admin privileges to install keyboards in this format. This format is an ancient beast, documented here: http://pascal.tsu.ru/en/xkb/ . Xmodmap is more "portable" in that a user can set it for their own sessions without needing admin privileges.
Moving "special keys" (modifiers, tab/delete/enter/arrows, etc.)
OSX: Ukelele can occasionally get you there, but if you do anything but the simplest remappings, you want KeyRemap4Macbook. It uses an separate xml format which is pretty clear, and can remap any key/modifier combination to any other. These remappings work in non-Cocoa apps, which is usually good, but annoying if you have another OS in Virtualbox because fixes can in some cases be applied twice.
Windows PKL: Works pretty well, but some configurations just won't work for no explicable reason. If you try again and add enough redundant curly brackets in the ini file, you can usually get them to work. Alt-tab is a problem. (insert further explanation here).
Windows KLC: From posts I've read on the internet (not experience), this works in some cases if you modify the .klc file by hand. Ie, ugly and hackish, but you can make it work.
Windows proprietary products: Supposedly works, but this is one of the "premium" features that costs most.
Linux xkb: Works.
Linux xmodmap: works.
Custom dead keys
OSX: with Ukelele, it Just Works. Anything you've remapped with KeyRemap4Macbook will stay that way regardless of the dead key; but at least in my case that's what I wanted to have happen anyway.
Windows/PKL: Dead keys only map one output to another. Thus if you have different key combos with the same base output, they cannot have different outputs when remapped by dead keys. This should be relatively easy to fix by small changes in the PKL source code.
Windows/KLC: Same problem as PKL, but probably impossible to fix.
Windows/Proprietary: I think these have the same problem, although this limitation is in no way built into the dll format which they output, so I wouldn't be surprised if there were a hack or workaround to make this work.
Linux/xmodmap: Use a ~/.XCompose file, as documented below in the ligatures section. This would naively seem to have the same problem as windows, but you can work around it by mapping shift-X to Y and not-shift-X to Z. For instance, if both shift-9 and altGr-d output "(", and you want them to have different outputs after a dead key, you'd do: <custom_dead_acute> ~Shift <parenleft> : "∆" \n <custom_dead_acute> Shift <parenleft> : "⁹" #<custom_dead_acute> must be replaced by whatever keysym is on the AltGr-level of e." (Thanks to Andreas Wettstei from the Neo forum for this tip).
Linux/xkb: This would involve some truly EEEVVVIILLL twisting of the xkb format; probably possible, but not for the weak of stomach.
Cheat sheet
Windows/pkl: this can be configured to show a picture of the keyboard. I have not figured out how to make it show only when a modifier or dead key is in effect, but I'm sure that would be possible, if nothing else with minor changes to PKL source code. Pkl source includes some perl and batch utilities to make html and then png files of your keyboard, but you have to install Perl and hand-install a Perl module to get the HTML, and get the correct versions of ImageMagick and Firefox to get the PNG. I got as far as the HTML and then hand-modified that before I made the PNGs; this is tolerable.
OSX: "Show Keyboard Viewer", done. Unfortunately, you just can't make this auto-show when you use dead keys and/or certain modifiers, as it has a bug: it will always initially show up with the unmodified keymap, and only react to modifiers and dead keys that occur after it appears. Grr.
On linux, obviously this would be possible, but it would probably be a fair amount of work.
"Ligatures" (multiple outputs from one keystroke)
OSX: just works
PKL: Just works, except ligatures cannot be modified by dead keys.
KLC: Annoying stupid format which allows only 4 outputs per ligature. Also, same restriction as PKL.
xkb: I don't know how you'd do this without seriously torturing the format. I think you're supposed to use xmodmap.
xmodmap: See other_linux below.
other_linux: Use an xcompose file. If you use a dead key to modify a ligature, then you can't also use that dead key to modify an initial substring of that ligature. Not the end of the world. Further documentation https://help.ubuntu.com/community/ComposeKey
Converting formats
As far as I can tell, ALL of the following are quickie conversion scripts tailored for a specific conversion. As such, they make good starting points, but you can expect glitches and should be able to tweak the output and/or source code.
klc → pkl: perl tool included with pkl source.
keylayout → klc (python): https://github.com/adobe-type-tools/keyboard-layouts
keylayout → pkl (python): my modification of above https://sourceforge.net/p/qwpr/code/ci/ … converter/
xkb → various (python): https://github.com/mnapoli/bepo/
txt → xmodmap (python): https://github.com/guileen/keyboard-hacking
Again, I want to keep the above up-to-date so if you reply with useful suggestions or links I'll try to edit them into the text above.