• You are not logged in.
  • Index
  • General
  • Updating the standard Colemak layouts to support more characters

    Updating the standard Colemak layouts to support more characters

    • Started by DanJacobs
    • 9 Replies:
    • Reputation: 3
    • Registered: 24-Mar-2008
    • Posts: 32

    Hello everyone,

    One of my favourite features of Colemak is everything available under AltGr – all the extra accents, dead keys, and punctuation is extremely useful.

    But since Colemak was created, other layouts have surpassed it in terms of available characters. For example, there are layouts like those in France's new national standard (NF Z71-300) and Germany's national standard (DIN 2137-1:2018-12).

    These layouts provide dead keys for much more accents/diacritics, including: dot above, double acute, double grave, covering bar, oblique bar, reversed brève, dot below, macron below, and comma below.

    They also support a host of additional symbols, the new capital eszett character, and the entire Greek alphabet. In the German layout, combining versions of all the accents are available (in addition to the conventional pre-composed variants). Germany's national standard also supports some very obscure features, such as the entire set of IPA phonetic transcription characters which require chained dead keys to activate.

    It would be lovely if Colemak also supported all these additional characters.

    It would be relatively easy to accommodate these extra characters, without moving any existing characters – there are plenty of unused spaces on the Group 2 Level 2 position of each key (i.e. holding Shift + AltGr).

    I would be quite happy to propose an update to the layout to bring its characters support up to the standard of these new national standards (without breaking any existing mappings). I could also provide an implementation for Windows (KLC file).

    @Shai, if I were to do this, would you consider making the resulting layout the new official standard for Colemak?

    Let me know if anyone's already attempted such a thing. :)

    Offline
    • 0
    • Reputation: 214
    • From: Viken, Norway
    • Registered: 13-Dec-2006
    • Posts: 5,367

    I have sort of taken up the torch when it comes to AltGr mappings. My AltGr mappings are named Colemak[eD] and contain both symbols and further dead keys. I don't think it falls short of other layouts, when it comes to symbols that people actually need. I've never needed a reversed breve. If I did, I'd use the Compose method for it, as spending a layout spot on a dedicated dead key for such a rarely needed accent would be a massive waste.

    A more complete IPA implementation would be lovely, and I have some ideas for it but I haven't finished such a project. Again, it's possible to use Compose instead.

    Read more about it in my BigBag, see my signature or follow this link: https://dreymar.colemak.org

    If you have any ideas for improving on my work in a consistent and clear fashion, please let me know!

    Cmk-ISO-eD-CurlAWide_96d-FShui.png

    Last edited by DreymaR (16-Jun-2023 12:29:27)

    *** Learn Colemak in 2–5 steps with Tarmak! ***
    *** Check out my Big Bag of Keyboard Tricks for Win/Linux/TMK... ***

    Offline
    • 1
    • Reputation: 3
    • Registered: 24-Mar-2008
    • Posts: 32

    Interesting. I hadn't realised there was a Colemak.org.

    Those AltGr mappings are very different to those which are available on the standard Colemak layout. I guess I had been hoping to provide an additive update that didn't change that which people were used to.

    And I must say, I rather like the current Colemak AltGr mappings. Most notably, they place a lot of dead keys within easy reach, as part of the ‘main’ section of the keyboard. It's great to have grave, acute, and diaresis right there in the home row. My inclination was to keep all this, and just flesh out the layout with more stuff.

    As for IPA, there's a standard for that – ISO/IEC 9995-9:2016 “Annex C - Table for special mode IPA”. This standard is used by Germany's standard DIN 2137-1, which is documented for free rather nicely here: https://de.wikipedia.org/wiki/E1_(Tastaturbelegung)

    I've implemented the matrix of IPA characters using chained dead keys in a KLC file, and I can confirm it works. (For Windows, at least.)

    In Germany's implementation, the initial dead key you press is the combined “double acute and phonetic transcription” key. It allows you to put a double acute accent on both OoUu; other keys start a series of keystrokes for the IPA table. It's very obscure, and I'm not sure anyone will use it. But... I thought I'd give a go of implementing it to see if it works. And it does work.

    Offline
    • 0
    • Reputation: 214
    • From: Viken, Norway
    • Registered: 13-Dec-2006
    • Posts: 5,367

    If you want to add to Shai's old AltGr mappings, feel free but I think they're pretty hopeless. Layers like that one and EURKey try to cram lots of accented letters into the AltGr layer without thinking about it, and end up lacking space for all and putting some in unintuitive positions. This is not the right answer. Also, Shai has said that he doesn't want to work more on the AltGr layers. They clearly bear every mark of being half finished, with all those tilde mappings. But suit yourself.

    What people are actually used to is open to some debate. Many users are simply used to not caring about special letters at all. Many use Colemak[eD], which has been around for many years now. Then some are like you I guess, and have gotten used to Shai's original mappings – haphazard though they may be. So I don't think there is a very clear answer to what people are used to, really!

    The reason I didn't go for the tack of having accent dead keys in the letter block, is that it didn't feel intuitive to me, and I felt it could lead to accidental dead key triggering which would be quite disruptive to typing flow. As it is, the diaresis on the `;:` key and the acute on the `'"` key are both intuitive and very easily reachable in my opinion, and the ones on the top row are mostly rare one; these get far(!) better placements in my locale variants for languages that need a specific subset of them. A few are a bit bothersome in my solution, granted: Mostly the grave and circumflex accents, which are needed fairly often and not placed so well in my setup. For those I sometimes use special tricks like having a special layer mapping for their dead key in addition to the top-row one. These days I've got a Compose method for Windows going, so I can use sequences to get stuff. It may mean a few more key presses, but it may also be a lot more intuitive.

    The E1 setup is very interesting! Have you shared your KLC file(s) for it somewhere?

    As I said, I've been toying with ideas for IPA implementation. The challenge is the number of similar forms of some letters. I feel it'd take quite a number of new dead keys to do it well in an intuitive fashion. I haven't got the time and inspiration to do it all on my own, so I guess I'm waiting for someone to become interested enough.

    Last edited by DreymaR (18-Jun-2023 15:12:36)

    *** Learn Colemak in 2–5 steps with Tarmak! ***
    *** Check out my Big Bag of Keyboard Tricks for Win/Linux/TMK... ***

    Offline
    • 0
    • Reputation: 3
    • Registered: 24-Mar-2008
    • Posts: 32

    Colemak's AltGr layer never struck me as feeling half-finished at all; for its time, it was the most thorough layout I'd ever seen. I've never been a great fan of the intrusive tildes though; I would propose getting rid of those in any 'official' update to the layout. Or replacing it with a whole new AltGr layer. That could work too. ;)

    I wasn't aware that Shai didn't want to work on fixing up AltGr. That's a shame.

    But here's the thing. If one was to want to propose to include Colemak in a commercial operating system, one would want to implement the closest thing to a standard. One would want a standard layout definition to be able to point to in a code review, to justify the choices made.

    And currently, the thing you can download at colemak.com seems to be the standard. But given that its AltGr layer is a little unmaintained, it'd be good for the standard to be updated prior to consideration for potential inclusion in a commercial operating system. So if it is eventually included, it would at least be the best version of the layout.

    I'm pretty open minded about which AltGr layout should be the standard. If Shai was to want to update the Colemak.com site to designate the COLEMAK[eD] version as the new standard, I'd be perfectly fine with that. Assuming it retained the very liberal (public domain declaration) licensing policy, one could then use that as the basis for a proposal for inclusion in a commercial operating system.

    I haven't yet shared my KLC file for the German E1 layout. This was implemented for work, so I do not own that implementation. I expect it will soon be publicly available.

    What do you think of the IPA implementation in that layout?

    Offline
    • 0
    • Reputation: 214
    • From: Viken, Norway
    • Registered: 13-Dec-2006
    • Posts: 5,367

    Interesting that you should find Shai's AltGr layers "the most thorough layout" you'd ever seen, when in fact it's anything but. Shai said back in 2007 or so that he'd thrown it together for access to multilingual glyphs but without much further analysis and thought as to what's best. It should work decently for Spanish but less so for other accented languages. He also said that he wasn't interested in working more on it, nor in filling out the ugly "tilde" blanks. In the 15 or so years since he said that he's held true to those words.

    There are other layer designs like Bépo that in the meantime have been truly thoroughly thought out. And I dare say I've put quite a lot of thought into Colemak[eD]'s design too. Several of these projects have been actively maintained for many years.

    Therefore, you cannot say about the standard Colemak that "its AltGr layer is a little unmaintained". You simply can not. It's been completely unmaintained for a decade and a half. As for Shai opting to adopt Colemak[eD] as a new AltGr standard for Colemak, that is also very unlikely to happen I believe. Another thing he's said in 2006 is that he won't change Colemak. So unless he considers these layers somehow "not Colemak" after all, they will not change.

    This is one point where I disagree with Shai's policy. I find that bundling the Colemak layout itself (the letter assignments) with AltGr layers and a Caps-To-Back remapping works great for simplicity but sucks for advanced users who want modularity and freedom of choice. I don't want that silly and crude Caps-To-Back trick! I want the far more refined and powerful Extend layers for that key. Others want other solutions. Shai's Linux implementation even breaks Linux design principles by forcing that trick on the Colemak user so it overrides the system option settings for their CapsLock key.

    I don't have the time and energy for a full run-down, but I'll briefly note some things I consider design flaws in the original Colemak AltGr layers:
    • The grave dead key is on AltGr+r, and ogonek on AltGr+g, for example. That's easily reachable, but completely unintuitive.
    • Other dead keys are organized by the first letter in their name, but that's not how I think of their accents. It's better to go by shape.
    • Forms of A and O are given some priority, but of course there's only room for one of each so Æ Å Ä Ã and Ö Õ are scattered around the A and O keys.
    • Doing this, letters end up in unintuitive places like Ø (an O/E form) on P, Ä/Æ (A/E forms) on Q/Z, etc.
    • Having exhausted those keys, other important accented forms of at least E I U have nowhere to go.

    KB_US-Colemak_with_AltGr.svg

    Sure, if you're, say, Portuguese you'll learn where ÃÕ are, and ÇÉÍÓÚ are in intuitive positions. But for À ÂÊÔ it's a mess.

    Okay, that may sound too harsh. It's well thought-out, of course, as is anything Shai has made I believe. But I still think it's less than it should be.

    Last edited by DreymaR (19-Jun-2023 09:08:50)

    *** Learn Colemak in 2–5 steps with Tarmak! ***
    *** Check out my Big Bag of Keyboard Tricks for Win/Linux/TMK... ***

    Offline
    • 0
    • Reputation: 3
    • Registered: 24-Mar-2008
    • Posts: 32

    When I say it looked thorough, in 2007 (a *long* time ago), I was comparing it to layouts included in Windows which were (and still are) generally pretty sparse. So by comparison, this was a luxurious amount of glyphs! I'm not claiming to have been much of an expert at the time.

    But ok, I get it. We could do much better. (Which is why I started this thread.) And I do agree that Bépo is very nice and well thought out.

    As for the bundling thing, I do see what you're saying. I think that's especially true with the Caps Lock key, which feels to me like something that should be managed at the OS level as part of configuring modifier key behaviour generally, separate from keyboard layout. And it often is, e.g. Mac OS allows Caps/Ctrl/Command/Option to be configured separately from the keyboard layout.

    However, the way all operating systems handle AltGr is that these are baked into the keyboard layout.

    We may *like* the idea that these things are formally defined separately.

    But...

    If I were to propose to include Colemak in a large commercial operating system, something many users have been clamouring for and creating petitions for, that layout's going to have to include *something* in the AltGr layer.

    So the question is... what version of an AltGr layer should it be? We've got options like:
    - Shai's 2007 mappings, the de facto standard but completely unmaintained
    - Colemak[eD]
    - Entirely empty (à la the standard US keyboard and Dvorak)
    - Something else entirely? Maybe something based on the ISO/IEC 9995 Common Secondary Group? https://en.wikipedia.org/wiki/ISO/IEC_9995

    I'm open to anyone's thoughts.


    Mac OS is an interesting comparison here. They've been shipping Colemak for a while, but they ship it without any Caps Lock trickery, and with their own standard-for-their-OS set of AltGr-equivalent mappings. The thing is, a certain other large commercial operating system doesn't have an established standard of AltGr mappings. And I'm loath to omit AltGr mappings entirely, since they're so useful.

    Offline
    • 0
    • Reputation: 214
    • From: Viken, Norway
    • Registered: 13-Dec-2006
    • Posts: 5,367

    You make very good points. I haven't got an answer, beyond my claim that there is no de facto Colemak AltGr standard today. My take on the modularity issue has been to consider Colemak per se the letter block assignments (including the semicolon key which obviously has to move), and my impression is that this is in accordance with the sentiments of nearly all of the Alt Keyboard Layout community. That is the Colemak layout as such, not the various implementations and their add-ons as I consider them.

    And yet, as you say, on most platforms a layout implementation will come with all layers defined. The US standard is a wasteland of missed opportunity, and entirely unsatisfactory for those of us who wish to use accents and typographic characters etc.

    I could suggest my own Colemak[eD] as a standard of course, but that'd be a fairly obvious plug and I won't. I'm not sure I'd even want to be the sole person in charge of such a responsibility. I have way more than enough work on my hands as it is with my BigBag, as you may well imagine. I'm always interested in these matters, but I already use too much time on them.

    I haven't had the time to look at the E1 properly yet. As I said, it looks interesting! I see a few things I'd have done differently, of course, but also things I agree with. I take it it's a kind of standard in Germany then? Anywhere else?

    In my works, I've looked to different semi-standard implementations such as Linux, MacOS, US International (albeit less than Shai did) and Bépo. But I've also made my own decisions. One was to abandon the double quote with AltGr for diaeresis: This common accent deserves a lot better than a shifted mapping! So I ended up with that on the easily accessible semicolon key, unshifted. It seems that some of the traditional layers have not given ergonomics the deserved attention.

    One challenge for making a standard for AltGr layers, is that of positional vs mnemonic mappings. To me it's obvious that ß goes on AltGr+s for mnemonic reasons, and you'll find quite a lot other such ones in my layers. However, typographic quotes are positionally mapped in Colemak[eD]: ‚‘’ on QWF and ‹› on UY. This is easy to remember, but layout specific. A standard, ideally, should work for more layouts than just Colemak I feel. But that raises some challenges, due to this issue. As you may see in my EPKL program's alt layout implementations, I've moved some keys around for some layouts to alleviate this but there's really no one-size-fits-all solution. Abandoning positional mappings altogether for the sake of unity seems an overly harsh answer.

    Last edited by DreymaR (19-Jun-2023 10:43:11)

    *** Learn Colemak in 2–5 steps with Tarmak! ***
    *** Check out my Big Bag of Keyboard Tricks for Win/Linux/TMK... ***

    Offline
    • 0
    • Reputation: 3
    • Registered: 24-Mar-2008
    • Posts: 32

    E1 is, to my knowledge, not yet actually used anywhere in practice. It's a published standard in Germany, since 2018. Apple adopted a very cut-down version of this standard and now ship it as a non-default option in Mac OS under the name "German Standard". (It's not standards-compliant though; most of the mappings required in the standard are absent, including the entire Group 2.)

    Linux distros seem to ship a version of it too. In Ubuntu at least, it's more complete than Mac OS's version, but it's still not actually complete.

    It's not included in Windows at this time. (Watch this space...)

    Complicating matters is that a newer version of the standard was published in 2020, but this version was never ratified; it remains a draft. I don't know what's happened to it and why it's been stuck on draft status for three years. The standard document does not list authors or contact details; I've tried using the contact form on the DIN store website, but to no avail.

    The 2020 draft isn't just additive; it moves things around. So it makes a choice about which version to ship more complex. I'm leaning towards using the ratified 2018 version, in alignment with Mac OS. But unlike Mac OS, it would be fully standards compliant and include all the characters and dead keys.

    The good news about E1 is that it is a superset of the existing German keyboard layout. So it'll be transparent to users if it replaces the old German layout as the default.

    By contrast the new French standard, which also includes a wonderful collection of symbols and dead keys, is dramatically different from the old conventional French layout. But like Bépo, it is now included in Windows: http://kbdlayout.info/kbdfrna

    Last edited by DanJacobs (19-Jun-2023 13:01:24)
    Offline
    • 1
    • Reputation: 3
    • Registered: 24-Mar-2008
    • Posts: 32

    DreymaR, if you're curious, you can now try Germany's new E1 and E2 layouts for yourself, complete with their unprecedentedly large set of dead keys and other things mapped under AltGr. This includes the IPA phonetic transcription features we'd discussed previously.

    These layouts are included in Windows Insider Build 25977, and are described in this blog post: https://blogs.windows.com/windows-insid … y-channel/

    They're implemented based on the latest version of the standard.

    Offline
    • 1
      • Index
      • General
      • Updating the standard Colemak layouts to support more characters