That is a cool idea! And it seems entirely doable in XKB.

If you try it out, let us know how smooth it feels in actual use!

Do you want Lv3+Space and Lv3+Shift+Space to be Return? Maybe you'll have to declare that by explicit actions; sometimes that's necessary. Otherwise, you have two wrongly placed key mappings on lv3-4.

Let's see: You've mapped level2 of Space to Mod5. That _should_ make AltGr(=Shift)+Space turn that mod on. Mapping the mod state explicitly like that is what I'm used to. However, for completeness and robustness I'd also map the key name (in this case, ISO_Level3_Shift) and not only the action alone.

You can clear mods with the clearmods argument; I've used that extensively in my Extend mappings. I hope it works with the SetMods() action...?

Try this:

   key <SPCE> { symbols[Group1]=[ NoSymbol , ISO_Level3_Shift , NoSymbol , NoSymbol ,
                   Return               , Return                , // Enter
                   space                , nobreakspace         ], // Space
                 actions[Group1]=[NoAction(),SetMods(mods=Mod5,clearmods=Shift),NoAction(),SetMods(mods=Shift),
                   Redirect(key=<RTRN>) , Redirect(key=<RTRN>)  , // <RTRN>
                   NoAction()           , NoAction()           ]
               };  // <SPCE>

That should make AltGr(=Shift)+Space into Mod5/Lv3_shift. But have you considered how you wanted to get Mod5+Shift to activate lv4? Unless you have a good answer to that, maybe your example #1 is the best one after all? In my AltGr mappings at least, there are many letters with a capital version such as Ʒʒ, Þþ, Ðð, Łł, Ʃʃ, Ħħ, Ææ, Øø...

There is a "real" lv3 modifier key in level3/modifier_mapping. Make sure this is activated somewhere:

  include "level3(modifier_mapping)"

then try redirecting?

   key <SPCE> { symbols[Group1]=[ NoSymbol , ISO_Level3_Shift , NoSymbol , NoSymbol ,
                   Return               , Return                , // Enter
                   space                , nobreakspace         ], // Space
                 actions[Group1]=[NoAction(),Redirect(key=<LVL3>,clearmods=Shift),NoAction(),SetMods(mods=Shift),
                   Redirect(key=<RTRN>) , Redirect(key=<RTRN>)  , // <RTRN>
                   NoAction()           , NoAction()           ]
               };  // <SPCE>

A thought: If you aren't remapping your Shift keys, then Shift+AltGr could be set to provide layer4 (as before).
AltGr → Shift (lv2)
Shift(old or AltGr)+Space → AltGr (lv3)
Shift+AltGr key → Shift+AltGr (lv4)

Try this? It sometimes helps to be very explicit.

Redirect(key=<LVL3>,mods=Mod5,clearmods=Shift)

Try it with the physical Shift key and not your remapped AltGr key first, to isolate the problems.

You're sure the include line I mentioned was active in your layout?

And: Are you editing the entry in the extend file? If you've added it elsewhere, there may be competing definitions at hand. Fortunately, I think extend is added last (since it's an option) so it should win. But to make sure, do try to define another level of space as something exotic for testing purposes. (Don't leave yourself stranded without a space though!)

Not sure how to help you without trying it out myself then (and I don't have time right now). Hope it works out! It's not the simplest problem I've come across but since I made Extend actually work I've come so far as to even impress the XKB gurus at times. ;-) So maybe we can pull it off.

In the name of testing, could you try remapping the Shift state of Space to another key? With redirect and clearmods, it should give the unshifted key then. (In my Extend layers, I've mostly used clearmods=all, but I don't see why clearmods=Shift shouldn't work!) If that works, try adding a mods=Mod5 to see if you get the AltGr mapping for that key.

When testing stuff, you always need to go from something simple that works to what you want to achieve, via as simple steps as possible.

Lastly, remapping the shift state of the Space key to ISO_Level3_Shift should be enough in itself; did you try that without any actions? If that works, maybe there's room for an action that kills the shift state then?

For max safety, the symbols section should contain this I think (although I believe it is usually included elsewhere anyway):

modifier_map Mod5 { ISO_Level3_Shift };

Note: According to the fabled Ivan Pascal, the only action that allows both setting and clearing of mods is indeed RedirectKey. There must be a way we could do this with that action...

Thanks for that clarification, Aldo!

Maybe using lv4 for AltGr+Space and keeping a normal AltGr key somewhere else is the best solution then. In that case, it seems too messy for me! Having to switch levels 3 and 4 is a no-go in my case.

Interestingly, with TMK (converter/controller firmware) I think it shouldn't be too hard to do... :-)

You could just shuffle everything around in a simple-yet-brilliant way, like you usually do. For instance:

RAlt   → RShift
RShift → RAlt(?)
LAlt   → Extend
Caps   → LAlt

Sounds nice to me. I'd keep LShift around since it's commonly used. Maybe you don't need your physical RShift? If lost, it could be included in the Extend layer instead, for the situations where it's actually needed.

For now, this seems like a simpler and more robust option than squeezing stuff into the Space key. Had you not opted to make RAlt your Shift key, it'd been much simpler.

Actually... here's a new idea: Instead of messing with the Shift state, how about making Extend+RAlt into AltGr? If you don't remove your LShift or RShift, that'd make lv4 possible. It's nice enough to press both Alt keys (left one first) for AltGr, isn't it?

If that works, it'd be so elegant that I'd consider it myself. For implementation at least, and maybe even personal use.

It'd also tie in with my ideas of higher-level Extend using RAlt+Extend etc. LAlt then RAlt would make AltGr, while RAlt then LAlt would make Extend2 etc. Nice.

While we're on that topic though: I'm not sure I'm entirely on board with making LAlt the Extend key? To me it's actually less comfy to use than Caps. I'm talking about the Ext+S+T+N-type combos for instance (which in addition seem to have some scan-matrix problems with LAlt+S+N on my boards?). Takes some effort to tuck the left thumb under there for me.

With the Wide mod, the RAlt key seems a lot easier. Using DH(DbgHk)-mod, my thumbs rest right under D and H (non-modded QWERTY V and M) which makes RAlt only a single-width jump but LAlt a full double-width jump that feels quite contortive! (Caps, too, is a single-key stretch... albeit with a weaker finger.)

The Space bar is actually stupidly wide. There could have been another key for the left thumb, or at the very least better Alt and Win/GUI key positions. Of course, without the Wide mod it's just as bad on the right hand. So I guess us Wide modders can find a small comfort in this design weakness on that accord, at least.

Hmmm... how about:

RAlt   → Extend
LAlt   → AltGr
Caps   → LAlt

...with advanced functions from Alt (and Alt+Caps?) combos?

Yes, you lose your new-found Shift key that way. I know. The simplest way of getting it would be to just swap RAlt and RShift, I suppose? But I'm warming up to the RAlt-as-Extend idea, I may try that one out...

Or the simplest way:

RAlt   → Extend
Caps   → AltGr

(Since Caps is unnecessary anyway, but nice enough as a key?)

• I'd think that swapping RShift and RAlt would be quite safe and mentally nice, as they're so close to each other! You wouldn't be losing the RShift key and it's a bit redundant anyway.

• On my laptop board the LAlt is under unmodded X, as on yours. That doesn't feel so comfortable for something as important as Extend to me. As mentioned, my left thumb rests a whole key width into the space bar so it's a double-key stretch inwards for the thumb. Good enough for Alt but not for Extend in my opinion.

• I agree that the Extend feels best on the left. One reason may be that the left hand on modifiers while the right hand hits arrow keys etc is the common way? But I'm also very used to hitting keys with the RAlt key down due to AltGr, so maybe that'll work just fine as my Extend modifier? Holding the RAlt while hitting, e.g., UNEI seems comfy enough. I want to try it.

• I hope RAlt doesn't cause me any further scan matrix woes? Could be, since it may mask one of the NEI keys. I think my LAlt masks the S key vertically in my matrix. It's quite horrible really if they put the modifiers in the same matrix as the other keys, but I wouldn't put it beyond them. The physical Shift key has to be on its own connection I guess, as it's chorded so much with the rest of the keys.

• The good way to access extra layers would be chording while accessing but only holding one of the modifiers thereafter. That way, fatigue/contortions and scan matrix worries shouldn't compound.

• I have some clearmod arguments in my Extend mappings to make sure keys are redirected without the level5 shift state. That works well it seems, even on the Extend-modifiers (A/S/T). But if the level5 state is cleared on redirecting to, say, an arrow key, then it wouldn't matter if it was kept up to that point. So that's a difference from the problem with the shift state and level3 above. I think.

Update: I made the RAlt-Extend with TMK and it seems to be working really well! And without any scan matrix trouble, to boot.

It's a bit more dramatic than using Caps since I was already using the AltGr key for something else as opposed to the largely unused Caps key. But it actually feels rather nice to have the Caps key as the lv3 shift right above the lv2 shift! Heh.

Also, it's interesting that Ext-Enter(Ext+Space) goes from 1-handed to 2-handed, but Ext-Back(Ext+O) vice versa! Not a problem, just unaccustomed.

I'll test this some more and then maybe make it a standard Extend-key option. If I do, I'll make both Alt keys potential Extend keys I guess, for freedom of choice.

I love those productive threads. <3

I hate remapping modifiers though. Ugh. First snag is PKL: Remapping AltGr (or something to AltGr) is not straightforward in AHK, since AltGr is interpreted as LCtrl&RAlt.

This fails for me:

extend_key = RAlt

This, too:

SC138 = CAPITAL	0	--	--	--	--	--	; RAlt/AltGr is Extend
SC03A = RMENU	VirtualKey

(As far as I could see, RMENU is the correct Windows Virtual Key Code for RAlt. Some of those codes are a bit weird.)

Any suggestions, anyone?

I used to do stuff like this in AHK for making a custom AltGr layer, which used to work fine for me:

RAlt & SC020::
  GetKeyState, sh, Shift
  if sh = D
    Send {→}
  else
    Send {{}
return

My KLFC is definitely not a PKL fork. But it generates keyboard layout files, including PKL ones. By changing the singleton key '[ "CapsLock", "Extend" ]' to '[ "Alt_R", "Extend" ]' in the extend example, it will generate PKL files where the important changes are the lines I gave.