Do you access most of these systems remotely (in which case you have a better chance of your layout being preserved, depending on remote software) or physically?  If physically, how many would you have (to be safe, let's say administrator) access to change?  Obviously, the Colemak/QWERTY ratio makes a huge difference in determining whether learning is "worth it".

The keyboards shouldn't make any difference, as the layout is most often determined by the machine itself.

It's generally not considered a good idea to train by looking at your keyboard - this can lead to bad habits. Rather, you should print out a copy of your layout and put it near the screen.  This makes backlight and keycap swapping irrelevant.  In fact, I generally get annoyed by backlight because it only serves as an additional distraction for no reason, and retaining QWERTY keycaps can be useful on those occasions where (due to hard defaults) you need to type QWERTY.

As a Linux user, and one that has to remote shell a lot, all I have to do is swap my layout locally, I can then SSH onto another machine, and not have to touch that machine's layout.  If on the other hand I need to walk down to the server room, it's more of a pain, well at least I think it is, as you need to be root to use loadkeys.

I think windows remote sessions are different, I think you get the remote layout when you Rdesktop.   That would be a pain for me.  I chose Dvorak in part for these rare occasions, but switching layouts isn't the easiest thing on Windows, despite what others say.  Newer versions (Windows 8) seem to do this better.

I was in the same boat, I'd gone for far too long in my mind, not touch typing.

It's hard to say what the gains are at this point.  But a rough summary goes something like this:  It took no time learning the key map, literally a day or two.  Don't bother swithing your keycaps (really, don't bother).  The first few weeks were really slow going, and a bit of a fight.  When I finally reached about 30WPM - I felt I was getting somewhere.    From there, practice and use edges your WPM slowly forward.

The gains:
Using the command line is far easier.   Typing in general is easier and isn't so much a bottle neck.  Less neck strain.  Faster WPM. 

The pains:
Feels like a long period of adjustment.  Typing still can feel alien.   Touch typing hard to get used to.  Loss of Qwerty skills.  Ridicule and ill understanding from peers (I try not to mention it.)  Switching layouts not easy across OSs (not a problem if you spend time on one PC.)

Stephen Fry pointed out that it takes longer to learn how to touch type than to learn to fly a plane.   So you might be better off saving the time and learning something else.   It's such an investment in time and energy (that's not to say it won't pay dividends), that you'll be reluctant to give it up!

To get things into perspective, there are MessageEase users on this forum who have notched up pretty impressive WPMs on their phones in a short amount of time.

There may be other efficiencies you can make (without learning to touch type), software and tools.

The other downside is that you might bore your friends senseless by trying to explain typing efficiencies of using a different layout.  But don't worry there are support groups for that kind of thing (See: Colemak Forum.)

I'd been using computers for 20+ years without touch typing, and I'm sure my hunt and pecking was in part responsible for some physical injuries.  As a late learner I didn't find the fingering that easy.  If you are a guitarist, flutist or pianist you might find it much easer than I.

Seems a shame to not touch type on those museum exhibits.

I wouldn't count on increased speed.  Normally, people can get to about the same speed regardless of layout.  Dvorak might have a little bit of an edge here due to hand-alternation, but not nearly as much as back when we were using typewriters.

The real difference is comfort.  And, unfortunately, that's something you'll have to judge for yourself.  There are no serious studies I know of on the advantages/disadvantages of keyboard layouts for health (even if there were, people still wouldn't switch from QWERTY, I guess).  On the upside, thanks to computers, it's easier than ever to collect statistics like distance traveled, repeated finger usage, etc (see Carplax, Keyboard Layout Analyzer, etc) and the story there is pretty telling, even if we have to interpret the data ourselves.

I've been learning Colemak for almost a month now and just breaking 50 WPM average compared to my 60-70 WPM average on QWERTY. Apart from comfort it's also much quieter! I do type racers and training at night and with QWERTY I was aware that I may be disturbing people in the next room but with Colemak there's hardly any noise at all.

One other thing when you're learning, it's much harder typing forum posts than copying text that you can see.

Stephen Fry is funny. However, I've spent a lot more time typing than flying (unfortunately?) so the time spent learning Colemak has been well worth it for me! :)