I would like to share the following experience. I used colemak for around 2 years and had come up to my previous qwerty speed. I also had rearranged a lot of the symbols and made use of a thumb button on my Kinesis to access a lot of symbols for programming.

Then I took 8 months of paternity leave (hooray for Sweden :)  where I didn't use much computers, and when I did, it was qwerty. Well, I am back at work again, and Colemak came instantly back to me, including all the placement of my custom symbols and thumb-modifiers and vim shortcuts. I was literally back up to my usual usage after a few days.

I attribute a large amount of this to the Kinesis keyboard, which I don't otherwise use, so I have very specific muscle memory for it, and an entirely different way of typing on qwerty (self-learned half-looking, not using all my fingers).