I noticed an interesting thing... I recently took six months of paternity leave (hooray for Swedens awesome social welfare!) and did not use my Typematrix keyboard with Colemak layout under that time (I did use my qwerty laptop a lot instead). I'd used Colemak full time for a few months prior to that and got up to about 50wpm. Now I'm back at work, and am happy to report that my speed is mostly unaffected!!  Perhaps the differences in keyboards helps, but it also seems that Colemak is more difficult to forget than I had imagined :)