I just started learning Colemak about a week ago, and I'm finally past the painful initial part and can maintain 30-40 wpm.
I picked up Dvorak about 8 years ago, and have maintained speed in both Dvorak and QWERTY at a sustained 100+ WPM. I spent about my first 6 months exclusively on Dvorak, but at that point I recall it being very easy to jump back into QWERTY. Once I was comfortable enough with both, I did drills where I would switch layouts every other sentence or word, and I've used different layouts by default on different computers.
I don't know how feasible it'll be to maintain three layouts, though. One of my bigger concerns is how similar Colemak and QWERTY are; I feel like that's going to make it harder to mentally distinguish those two layouts. Have you tried this? Even if you're just maintaining QWERTY and Colemak, are the similarities an issue ever?