There are different ways of using Vim. If you're using it mnemonically, then moving the whole thing to Colemak won't be a problem. You could use an Extend mapping layer (see my sig topics) to complement with sensible movement keys to compensate for the jumbling of HJKL if you like; I love such mappings since they're omnimode/omniapp.

If you're using Vim purely by key placement, then I think you could "reverse remap" it so that only the text-mode input becomes remapped to Colemak and the rest remains as it was. Haven't seen anyone do that but it should be entirely possible. Then you'd run QWERTY in the background and Vim would change the text input to Colemak for you.