Use sticky keys. You'll find this on the accessibity settings of ubuntu. I recommend that you use an applet for showing the status of the modifiers. I know gnome has one, but I don't know its name. Look these up for more detailed info., as don't have ubuntu/gnome here at hand.

You can't have shift+caps turning caps on/off because (unshifted) caps lock starts acting as both backspace and caps lock. It's a bug or something.

But you can try some other key. To edit your keyboard layout, type:
sudo gedit /usr/share/X11/xkb/symbols/us
Find the Colemak section.  Inside the curly and square brackets you will see four columns: normal, shift+, third level chooser+, and shift+third level chooser+

The three- or four-character names of the keys you can use inside the angle brackets can be found here:
http://www.charvolant.org/~doug/xkb/htm … 0000000000
( Caps Lock is CAPS )

And the names of the characters and keys you can type inside the curly and square brackets can be found here:
http://cvsweb.xfree86.org/cvsweb/*check … h?rev=1.14
( remove the XK_ )
( Caps Lock is Caps_Lock )
( By the way, the third-level chooser is ISO_Level3_Shift )

Optionally, if you haven't done so already, you can add these lines in your .profile ( gedit ~/.profile ):

# turn on repeating for key 66 (Caps Lock)
xset r 66