I was messing around with swapping the Caps Lock key and the Escape key, and this in combination with using the Colemak keyboard layout ended up causing some confusion.
Anyway it turns out to reset any options you may have set with setxkbmap it's not
as simple as just running setxkbmap gb. Instead, the solution for me was to run
setxkbmap -option ''. Not very intuitive, but there you go.