Type C-[ instead. This should send ASCII code 27 just like an Escape key would. C-3 may also work on some terminal (but not under X). For many terminals (notably DEC terminals) F11 generates ESC. If not, the following form can be used to bind it:
;; F11 is the documented ESC replacement on DEC terminals. (define-key function-key-map [f11] [?\e])