Emacs: the code in the body of a defun or defmacro cannot refer to surrounding lexical variables?

Question

Update 2013 May: As of GNU Emacs 24.3.1, (let .. (defun..)) bytecompiles just fine without warning and the bytecompiled code works the same as not-compiled code. Just don't forget to add the file variable lexical-binding: t to the file to be bytecompiled. Workarounds at the end of this question is now not necessary.

---

Lexical Binding - Emacs Lisp Manual has this paragraph:

Note that functions like symbol-value, boundp, and set only retrieve or modify a variable's dynamic binding (i.e. the contents of its symbol's value cell). Also, the code in the body of a defun or defmacro cannot refer to surrounding lexical variables.

I am not sure if I am getting the meaning of the second sentence right. In the following code which should be run in lexical binding mode, the code in the body of a defun is successfully referring to the lexical binding value of the name n.

(let ((n 0))
  (defun my-counter ()
    (incf n)))

(my-counter) ;; 1
(my-counter) ;; 2

Is the sentence simply saying that (let .. (defun ..)) is a bad practice?

---

Workarounds:

;; -*- lexical-binding: t -*-

;; a way to define the counter function without byte-compile error or warning

(defvar my--counter-func
  (let ((n 0))
    (lambda ()
      (setq n (1+ n)))))

(defun my-counter ()
  (funcall my--counter-func))

;; another way to define the counter function, again without byte-compile error or warning

(fset 'my-another-counter
      (let ((n 0))
        (lambda ()
          (setq n (1+ n)))))

And here's the code for testing the above code:

;; run:
;; emacs -q --load path-to-the-el-file-of-this-code.el

(load "path-to-file-defining-my-counter.elc") ;; loading the ELC file to test if byte-compiled code runs as expected.

(print (my-counter)) ;; 1
(print (my-counter)) ;; 2

(print (my-another-counter)) ;; 1
(print (my-another-counter)) ;; 2

Answer 1

The code does not byte-compile well at least in Emacs 24.1.1. I saved the following code in the foo.el file, which uses setq in place of incf in order to avoid any possible effects by the cl library:

;; -*- lexical-binding: t -*-

(let ((n 0))
  (defun my-counter ()
    (setq n (1+ n))))

When I tried to byte-compile it (M-x byte-compile-filefoo.el), I got the following warning messages:

foo.el:3:1:Warning: Function my-counter will ignore its context (n)
foo.el:3:1:Warning: Unused lexical variable `n'
foo.el:5:11:Warning: reference to free variable `n'
foo.el:5:17:Warning: assignment to free variable `n'

All of the messages are indicating that the code in the body of the defun construct cannot refer to the surrounding lexical variable n as the manual claims.

Actually, when I loaded the byte-compiled code (M-x load-filefoo.elc) and evaluted the (my-counter) form, I got the following erorr:

Debugger entered--Lisp error: (void-variable n)
  ...

Unfortunately, I'm not sure why the code appears to work when evaluated in the form of source code.

Answer 2

As I replied on gnu.emacs.help, you can use (defalias 'foo (lambda ...)) to work around that limitation. And that limitation is lifted in Emacs's development code.

Answer 3

It's perfectly fine to refer to variables in lexical scope(*) from within a defun, just as you are doing above, and just as the "my-ticker" example on that manual page does as well.

Either I am missing something the line in the manual that says:

the code in the body of a defun or defmacro cannot refer to surrounding lexical variables.

should say something more like:

code in the body of a defun can only access lexical variables if they are defined within the same lexical scope.

NOTE: There are comments in the other answers about problems byte-compiling this kind of code. Those should be fixed in the latest emacs. I've verified in v24.2.50.1 that this byte compiles and loads correctly.