What's so great about lexical scoping?

Question

The reason I don't understand why Emacs 24's new lexical scoping features are that great is that I can't think of any new functionality that couldn't have been implemented without them. For example, the following closure:

(setq lexical-binding t)
(defun f1 (num1)
  (lambda (num2)
    (setq num1 (* num1 num2))))

(fset 'f2 (f1 5))
  ==> (closure ((num1 . 5) t) (num2) (setq num1 (* num1 num2)))
(f2 5)
  ==> 25
(f2 2)
  ==> 50

Can be implemented with regular dynamic scoping like so:

(defun f1 (num)
  (let ((tmpvar (make-symbol "num")))
    (set tmpvar num)
    `(lambda (num2)
       (set ',tmpvar (* (eval ,tmpvar) num2)))))

(fset 'f2 (f1 5))
  ==> (lambda (num2) (set (quote num) (+ (eval num) num2)))
(f2 5)
  ==> 25
(f2 2)
  ==> 50
(fset 'f3 (f1 9))
  ==> (lambda (num2) (set (quote num) (+ (eval num) num2)))
(f3 3)
  ==> 27
(f3 2)
  ==> 54
(f2 10)
  ==> 500

Okay, so not all languages have something analogous to elisp's uninterned symbols, so I understand why lexical scoping is so great in their case. But what about elisp? Can you think of anything that I can do now (as of Emacs 24) that I couldn't do before, thanks to lexical scoping?

Answer 1

There have always been workarounds to simulate lexical binding in Emacs, so it's not so much about being able to do new things.

The manual says:

Lexical binding opens up a lot more opportunities for optimization, so Emacs Lisp code that makes use of lexical binding is likely to run faster in future Emacs versions. Such code is also much more friendly to concurrency, which we want to add to Emacs in the near future.

I think that is the primary benefit.

The flip side to this is that dynamic binding was purposely chosen when Emacs was written for good reasons which still hold true today, and so lexical binding should certainly not be considered the New Way of doing things.

Global variables are generally considered a bad idea in programming, but Emacs is an unusual case where this doesn't really apply, because much of its immense flexibility -- one of the key things that makes Emacs great -- derives directly from dynamic binding. Without dynamic binding, it would not be possible to bend the application to the requirements of the individual user to anything like the extent that Emacs allows.

In my opinion, lexical binding should be used cautiously and only for variables for which another user could not conceivably find a reason to override. By default variables should be defvar'd so that the ability to customize the behaviour (even in ways that the author did not anticipate) is preserved.

Answer 2

You don't need uninterned symbols, use cons instead of make-symbol, car instead of eval, and setcar instead of set and your example will work just as well (and be more efficient).

Note also that the progression from machine-language to ever higher-level languages has been mostly based on making more and more things impossible (or at least much harder). Of course those facilities taken away from the programmers were rarely used and/or considered too dangerous. Think of using uninitialized variables (possible in C, but impossible in Java and many other languages), or jumping into the middle of an instruction.

As for some downside of your example code: not only it's less readable, but the compiler basically won't be able to know that you're constructing code, so it won't be allowed to look inside that "`(lambda ...)" to compile it, expand its macro calls, give you warnings about suspicious elements, ...

Answer 3

I think implementations of OO programming tend to fit well with lexical scope. An object with state maps rather directly to a lexical closure.

I'm sure that the CLOS implementation in common lisp leverages lexical scope heavily. It's hard for me to imagine how that spec could be implemented with dynamic scope only, but I'm sure it's possible.