Why don't reader macro extensions propagate to runtime (read)?

Question

Why does the following not work?

;;;; foo.lisp
(in-package :cl-user)

(eval-when (:compile-toplevel :load-toplevel :execute)
  (require :cl-interpol))

(cl-interpol:enable-interpol-syntax)

(defun read-and-eval (s)
  (eval (read-from-string s)))

(cl-interpol:disable-interpol-syntax)

then:

LISP> (load (compile-file "foo.lisp"))
=> T

LISP> (read-and-eval
        "(let ((a \"foo\")) (princ #?\"${a}\"))")
=> no dispatch function defined for #\?

Answer 1

CL:READ dispatches based on the readtable bound to CL:*READTABLE* at the time the call to READ runs. Under the hood ENABLE-INTERPOL-SYNTAX is creating a new readtable, setting CL:*READTABLE* to hold it, and stashing the old value of CL:*READTABLE*. DISABLE-INTERPOL-SYNTAX is unstashing the previous readtable and setting CL:*READTABLE* to again hold it. Minimally changing your original setup, you can arrange for the behavior you wanted by the following:

(in-package :cl-user)

(eval-when (:compile-toplevel :load-toplevel :execute)
  (require :cl-interpol))

(cl-interpol:enable-interpol-syntax)

(defvar *interpol-reader* *readtable*)

(cl-interpol:disable-interpol-syntax)

(defun read-and-eval (s)
  (let ((*readtable* *interpol-reader*))
    (eval (read-from-string s))))

The call to disable the syntax could be placed anywhere after the defvar and read-and-eval will still work, but if you want to directly input interpol syntax in the file that syntax will have to be placed between the enable and disable calls. For that latter purpose it is significant that the interpol calls expand into EVAL-WHENs, for the same reason that it is necessary for your call to REQUIRE to be within an EVAL-WHEN; that is, the effects need to have already happened when the latter forms are READ.

CL-INTERPOL's interface abstracts what is happening, so I will show you how you might manually create and change a readtable:

;; Create a fresh readtable with standard syntax
(defvar *not-readtable* (copy-readtable nil))

;; A simple reader function
(defun not-reader (stream char &optional count)
  "Like ' but for (not ...) instead of (quote ...)"
  (declare (ignore count char))
  `(not ,(read stream t nil t)))

;; Mutate that readtable so that the dispatch character you want
;; calls the function you want
(set-macro-character #\! 'not-reader nil *not-readtable*)

;; Try it out
(let ((*readtable* *not-readtable*))
  (read-from-string "(if !foo bar baz)"))

=>
(IF (NOT FOO)
    BAR
    BAZ)

Answer 2

Because there's only a single reader, with global state. You're effectively turning your macros on and off. In this case the reader macros are enabled only for the duration that your read-and-eval function is read at compile time.

In this case you would need to set the macros up within the read-and-eval function to ensure the reader is in the proper state when you need it.