Differences in whether realization of a lazy sequence inside of a lazy sequence occurs

Question

I wondered: What happens when you embed an expression that forces realization of a lazy sequence inside of an outer lazy sequence that's not realized?

Answer: It seems to depend on how you create the outer lazy sequence. If the outer sequence comes from map, the inner sequence is realized, and if the outer sequence comes for iterate, it's not.

Well, I'm pretty sure that that is not the right way to describe what happens below--I'm pretty sure that I'm not understanding something. Can someone explain?

(There is one quirk, which is that while map returns a LazySeq, iterate returns a Cons wrapped around a LazySeq. So in the tests for class and realization below, I look at the rest of the output of iterate. I don't believe that this difference between map and iterate has anything to do with my question.)

(def three-vec (range 1 4))

(defn print-times-10-ret [x]
  (let [y (* 10 x)] 
    (println "[" y "] " ) 
    y))

(defn once [xs] (map print-times-10-ret xs))

(defn doall-once [xs] (doall (map print-times-10-ret xs)))

(defn doa-twice [xs] (once (doall-once xs))) ; "doa" since only half doall-ed

;; Here the inner sequence seems to get realized:
(def doa-twice-map (doa-twice three-vec))
; printed output:
; [ 10 ]
; [ 20 ]
; [ 30 ]

;; Here we create a lazy sequence that will call doall-once when
;; realized, but nothing gets realized:
(def doall-once-iter (iterate doall-once three-vec))
; no printed output

(class doa-twice-map)
; => clojure.lang.LazySeq

;; Note that this is not realized, even though the inner seq was realized (?):
(realized? doa-twice-map)
; => false

(class (rest doall-once-iter))
; => clojure.lang.LazySeq

(realized? (rest doall-once-iter))
; => false

Answer 1

"What happens when you embed an expression that forces realization of a lazy sequence inside of an outer lazy sequence that's not realized?"

If the expression which forces the realization of the inner sequence is in the unrealized portion of the outer sequence, then nothing.

"Answer: It seems to depend on how you create the outer lazy sequence. If the outer sequence comes from map, the inner sequence is realized, and if the outer sequence comes for iterate, it's not."

No, it only depends on whether your forcing expression is in the unrealized portion. Neither map nor iterate realize anything that isn't already realized.

"describe what happens below"

You need to think through the evaluation rules. The behavior of your examples is mainly a consequence of eager vs. lazy evaluation (Clojure is eager) and is only tangentially related to lazy sequences.

First example

First consider the form

(def doa-twice-map (doa-twice three-vec))

The first element def indicates a special form, with special evaluation rules, but in particular when a second argument is supplied it is evaluated. In particular, you are evaluating

(doa-twice three-vec)

The evaluation of this form is an invocation of doa-twice on three-vec, which looks like the following (after substitution)

(once (doall-once three-vec)))

To evaluate this form, the arguments have to be evaluated first (Clojure does eager evaluation). In particular, you are evaluating

(doall-once three-vec)

And your doall-once is invoked with argument three-vec. Now you are evaluating

(doall (map print-times-10-ret three-vec))

This invokes doall on the argument, which is first evaluated to create a lazy-sequence. The doall forces realization of that lazy-sequence by definition. During that realization you are invoking print-times-10-ret on successive elements of three-vec, forcing realization of it along the way.

So the behavior your are seeing here is the consequence of the chaining of the eager evaluation. Eager vs lazy evaluation (Clojure is eager) is not to be confused with lazy vs. non-lazy sequences (Clojure has both).

Second Example

When you evaluate the special def form

(def doall-once-iter (iterate doall-once three-vec))

This causes evaluation of the second argument

(iterate doall-once three-vec)

Evaluation of this form invokes iterate on its arguments. Iterate creates a Cons-cell of three-vec and a lazy-sequence. That Cons-cell which is the result of the evaluation is a value, thus ending the descent of the chaining of eager evaluation here. This value is set as root binding of a doall-once-iter var by the def. This is the end of the evaluation of the def form.

The Parenthetical

"(There is one quirk, which is that while map returns a LazySeq, iterate returns a Cons wrapped around a LazySeq. So in the tests for class and realization below, I look at the rest of the output of iterate. I don't believe that this difference between map and iterate has anything to do with my question.)"

Correct, this does not play a role. If iterate returned a LazySeq object it would still be a value and stop the descent of the evaluation chain as above.

See also count-realized for how to count the realized/non-lazy portion of a sequence without forcing realization. Or, seq-realized? in response to what I think is also your question on the Clojure Google Group.