Question

Spoiler alert: this is the answer to number 7 on Project Euler.

I'm learning Lisp and I was using compileonline.com to run my code. It was running out of memory on a simple program though, so I switched to a desktop version. However, even that is running out of memory. Can somebody tell me why this is so bad?

In its current form it doesn't hang, but if I change it to:

(if (= primecount 10001) 

it hangs. Even something as small as 1000 hangs.

(setq current 3)
(setq primecount 1)
(setq primes (list 2))
(setq isprime "yes")

(loop for x from current do
  (list 
    (setq isprime "yes")
    (loop 
      for prime in primes do 
        (if (= (mod current prime) 0) 
          (list (setq isprime nil) (return)) 
          nil
        )
    )
    (if isprime 
      (list 
        (setq primecount (+ primecount 1)) 
        (setq primes (cons current primes)) 
        (if (= primecount 100) 
          (return) 
          nil
        )
      )
      (setq current (+ current 1))
    )
  )
)

(write-line (write-to-string primes))
Was it helpful?

Solution 2

TL;DR: the problem is not with memory, but with speed.

You were very inventive in overcoming your lack of knowledge. :) Actually, we have progn in Common Lisp, that groups several expressions to be evaluated in order:

(defun myprimes-to (n &aux          ;; &aux vars may have init values
                      (current 3)
                      (primecount 1)
                      (isprime t) ; "yes"   ;; t is better 
                      (primes (list 2))
    (loop  ;for x from current do   ;; for a bare loop, just `loop` is enough
      (progn  ;list                 ;; `progn` groups expressions together to be
        (setq isprime t) ; "yes"    ;;    evaluated in order, one after another
        (loop                       ;;      (not even needed here, see comments below)
          for prime in primes do 
            (if (= (mod current prime) 0) 
                (progn  ;list
                  (setq isprime nil) 
                  (return))         ;; exit the loop
                nil))               ;; don't have to have alternative clause
        (if isprime 
            (progn  ;list 
              (incf primecount)  ; (setq primecount (+ primecount 1)) ;; same but shorter
              (setq primes (cons current primes)) 
              (if (= primecount n)  ;; abstract the n as function's argument
                  (return)          ;; exit the loop
                  nil))
            ;; What?? increment your loop var _only_ if it's not prime?
            (setq current (+ current 1)))))
    ;; (write-line (write-to-string primes))
    primes)                         ;; just return them

So if current is prime, it is retried. Since it was last added into primes, it'll now deem itself as non-prime, and the iteration will proceed. The results are correct, but the logic is garbled.

More importantly, there are two major inefficiencies there, algorithm-wise: you test by all primes so far, and in reverse. But, any given number is far more likely to have a smaller divisor than a bigger one; so we should test by primes in ascending order. And, we can stop when a prime p is p*p > current – because if current == a*b and a <= b, then a*a <= b*a == current. Stopping early gives enormous speedup, it changes run time complexity from ~ n^2 to, roughly, ~ n^1.5 (in n primes produced).


Do notice this: your code is worse than ~ n^2, algorithmically. It would be ~ n^2, if it were testing by primes in ascending order. -- Why is this important? It provides partial answer to your question: your problems are not with memory, but speed. Whatever time it takes your code to finish for n=100, it will take more than 100x that to get to n=1000 (because 1000/100 == 10, and 10^2 = 100). And it will take it more than another 100x that, to get to the n=10000. So if the n=100 problem took a second of run time, n=1000 will take 100 seconds (~ 2 minutes), and n=10001 – 200 minutes (more than 3 hours).

We can always take estimates of run-time orders of growth empirically; see this WP article.


So you need to maintain primes in ascending order, and append each new prime to their end, in O(1) time. You need to achieve same results as the code line (setq primes (append primes (list current))) would achieve; but you can't use this code line because it is O(n) in time, and will drastically slow down the overall program. Same with (setq primes (reverse (cons current (reverse primes)))). And we can't maintain primes in reverse order because we must test in ascending order, and simply calling (reverse primes) for each new current is also an O(n) time operation.

The solution is to use rplacd: (rplacd (last primes) (list current)). Even this is still not right, because last is also an O(n) operation. Instead, you will have to maintain an additional variable, pointing at the primes list's last cons cell, for that, and advance it each time you add a new prime there: (setq last_cell (cdr last_cell)).

When you'll run the fixed code, you will find it finishes the 10,000 job in about 0.1 second, and runs at about ~ n^1.4 run-time orders of growth in that range.

OTHER TIPS

Can you explain

  • what for the variable X is and why it is unused?

  • what are the calls to LIST doing?

  • why are you using all these global variables?

  • why are you not defining any functions?

  • why is the code iterating over all primes?

Here is refactored and slightly improved version, which is based on your code:

(defun find-primes (n &aux (primes (list 2)) (l-primes primes) (n-primes 1))
  "returns a list of the first n primes"
  (flet ((prime-p (i &aux (limit (isqrt i)))
           "returns T if i is a prime number"
           (loop for prime in primes
                 until (> prime limit)
                 when (zerop (mod i prime))
                 do (return-from prime-p nil))
           t))
    (loop for i from 3
          until (>= n-primes n)
          do (when (prime-p i)
               (setf (cdr l-primes) (list i)
                     l-primes (cdr l-primes))
               (incf n-primes))))
  primes)

(print (find-primes 10000))

Features of the above function:

  • it keeps a list of primes, increasing
  • it keeps a reference to the last cons of the list of primes, to add to the end of a list efficiently
  • it has a predicate sub-function prime-p which returns T if a number is a prime
  • the predicate uses the function isqrt to limit the numbers to search

With a check for N and a local macro for the push to the end it looks like this:

(defun find-primes (n &aux (primes (list 2)) (l-primes primes) (n-primes 1))
  "returns a list of the first n primes"
  (check-type n (integer 1))
  (flet ((prime-p (i &aux (limit (isqrt i)))
           "returns T if i is a prime number"
           (loop for prime in primes
                 until (> prime limit)
                 when (zerop (mod i prime))
                 do (return-from prime-p nil))
           t))
    (macrolet ((push-last (obj place)
                 `(setf (cdr ,place) (list ,obj)
                        ,place (cdr ,place))))
      (loop for i from 3
            until (>= n-primes n)
            do (when (prime-p i)
                 (push-last i l-primes)
                 (incf n-primes)))))
  primes)

Let's try it:

CL-USER 3 > (time (find-primes 10000))
Timing the evaluation of (FIND-PRIMES 10000)

User time    =        0.052
System time  =        0.000
Elapsed time =        0.044
Allocation   = 292192 bytes

Your code is somewhat buggy. list function is there to construct and return a list, it shouldn't enforce an evaluation order. You want progn to execute a bunch of forms as one, and it return the value of its last form.

I rewrote it a bit:

(defvar current)
(setf current 3)
(defvar primecount)
(setf primecount 1)
(defvar primes)
(setf primes (list 2))
(defvar isprime)

(loop for x from current do
  (setq isprime t)
  (loop 
    for prime in primes do 
      (when (zerop (mod current prime))
        (setq isprime nil) (return)))
  (if isprime 
      (progn 
        (incf primecount)
        (push current primes)
        (when (= primecount 10001) 
          (return)))
      (incf current)))
(write-line (write-to-string primes))

I hazard a guess that your code uses a lot of memory because you use lists as your data structure, and as they get huge, the garbage collector struggles with creation and destruction of lists.

Your program works in my SBCL for 10001 like this (wrapped in a do-stuff function):

CL-USER> (time (do-stuff))
Evaluation took:
  13.486 seconds of real time
  13.438000 seconds of total run time (13.437000 user, 0.001000 system)
  99.64% CPU
  40,470,003,369 processor cycles
  1,674,208 bytes consed

CL-USER> (room)
Dynamic space usage is:   134,466,112 bytes.
Read-only space usage is:      5,856 bytes.
Static space usage is:         4,032 bytes.
Control stack usage is:        8,808 bytes.
Binding stack usage is:        1,056 bytes.
Control and binding stack usage is for the current thread only.
Garbage collection is currently enabled.

Breakdown for dynamic space:
  45,803,376 bytes for   569,191 instance objects.
  22,831,488 bytes for 1,426,968 cons objects.
  16,472,576 bytes for    16,135 code objects.
  15,746,176 bytes for   130,439 simple-vector objects.
  13,794,096 bytes for    36,381 simple-character-string objects.
  19,850,400 bytes for   440,297 other objects.
  134,498,112 bytes for 2,619,411 dynamic objects (space total.)
Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top