How do I automatically (re)compile ELPA packages?

Question

I'm now installing as much as I can through MELPA and Marmalade, and I manage my ~/.emacs.d using git. However, I have git ignore *.elc files.

This means that when I install a package on one system and then start using another system, git pull only gives me the *.el files. Using these files is often slower than using *.elc.

I tried adding the following to ~/.emacs.d/init.el:

;; load the packages we've installed. Note that package-install
;; byte-compiles the packages, but .elc is ignored by git so we force recompilation here
(byte-recompile-directory (expand-file-name "~/.emacs.d/elpa") 0)
(package-initialize)

Unfortunately, this isn't equivalent to the compilation done by package.el. For example, if I install emacs-eclim, package.el doesn't compile emacs-eclim/company-emacs-eclim.el, and I get the following error:

Leaving directory `/home/wilfred/.emacs.d/elpa'

Compiling file /home/wilfred/.emacs.d/elpa/emacs-eclim-20130310.1237/company-emacs-eclim.el at Mon Mar 11 15:40:01 2013
Entering directory `/home/wilfred/.emacs.d/elpa/emacs-eclim-20130310.1237/'
company-emacs-eclim.el:35:1:Error: Cannot open load file: eclim
Warning: reference to free variable `multiple-cursors-mode'
Warning: reference to free variable `mc--read-char'
Warning: assignment to free variable `mc--read-char'
Warning: reference to free variable `multiple-cursors-mode'
Warning: reference to free variable `mc--read-quoted-char'
Warning: assignment to free variable `mc--read-quoted-char'
Warning: reference to free variable `rectangular-region-mode'
Warning: reference to free variable `rectangular-region-mode'

How do I make Emacs byte-compile only the same files as package.el would?

Answer 1

package.el actually uses exactly the same method, it's just recompiling on startup makes the errors more noticable.

The function used is package--make-autoloads-and-compile, which calls:

(byte-recompile-directory pkg-dir 0 t)

So the original code in the question is correct. However, to recompile a directory that isn't yet compiled, you can do the following:

(require 'dash)
(require 'f)

(defun was-compiled-p (path)
  "Does the directory at PATH contain any .elc files?"
  (--any-p (f-ext? it "elc") (f-files path)))

(defun ensure-packages-compiled ()
  "If any packages installed with package.el aren't compiled yet, compile them."
  (--each (f-directories package-user-dir)
    (unless (was-compiled-p it)
      (byte-recompile-directory it 0))))

(ensure-packages-compiled)

Answer 2

I recommend not keeping the packages in version control (you wouldn't put .o files under revision control, would you?). Here is the code I use to keep my packages in sync:

(setq jpk-packages
      '(
        ac-dabbrev
        ...
        yasnippet
        ))

(package-initialize)
(add-to-list 'package-archives
             '("melpa" . "http://melpa.milkbox.net/packages/"))
(add-to-list 'package-archives
             '("org" . "http://orgmode.org/elpa/"))

(when (not package-archive-contents)
  (package-refresh-contents))

(dolist (pkg jpk-packages)
  (when (and (not (package-installed-p pkg))
           (assoc pkg package-archive-contents))
    (package-install pkg)))

(defun package-list-unaccounted-packages ()
  "Like `package-list-packages', but shows only the packages that
  are installed and are not in `jpk-packages'.  Useful for
  cleaning out unwanted packages."
  (interactive)
  (package-show-package-list
   (remove-if-not (lambda (x) (and (not (memq x jpk-packages))
                            (not (package-built-in-p x))
                            (package-installed-p x)))
                  (mapcar 'car package-archive-contents))))

I put the above in init.el (which of course is under version control), and it installs any packages that don't exist yet when emacs starts up. Updating is done from the buffer package-list-packages creates. package-list-unaccounted-packages shows all the packages that are installed but not in my jpk-packages list, and makes it easy to remove the ones that I took out of the list.

To answer your specific question, I'd just delete the elpa directory and reinstall everything (using the above code).

Answer 3

The solution I like to the more general question of "How do I automatically re-compile any outdated .elc file" is to use https://github.com/tarsius/auto-compile

I used to have a handful of custom solutions which covered most situations I ran into, but this library covers everything I was doing and more besides. Just initialise it before loading anything else, and you're sorted.

As to the original question of not compiling files which were not originally compiled, that's what the 0 argument to byte-recompile-directory does, so you're explicitly asking for that behaviour. The default behaviour is to only recompile existing outdated .elc files, so you should simply remove that 0.

Answer 4

I do not know the answer to your specific issue about byte-compiling packages exactly the same way package.el would.

However, for your more general problem about maintaining your emacs configuration directory in git while installing extensions through Melpa or marmalade, I suggest you give a look to pallet, which has been designed to solve this very problem.

Answer 5

(byte-compile-file) worked for me when I need to manually recompile packages I have previously installed using elpa.

Particularly for this situation: https://github.com/yjwen/org-reveal/issues/76t

Answer 6

The problem you have is simply that you byte-compile files before initializing your packages. If you switch the two lines, your problem should disappear. The error you get (Cannot open load file: eclim) is because ~/.emacs.d/elpa/emacs-eclim-20130310.1237/ is not yet in your load-path, and package-initialize will add it to your load-path (along with a few other things).