deft mode terribly slow

Question

I just started using emacs24 (the one that comes with Ubuntu) in order to use deft mode to mimic the behavior of notational velocity on the Mac. My ~/.emacs-d/init.el looks like this:

(add-to-list 'load-path "~/.emacs.d")
(require 'deft)
(setq deft-directory "~/Notes")
(setq deft-use-filename-as-title t)

in order to load my folder with txt files. I have a few thousand text files in the folder with file names that include CJK characters. Now, running deft mode is now /extremely/ slow, it's impossible to even type to search among the files. After typing a word I have to wait for several minutes before it shows up on the screen and searches through the notes. Even turning emacs off takes a long time. My computer is a Lenovo W540 64-bit i7-4800MQ CPU @ 2.70GHz × 8 and 23.2 GiB memory (it tells me). I'm running Ubuntu 14.04. What might be wrong with my emacs?

I'm new to emacs and frankly I don't even understand what I just wrote above, I'm just trying to piece the setup I want together from tutorials I find online, so forgive any stupidities that might have slipped into my question. Thanks in advance!

Answer 1

I'd be happy to be proven wrong, but I don't think deft is intended to work with "a few thousand text files"! You would surely want some kind of specialised indexing and searching tool to efficiently work with that quantity of data.

I'm a bit out of date, but I believe deft is just opening the files in Emacs and using regexp searching to find matches (which is just fine for smaller numbers of files, but it doesn't scale to your use-case).

The only suggestion I can make is to ensure that deft.el is byte-compiled (i.e. you have a deft.elc file). Use M-x byte-compile-file otherwise. But even if you're currently using an uncompiled version, I don't believe that is going to provide the sort of performance improvements you'd need to make this a viable solution.

Answer 2

I found this solution that limits the number of notes shown when using incremental search: https://github.com/jrblevin/deft/issues/43

  ;; Overwrite `deft-current-files` for the `deft-buffer-setup` and limit it to 30 entries
  (defun anks-deft-limiting-fn (orig-fun &rest args)
    (let
        ((deft-current-files (-take 30 deft-current-files)))
      (apply orig-fun args)))

  (advice-add 'deft-buffer-setup :around #'anks-deft-limiting-fn)

It helps me somewhat, although deft is still not really snappy when using a couple of thousand notes, as I also do.

Answer 3

As of version 0.9, a variable deft-file-limit was added, setting the "maximum number of files to list in the Deft browser."

So to replicate trmdttr's configuration answer for 30 files, you'd add this to your configuration:

(setq deft-file-limit 30)

Answer 4

notdeft is a package that was created with this sort of use case in mind; it uses an external backend for indexing and searching (and can handle a huge number of huge files)

EDIT: Here’s a link to the GitHub: https://github.com/hasu/notdeft

It’s a little bit less intuitive to setup (that’s the biggest highlight of Deft I think). You have to get the backend compiled, but the basic interface is the same; you can just type words, and it updates the results in real-time. There’s an advanced way to use the query syntax, but it’s not necessary. Just searching for “noteContent1 noteContent2 :tagName:” will list files that have both the contents and the tag somewhere, and it scales to gigabytes of notes.

Answer 5

I found deft searching to be unusably slow with Emacs 28 despite setting deft-file-limit. I used notdeft, which worked reasonably well, but didn't work well with my cross-platform .emacs.d directory. I'm sure I could have hacked notdeft to look in an OS + arch specific directory, but it wasn't critical enough for me strong textto invest the time.

Emacs 29 restores deft to something close to its previous speed. Deft with Emacs 29 might not be sufficient for the "thousands of files" case, but it works reasonably well with around a thousand files. It's not super fast, but entirely livable.

Emacs 29 isn't officially released yet, but I've found its various speed improvements (like not be crippled by long lines) to be worth jumping on the Emacs 29 bandwagon.