Emacs syntax highlighting of nested regular expressions

Question

I'm trying to write an Emacs major mode for working with biological sequence data (i.e. DNA and peptides), and I want to implement syntax highlighting where different letters are colored differently. Since the mode needs to be able to differentiate DNA sequences and amino acid sequences and color them differently, I am putting each sequence in the files on a single line with a single-character prefix (+ or #) that should indicate how the following line should be highlighted.

So, for example, if the file contained a line that read:

+AAGATCCCAGATT

The "A"s should all be in one color that is different from the rest of the line.

I have tried the following as a test:

(setq dna-keyword
 '(("^\+\\([GCT\-]*\\(A\\)\\)*" (2 font-lock-function-name-face))
  )
)

(define-derived-mode bioseq-mode fundamental-mode
  (setq font-lock-defaults '(dna-keyword))
  (setq mode-name "bioseq mode")
)

But that only matches the last A instead of all of them.

My first thought was to try to match the whole line with one regexp and then use another regexp to match just the A's within that line, but I have no idea if that's possible in the context of font-lock-mode or how it would be accomplished. Any ideas on how to do something like that, or how to accomplish this in a different way?

Answer 1

Indeed, Emacs provides just what you need to do this with the "anchored match" feature of font-lock mode. The syntax is a bit hairy, but it allows you to specify additional "matchers" (basically a regexp, subexpression identifier and face name) which (by default) will be applied following the position where the main "matcher" regexp finished up to the end of the line. There are more complicated ways of customizing exactly what range of text they apply to, but that's the general idea.

Here's a simple example which also shows how you could define your own faces for the purpose:

(defface bioseq-mode-a
  '((((min-colors 8)) :foreground "red"))
  "Face for As in bioseq-mode")

(defface bioseq-mode-g
  '((((min-colors 8)) :foreground "blue"))
  "Face for Gs in bioseq-mode")

(setq dna-keyword
      '(("^\\+" ("A" nil nil (0 'bioseq-mode-a)))
        ("^\\+" ("G" nil nil (0 'bioseq-mode-g)))))

You can also specify two or more anchored matchers for one main matcher (the main matcher here being the regexp "^\\+"). To make this work, each anchored matcher after the first needs to explicitly return to the beginning of the line before beginning its search; otherwise it would only begin highlighting after the last occurrence of the previous anchored matcher. This is accomplished by putting (beginning-of-line) in the PRE-MATCH-FORM slot (element 2 of the list; see below).

(setq dna-keyword
      '(("^\\+"
         ("A" nil nil (0 'bioseq-mode-a))
         ("G" (beginning-of-line) nil (0 'bioseq-mode-g)))))

I think it's mostly a matter of taste which you prefer; the second way might be slightly clearer code if you have many different anchored matchers for a single line, but I doubt there's a significant performance difference.

Here's the relevant bit of the documentation for font-lock-defaults:

HIGHLIGHT should be either MATCH-HIGHLIGHT or MATCH-ANCHORED.

[....]

MATCH-ANCHORED should be of the form:

(MATCHER PRE-MATCH-FORM POST-MATCH-FORM MATCH-HIGHLIGHT ...)

where MATCHER is a regexp to search for or the function name to call to make the search, as for MATCH-HIGHLIGHT above, but with one exception; see below. PRE-MATCH-FORM and POST-MATCH-FORM are evaluated before the first, and after the last, instance MATCH-ANCHORED's MATCHER is used. Therefore they can be used to initialize before, and cleanup after, MATCHER is used. Typically, PRE-MATCH-FORM is used to move to some position relative to the original MATCHER, before starting with MATCH-ANCHORED's MATCHER. POST-MATCH-FORM might be used to move back, before resuming with MATCH-ANCHORED's parent's MATCHER.

The above-mentioned exception is as follows. The limit of the MATCHER search defaults to the end of the line after PRE-MATCH-FORM is evaluated. However, if PRE-MATCH-FORM returns a position greater than the position after PRE-MATCH-FORM is evaluated, that position is used as the limit of the search. It is generally a bad idea to return a position greater than the end of the line, i.e., cause the MATCHER search to span lines.

I always find that I have to read the font-lock documentation about three times before it starts to make sense to me ;-)