Fine-tuning: `set-process-sentinel` | `set-process-filter` | `start-process`

Question

There are very few examples on the internet involving all three of the functions at issue in this question -- i.e., set-process-sentinel; set-process-filter; and start-process.

I've tried a few different methods of fine-tuning the the processes in an effort to force process number 1 (push) to finish prior to the commencement of process number 2 (push). In all of my attempts, the second process always runs and finishes before I have finished entering my password for process number 1. Process number 2 has a password stored in the osxkeychain.

The first method I tried was with Magit, both synchronous and asynchronous processes. The second method I tried was with using the function while . . . to search the list of remotes in the buffer containing said list. The third attempt is listed below -- it uses a list of remotes that is created at the outset of the function and then mapcars down the list to push with Git.

Any ideas on how to better control process number 1 (push) so that it successfully finishes prior the commencement of process number 2 (push) would be greatly appreciated.

It is not the end of the world that process number 2 starts and finishes too early, but it is a matter of learning how to take control over Emacs processes -- rather than the processes taking control of me.

---

EDIT (April 23, 2014):  Added a doc-string. Revised handling of buffer *REMOTES* -- i.e., kill-local-variable 'git-remote-list and erase-buffer now works correctly by using with-current-buffer ...

---

(defvar git-remote-list nil
"List of remote locations -- e.g., lawlist_remote or github_remote.")
(make-variable-buffer-local 'git-remote-list)

(defvar git-commit-message (format "Committed -- %s" (current-time-string))
"The predetermined Git commit message.")
(make-variable-buffer-local 'git-commit-message)

(defun my-process-filter (proc string)
  (when (string-match "password" string)
    (process-send-string
      proc
      (concat (read-passwd "Password: ") "\n"))))

(defun my-process-sentinel (proc string)
  (when (= 0 (process-exit-status proc))
    (message "Process `%s` has finished." proc)))

(defun stage-commit-push-all ()
"This function does the following:
  * Save the current working buffer if it has been modified.
  * Gather a list of all remotes associated with working directory Git project.
  * Stage all -- `/usr/local/git/bin/git add .`
  * Commit all -- `/usr/local/git/bin/git commit -m [git-commit-message]`
  * Push to all remotes:  `/usr/local/git/bin/git push -v [remote] [current-branch]`
Obtaining the current branch presently requires installation of Magit."
(interactive)
  (when (buffer-modified-p)
    (save-buffer))
  (when (get-buffer "*REMOTES*")
    (with-current-buffer (get-buffer "*REMOTES*")
      (kill-local-variable 'git-remote-list)
      (erase-buffer)))
  (set-process-sentinel
    (start-process
      "list-remotes"
      "*REMOTES*"
      "/usr/local/git/bin/git"
      "remote"
      "-v")
    (lambda (p e) (when (= 0 (process-exit-status p))
      (let* (
          beg
          end
          git-remote-name)
        (with-current-buffer (get-buffer "*REMOTES*")
          (goto-char (point-max))
          (while (re-search-backward "\(push\)" nil t)
            (beginning-of-line 1)
            (setq beg (point))
            (re-search-forward "\t" nil t)
            (setq end (- (point) 1))
            (setq git-remote-name (buffer-substring-no-properties beg end))
            (setq git-remote-list
              (append (cons git-remote-name git-remote-list)))) ))
      (set-process-sentinel
        (start-process
          "stage-all"
          "*OUTPUT*"
          "/usr/local/git/bin/git"
          "add"
          ".")
        (lambda (p e) (when (= 0 (process-exit-status p))
          (set-process-sentinel
            (start-process
              "commit-all"
              "*OUTPUT*"
              "/usr/local/git/bin/git"
              "commit"
              "-m"
              git-commit-message)
            (lambda (p e) (when (= 0 (process-exit-status p))
              (mapcar (lambda (x)
                (let ((proc
                    (start-process
                      "push-process"
                      "*OUTPUT*"
                      "/usr/local/git/bin/git"
                      "push"
                      "-v"
                      (format "%s" x)
                      (magit-get-current-branch))))
                  (set-process-filter proc 'my-process-filter)
                  (set-process-sentinel proc 'my-process-sentinel) ))
                (with-current-buffer (get-buffer "*REMOTES*") git-remote-list)
    )))))))))))

Answer 1

In order to obtain more precise control over set-process-sentinel | set-process-filter | start-process, one may wish to consider incorporating recursive-edit to pause an elisp function while a process is pending -- and either use exit-recursive-edit or (throw 'exit nil) to return control to the elisp function that was paused.

---

EDIT (April 23, 2014): Redacted to create a minimal example incorporating recursive-edit. A working example will be maintained and updated in a related thread: https://stackoverflow.com/a/23178396/2112489

---

(defun my-git-password-process-filter (proc string)
  (when (string-match "password" string)
    (process-send-string
      proc
      (concat (read-passwd "Password: ") "\n"))))

(defun my-process-sentinel (proc string)
  (when (= 0 (process-exit-status proc))
    (message "Process `%s` has finished pushing to `%s`." proc git-remote-name)
    (throw 'exit nil)))

(defun example-using-recursive-edit ()
;;
;; For a more detailed working example, please see this related thread:
;; https://stackoverflow.com/a/23178396/2112489
;;
;; * * * REDACTED TO CREATE MINIMAL WORKING EXAMPLE * * *
;;
  (set-process-sentinel
    (start-process
      "commit-all"
      "*Messages*"
      "/usr/local/git/bin/git"
      "commit"
      "-m"
      git-commit-message)
    (lambda (p e) (when (= 0 (process-exit-status p))
      (mapcar (lambda (git-remote-name)
        (let ((proc
            (start-process
              "push-process"
              "*Messages*"
              "/usr/local/git/bin/git"
              "push"
              "-v"
              (format "%s" git-remote-name)
              (format "%s"
                (with-current-buffer (get-buffer "*REMOTES*")
                  git-branch-name)) )))
          (set-process-filter proc 'my-git-password-process-filter)
          (set-process-sentinel proc 'my-process-sentinel)
          (recursive-edit) ))
        (with-current-buffer (get-buffer "*REMOTES*")
          git-remote-list) )))))

Answer 2

The approach that springs to mind is simply to make each call to start-process dependent on the sentinel for the previous process.

In essence, generate a queue of things you want to do, trigger the processing of the first queue item, and let each sentinel trigger start the process for the next queue item (if any) once its own process has completed.

Answer 3

The following is an example of throwing a result at the end of a function, when that result is dependent upon the process finishing beforehand. As the other answer in this thread indicates, using recursive-edit is helpful -- in this case, it solves the dilemma by preventing the result from being thrown until the process concludes.

This particular example uses a version of ls found in coreutils-8.21 for OSX, which may have some options not available in other versions. This example also contains a creative method to obtain the absolute paths/file-names using start-process, which would not ordinarily permit regexp arguments such as is used to obtain files that are both visible and hidden.

NOTE:  A process-filter does not work reliably to create an exact list (including spaces between ls columns), however, that explanation is beyond the scope of this example. The process output buffer, on the other hand, is reliable. [I may submit a bug report regarding the process filter at some point in the future when I have the time.]

;;; EXAMPLE:  (dolist (x (create-ls-list "~/")) (message "%s" x))
(defun create-ls-list (&optional dir)
(interactive)
  (let* (
      result
      (directory
        (cond
          (dir
            (if (equal dir "/") nil (directory-file-name dir)))
          (t
            (directory-file-name default-directory))))
      (output-buffer "*LS-OUTPUT*")
      (ls-command (concat
        ;;; Some versions of `ls` do not support TIME-STYLE argument.
        "/Users/HOME/.0.data/.0.emacs/.0.macports/bin/gls"
        " "
        "-lhd" ;; listing switches
        " "
        directory ;; `nil` if root "/"
        "/.*" ;; hidden files
        " "
        directory ;; `nil` if root "/"
        "/*" ;; everything except hidden fies
        " "
        "--time-style=\"+%m-%d-%Y %H:%M:%S\""
        " "
        "--group-directories-first")) )
    (when (get-buffer output-buffer)
      (with-current-buffer (get-buffer output-buffer)
        (erase-buffer)))
    (set-process-sentinel
      (start-process
        "ls-process"
        output-buffer
        "/bin/bash"
        "-c"
        ls-command)
      (lambda (p e) (when (= 0 (process-exit-status p))
        (let* (
            (output-buffer (get-buffer "*LS-OUTPUT*"))
            (regexp (concat
              "\\(^[sldrwxt+-]+\\)" ;; 1
              "\\(\s+\\)"           ;; 2
              "\\([0-9]+\\)"        ;; 3
              "\\(\s+\\)"           ;; 4
              "\\([a-zA-Z]+\\)"     ;; 5
              "\\(\s+\\)"           ;; 6
              "\\([a-zA-Z]+\\)"     ;; 7
              "\\(\s+\\)"           ;; 8
              "\\([0-9.kKMGT]+\\)"  ;; 9
              "\\(\s+\\)"           ;; 10
              "\\([0-9-]+\\)"       ;; 11
              "\\(\s+\\)"           ;; 12
              "\\([0-9:]+\\)"       ;; 13
              "\\(\s+\\)"           ;; 14
              "\\(.*$\\)" )))        ;; 15
          (with-current-buffer output-buffer
            (save-excursion
              (goto-char (point-max))
              (while (re-search-backward regexp nil t)
                (let* (
                    (one (match-string 1))
                    (two (match-string 2))
                    (three (match-string 3))
                    (four (match-string 4))
                    (five (match-string 5))
                    (six (match-string 6))
                    (seven (match-string 7))
                    (eight (match-string 8))
                    (nine (match-string 9))
                    (ten (match-string 10))
                    (eleven (match-string 11))
                    (twelve (match-string 12))
                    (thirteen (match-string 13))
                    (fourteen (match-string 14))
                    (fifteen (match-string 15)) )
                  (push
                    (list
                      one two three four five six seven eight nine
                      ten eleven twelve thirteen fourteen fifteen)
                    result)))))
          (throw 'exit nil)))))
    (recursive-edit)
    result))