Previously, we learned how to send some input to an external process via its stdin and receive its output from its stdout. The process, however, was simple and very reliable. In the real world, an external process can take its time (especially if it uses a database or a network to operate), die or even hang, and it’s good to support such cases.
Let’s start with creating a very simple program which can simulate such an unreliable process. Here is a short Node.js script which does what I want. Notice the readline module which is great for purposes like that. Also, the check for Math.random() being less than a half and calling sleep if it is means that half of the time, the last part of output will include the prompt and half the time the prompt will effectively come in a seperate batch.
#!/bin/env node
import readline from 'node:readline';
import {setTimeout as sleep} from 'node:timers/promises';
const output = JSON.stringify({a: 1, b: [2, 3], c: {d: 4, e: 5}}, null, 4).split('\n');
async function print_lines_slowly(output) {
for (const line of output) {
await sleep(400 * Math.random());
console.log(line);
}
}
async function main() {
const rl = readline.createInterface({
input: process.stdin,
output: process.stdout,
terminal: false,
});
console.log('commands: json, bye, part, die, hang');
rl.prompt();
rl.on('line', async(line) => {
if (line === 'json') {
await print_lines_slowly(output);
if (Math.random() < 0.5) {
await sleep(400 * Math.random());
}
rl.prompt();
} else if (line === 'bye') {
rl.close();
console.log('Bye!');
return;
} else if (line === 'part') {
await print_lines_slowly(output.slice(0, 4));
if (Math.random() < 0.5) {
await sleep(400 * Math.random());
}
rl.prompt();
} else if (line === 'die') {
await print_lines_slowly(output.slice(0, 4));
process.exit(1);
} else if (line === 'hang') {
await print_lines_slowly(output.slice(0, 4));
rl.close();
setInterval(() => {}, 1 << 30);
} else {
console.log('command unrecognized');
if (Math.random() < 0.5) {
await sleep(400 * Math.random());
}
rl.prompt();
}
});
}
main();
Now that we have this program, let’s write some Elisp to drive it. We’ll reuse the code from the previous part, but we’ll have to modify it rather heavily. It has to deal with several cases, and also handle the prompt. Luckily for us, the prompt can never be mistaken for part of the output, so we can only have these cases:
- the output is a valid JSON plus the prompt, or
- the output is the word “Bye!” and the process exits cleanly, or
- the output is a partial JSON plus the prompt, or
- the output is a partial JSON and the process dies, or
- the output is a partial JSON and the process hangs, or
- the output is the phrase “command unrecognized” plus the prompt.
It is fully conceivable to have a process which can do one or two other things (for example, print a valid response and die then), but the above cases cover enough for me right now.
Note that if we interacted with something with more free-form output (like a shell), where normal output can be identical to a prompt, things would probably get pretty complicated, and frankly, I would be a bit afraid to even look for support for that in (for example) Comint mode – it is well possible there is some support for that, and if I have to choose between diving into that or preserving my sanity, the choice is obvious.
Well, I lied. Or rather, had a change of mind. It dawned on me that it is really easy to check if Comint mode has support like this. I fired M-x shell and said
echo -n "$ " ; sleep 2; echo "now I'm done!"
and immediately felt better for not wanting to support cases like this – neither does Comint mode! It turned out that he above trick actually made it behave as the output from my code was a prompt. But then I noticed that so does
echo -n "and... " ; sleep 2; echo "now I'm done!"
and I realized that it is not even obvious what it could mean to “support cases like this” – after all, what the “correct” behavior would be?
Anyway, here is the plan. Instead of the process filter checking whether the output is a valid, complete JSON, it will just insert the output and look for the prompt – if it is not present, it’ll just wait. And if it is, we’ll check for valid JSON in the case the external program is buggy. At the same time, we are going to detect if the process died and restart it then (using a process sentinel). This approach will cover all of our scenarios except the “hanging” one. For that one, let’s set up a timer and just restart the process if it does not output anything for a specified amount of time.
So, let’s get coding!
(defvar wonky-process--process nil
"The wonky process.")
(defvar wonky-process--auto-restart-p t
"Non-nil means restart the wonky process after it dies.")
(defun wonky-process--sentinel (process event)
"Restart the wonky process if it is dead."
(when (and wonky-process--auto-restart-p
(not (process-live-p wonky-process--process)))
(message "Wonky process died, restarting.")
(wonky-start-process)))
(defun wonky-start-process ()
"Start `wonky.js' in the background unless it's already started."
(interactive)
(unless (process-live-p wonky-process--process)
(setq wonky-process--auto-restart-p t)
(setq wonky-process--process
(start-process "wonky"
(get-buffer-create "*wonky*")
"wonky.js"))
(set-marker-insertion-type (process-mark wonky-process--process) t)
(setq wonky-process--output-start
(copy-marker
(process-mark wonky-process--process)))
(set-process-filter wonky-process--process
#'wonky-process-filter)
(set-process-sentinel wonky-process--process
#'wonky-process--sentinel)))
(defun wonky-stop-process ()
"Stop the wonky process and do not restart it."
(interactive)
(setq wonky-process--auto-restart-p nil)
(kill-process wonky-process--process))
(defun wonky-process--start-countdown ()
"Start countdown to restart the wonky process when in case it hangs."
(wonky-process--stop-countdown)
(setq wonky-process--timeout-timer
(run-with-timer wonky-process-timeout
nil
#'wonky-process--restart-process)))
(defun wonky-process--stop-countdown ()
"Stop the countdown started by `wonky-process--start-countdown'."
(when (timerp wonky-process--timeout-timer)
(cancel-timer wonky-process--timeout-timer)))
(defun wonky-send-input (wonky-input)
"Send INPUT to the wonky process."
(interactive "sinput: \n")
(if (process-live-p wonky-process--process)
(with-current-buffer (process-buffer wonky-process--process)
(goto-char (process-mark wonky-process--process))
(insert wonky-input "\n")
(setq wonky-process--output-start (point-marker))
(process-send-string wonky-process--process
(concat wonky-input "\n"))
(wonky-process--start-countdown))
(user-error "wonky process is not alive")))
(defvar wonky-process--output-start nil
"The place the current output should start.")
(defcustom wonky-process-timeout 2
"The time (in seconds) to wait for output from `wonky.js'.")
(defvar wonky-process--timeout-timer nil
"The timer used to restart the wonky process when it times out.")
(defun wonky-process--restart-process ()
"Restart the wonky process.
It does so by just killing the process; the actual restarting is handled
by the sentinel."
(when (processp wonky-process--process)
(kill-process wonky-process--process)))
(defun wonky-process-filter (process output)
"Insert OUTPUT to the buffer of PROCESS.
Also, message the user."
(let ((buffer (process-buffer process)))
(when (buffer-live-p buffer)
(with-current-buffer buffer
(save-excursion
(goto-char (process-mark process))
(insert output))
(save-excursion
(goto-char wonky-process--output-start)
(if-let* ((output-end
(and (re-search-forward "^> $" nil t)
(match-beginning 0))))
(progn (goto-char wonky-process--output-start)
(condition-case error
(unless (looking-at "commands:")
(message "%s" (json-parse-buffer)))
(json-parse-error
(message "invalid JSON received: %s"
(buffer-substring-no-properties
wonky-process--output-start
(1- output-end)))))
(setq wonky-process--output-start
(1+ (match-end 0)))
(setq mark wonky-process--output-start)
(wonky-process--stop-countdown))
(wonky-process--start-countdown)))))))
As you can see, this code, while not extremely complex, is not trivial, either. It is still not ideal – for example, if the “wonky” process dies in the middle of producing an output, that partial output is lost (and is not even displayed as “invalid JSON”). Even worse, the same would happen if it died after writing the complete JSON, but before displaying the prompt. It would probably be not very difficult to fix that, but let’s be honest – it’s really an edge-case. Still, it seems to handle the most important cases pretty well: it can parse the results even as they don’t come all at once, it restarts the process if it dies, it can handle the process displaying invalid JSON and prompting again, and it even restarts the process when it hangs (after waiting for some time). It seems it’s all I need! And in two weeks I’m going to show a real tool which does something actually useful and is driven via readline. so we’re getting closer to applying all the knowledge we’ve gathered so far.
.
