DEV Community

ByteBridge Talent LLC
ByteBridge Talent LLC

Posted on

Your multiprocessing Loop Won't Stop the App It Launched: Fixing Process(target=f(x)) and os.system

The problem

A learner in an online forum wanted a script that keeps opening an external program (Notepad, in their case) in a loop. They wanted the loop to stop when a key is pressed. They used multiprocessing and os.system, and it did not behave. The script hung, and the stop key did nothing useful.

The replies spotted three separate bugs. We rebuilt each one on Linux with Python 3.13 so you can see them happen, then wrote a fix with tests.

  1. Process(target=f(path)) runs f right away, in the parent, not in a new process.
  2. os.system() starts the app through a shell and waits for it. Calling terminate() on the worker does not reach the app.
  3. The key check ran once, before the loop even began, so it could never stop anything later.

Bug 1: you called the function instead of passing it

import os, time
from multiprocessing import Process

def work(n):
    print("work() running in pid", os.getpid())
    time.sleep(n)

p = Process(target=work(1))   # work(1) runs NOW, then returns None
Enter fullscreen mode Exit fullscreen mode

Python must evaluate work(1) before it can hand the result to Process. So work runs in the parent and blocks it. When it returns, target is None. Our run printed the same pid for the parent and for work(). The Process(...) line took 1.0 seconds, and target was None.

Fix: pass the function and its arguments separately.

p = Process(target=work, args=(1,))
Enter fullscreen mode Exit fullscreen mode

If the loop never ends, as in the original post, the parent stays stuck on that line forever. That is why it looked like the script "hangs".

Bug 2: terminate() kills the worker, not the app

Even with the target fixed, this still leaks:

import os, sys, time
from multiprocessing import Process

APP = f'"{sys.executable}" fake_app.py 60'

def run_forever():
    while True:
        os.system(APP)        # shell -> app; blocks until the app exits

if __name__ == "__main__":
    p = Process(target=run_forever)
    p.start(); time.sleep(1.5)
    p.terminate(); p.join()
Enter fullscreen mode Exit fullscreen mode

In our test, the worker was gone after terminate(), but two processes were still running: the shell that os.system started, and the app under it. We also forced the "spawn" start method on Linux (the docs say it is the default on Windows and macOS). There, the script printed its last line but did not exit until the leftover app closed.

This matches the Python docs. os.system() runs the command "in a subshell" by calling C's system(). And Process.terminate() notes that descendant processes "will not be terminated – they will simply become orphaned." The docs also warn that terminate() skips finally blocks and exit handlers. It can also break a pipe, queue or lock the process was using.

Root cause: your code never holds a handle to the app. os.system hides it behind a shell and only gives you an exit status at the end. You can't signal something you can't see.

The fix: own the app with Popen, stop with an Event

Two changes do most of the work:

  • Start the app with subprocess.Popen([...]), using a list of arguments. There is no shell in between, and you get a handle with poll(), terminate(), kill() and wait().
  • Use an Event as the stop signal. The loop checks it often, and the loop itself closes the app. Nobody has to kill the loop from outside.
"""Relaunch an external app in a loop until asked to stop, then close it cleanly."""
import subprocess
import threading


def stop_app(proc, grace=3.0):
    """Ask the app to exit, wait a bit, then force it."""
    if proc.poll() is not None:
        return proc.returncode
    proc.terminate()                      # SIGTERM on POSIX, TerminateProcess on Windows
    try:
        return proc.wait(timeout=grace)
    except subprocess.TimeoutExpired:
        proc.kill()                       # SIGKILL on POSIX
        return proc.wait()


def run_app_loop(cmd, stop, max_runs=None, poll=0.2):
    """Start cmd, restart it each time it exits, until `stop` (an Event) is set."""
    runs = 0
    while not stop.is_set():
        if max_runs is not None and runs >= max_runs:
            break
        proc = subprocess.Popen(cmd)      # list of args, no shell in between
        runs += 1
        try:
            while proc.poll() is None:    # app still running
                if stop.wait(poll):       # sleeps up to `poll`, wakes early on stop
                    break
        finally:
            stop_app(proc)                # runs on stop, Ctrl+C, or an exception
    return runs


def watch_for_q(stop):
    """Portable key check: type q and press Enter."""
    try:
        while True:
            if input().strip().lower() == "q":
                break
    except EOFError:                      # stdin closed: treat as stop too
        pass
    stop.set()


if __name__ == "__main__":
    import sys
    stop = threading.Event()
    threading.Thread(target=watch_for_q, args=(stop,), daemon=True).start()
    app = [sys.executable, "fake_app.py", "5"]      # e.g. ["notepad.exe"] on Windows
    print("running; type q + Enter to stop", flush=True)
    print("app started", run_app_loop(app, stop), "times")
Enter fullscreen mode Exit fullscreen mode

A few details matter here:

  • stop.wait(poll) works as a sleep that wakes early when the event is set. So the stop is fast, but the loop does not spin the CPU.
  • stop_app() tries a polite stop first, then waits with a timeout, and only then forces it. The finally makes sure the app is closed even on Ctrl+C or an error.
  • The key check runs in its own thread. It loops, so it checks every line you type, not just once. We used input() (type q and press Enter) because it works the same on every OS. Packages that read single key presses have their own platform rules, so check their docs before you use one.

Do you even need multiprocessing?

Often you don't. The external app is already a separate process. Your Python loop spends nearly all its time waiting, so one process with a small thread is enough, as above.

Save the code above as supervisor.py.

If the loop really must live in its own process (for example, a larger program starts it), share a multiprocessing.Event. The docs describe it as "a clone of threading.Event". Then ask the worker to stop instead of terminating it:

"""Same idea when the loop must live in its own process."""
import multiprocessing as mp
from supervisor import run_app_loop


def worker(cmd, stop):
    run_app_loop(cmd, stop)


if __name__ == "__main__":
    import sys, time
    stop = mp.Event()
    p = mp.Process(target=worker, args=([sys.executable, "fake_app.py", "60", "BB_MARK_MP"], stop))
    p.start()
    time.sleep(1.5)
    stop.set()                 # ask, don't terminate()
    p.join(timeout=10)
    if p.is_alive():           # last resort only
        p.terminate(); p.join()
    print("worker exit code:", p.exitcode)
Enter fullscreen mode Exit fullscreen mode

In our run, the worker exited with code 0 and no app process was left. terminate() stays only as a last resort, after a timeout.

Trade-offs to know

  • Grace time vs. speed. A polite stop lets the app save or clean up. A force kill is instant but can lose data. We used 3 seconds. Pick what fits your app.
  • Windows is blunter. The docs say that on Windows, Popen.terminate() calls TerminateProcess(), and kill() is just an alias for it. So there is no polite-then-force step there. If you need a graceful close on Windows, you need an app-specific method. We tested on Linux only. The Windows notes come from the docs, not from our runs.
  • Apps that start their own children. terminate() reaches only the process you started. In our test, a launcher that started a child left that child running after terminate(). On POSIX, starting it with start_new_session=True and then calling os.killpg(p.pid, signal.SIGTERM) cleared the whole group (0 left). That option is POSIX only. On Windows, look at creationflags such as CREATE_NEW_PROCESS_GROUP and test it yourself.
  • Pipes. If you add stdout=PIPE, the docs warn that wait() can deadlock when the app writes a lot of output. Use communicate(), or don't pipe output you don't read.
  • Restart storms. If the app exits at once (a bad path, for example), the loop will relaunch it over and over. Add a short delay or a max_runs limit. Also, log errors instead of a bare except: pass, which hid problems in the original code.

How to verify the fix

Don't trust "it looks closed". Count processes. We gave each test app a unique marker argument and counted matches with pgrep -f. These five tests all passed:

def alive(mark):
    out = subprocess.run(["pgrep", "-f", mark], capture_output=True, text=True).stdout
    return len(out.split())

def test_stop_kills_running_app_fast(self):
    stop = threading.Event()
    cmd = [sys.executable, "fake_app.py", "60", "BB_MARK_T1"]
    th = threading.Thread(target=run_app_loop, args=(cmd, stop)); th.start()
    time.sleep(1.0); self.assertEqual(alive("BB_MARK_T1"), 1)
    t = time.time(); stop.set(); th.join(5)
    self.assertFalse(th.is_alive()); self.assertLess(time.time() - t, 2)
    self.assertEqual(alive("BB_MARK_T1"), 0)
Enter fullscreen mode Exit fullscreen mode
  • Restarts: an app that exits at once was started exactly 3 times with max_runs=3.
  • Fast stop: a 60-second app was closed in under 2 seconds after stop.set(), with 0 left.
  • Force kill: an app that ignores SIGTERM was killed after the 1-second grace time (return code -9).
  • Worker version: the multiprocessing worker exited with code 0, with 0 app processes left.
  • Key stop: typing q stopped a 5-second app early, in under 2 seconds.

fake_app.py is just time.sleep(seconds), standing in for the real program. On Windows, swap pgrep for Task Manager or tasklist to do the same count.

Official sources


ByteBridge Talent LLC is an IT staffing company. We help companies find and hire IT professionals. Website: https://www.bytebridgetalent.com · Email: info@bytebridgetalent.com

Top comments (1)

Collapse
 
acemeridian profile image
Ace Meridian •

I ran your supervisor.py with the functions unchanged (Linux, Python 3.14.4, a 60-second fake_app) and tried an exit your five tests don't cover: the supervisor itself getting SIGTERM, for example kill <pid> or a parent process calling terminate() on it.

SIGINT (Ctrl+C) is fine: KeyboardInterrupt unwinds through the finally, stop_app() runs, 0 apps left. SIGTERM is not. Python installs no handler for it, so the default action ends the process without running finally blocks: the supervisor exited with -15 and the fake_app was still running a second later, orphaned. Same leak as your Bug 2, one level up. The multiprocessing version hits it in the last-resort branch: p.terminate() on the worker gave exitcode -15 and left the app running.

What fixed it in my runs:

import signal

def on_sigterm(signum, frame):
    raise SystemExit(128 + signum)   # unwinds through run_app_loop's finally, like Ctrl+C

signal.signal(signal.SIGTERM, on_sigterm)   # in __main__, before run_app_loop()
Enter fullscreen mode Exit fullscreen mode

With that, SIGTERM exited with 143 and 0 apps were left (calling stop.set() from the handler instead also worked: exit 0, 0 left). SIGKILL can't be caught, so kill -9 still orphaned the app. On Linux, launching it as ["setpriv", "--pdeathsig", "TERM", "--", *cmd] has the kernel send TERM to the app when the supervisor dies: 0 left after kill -9. On Windows a parent's terminate() is TerminateProcess, which no Python handler sees; a Job Object with kill-on-close is the usual answer there, but I only ran Linux.

Does anything besides the keyboard stop the script in your learner's setup, like a parent script or a scheduler?