From 44828ebd01a0318825ad0ddfe1e7aa317607391b Mon Sep 17 00:00:00 2001 From: Xevion Date: Wed, 26 Jul 2023 17:28:00 -0500 Subject: [PATCH] partial: Race Conditions in Signal Handlers --- ...7-26-race-conditions-in-signal-handlers.md | 117 ++++++++++++++++++ 1 file changed, 117 insertions(+) create mode 100644 _posts/2023-7-26-race-conditions-in-signal-handlers.md diff --git a/_posts/2023-7-26-race-conditions-in-signal-handlers.md b/_posts/2023-7-26-race-conditions-in-signal-handlers.md new file mode 100644 index 0000000..77dad11 --- /dev/null +++ b/_posts/2023-7-26-race-conditions-in-signal-handlers.md @@ -0,0 +1,117 @@ +--- +layout: default +title: Race Conditions in Signal Handlers +date: 2023-07-26 16:08:12 -0500 +tags: tar signals interrupt handler process unix race-condition +_preview_description: Signals offer a unique, low-level way of communicating with processes. But under +certain circumstances, they can kill processes, even when they should work. +--- + +> This article is a deep dive on a classic race condition issue. If you're hoping for an elegant and interesting article +> on how +> I identified a critical vulnerability in `tar`, I'm sorry to say - there's no such vulnerability. + +Signals are a special, but very primitive way for processes to communicate functionality. Signals are useful as they are +a standardized interface available to 99.99% of programs run on UNIX systems (in existence). Interaction can be done +with just the `kill` command. + +While the signals API can be quite bare bones and simple, it's technically much less complex compared to a network +interface, usage of STDIN/STDOUT, a file, or even a shared memory segment. These other options might have a lot more +features, +but none of them are perfectly standardized, completely secure, or simple to use. + +If you're looking to allow basic communication with your program for very specific use cases and don't need complexity +or I/O, signals can be a great way to go. + +## The `tar` command + +> This section is a bit of a tangent, but it's a great example of how signals can be used in practice, as well +> as how I came across this issue. Skip to the next section if you just want to hear the error & solution. + +The `tar` command is a ubiquitous tool for creating and extracting archives. It's a very simple tool, but it's +extremely powerful. It's also a great example of a program that uses signals. + +A couple months ago, I was writing software to help bootstrap embedded devices. The software would use `tar` to extract +a filesystem onto the device's eMMC. Due to the size of the filesystem and the speed of the device, this process could +take some time - I wanted to add a progress bar to confirm that the process was still running & progress was being made. + +Unfortunately, `tar` doesn't emit progress information under normal circumstances, and no alternatives were available +in my language of choice that maintained the speed of `tar`. But looking into the documentation, `tar` could receive +specific, designated signals to [emit progress information][checking-tar-progress] for both archival and extraction +operations. + +By starting `tar` with the `--totals` flag, it would emit a statistic when the operation completes. But to request +information during the operation, a signal must be chosen, like so `tar -x -f archive.tar --totals=SIGUSR1`. + +Emitting a signal can be done with the `kill` command, like so: `kill -USR1 `. This will send the `USR1` signal +to the process with the given PID. The `USR1` signal is a user-defined signal, and is not used by the system. + +And so, my plan was to start a tar process as usual with the `--totals` flag, and then send the `USR1` signal to the +process occasionally to query an extraction operation's progress. In Python, I used the `subprocess` module to start +and manage the process. + +```python +import os +import subprocess +import signal +import time +import sys + +# Define the command to execute +command = ["tar", "-xpf", sys.argv[2], "-C", sys.argv[1], "--totals=SIGUSR1"] + +# Start the subprocess +print(' '.join(command)) +process = subprocess.Popen(command, preexec_fn=os.setsid, stderr=subprocess.PIPE) + +try: + while True: + # Ping the subprocess with SIGUSR1 signal + # NOTWORK: process.send_signal(signal.SIGUSR1) + # NOTWORK: os.killpg(os.getpgid(process.pid), signal.SIGUSR1) + subprocess.Popen(["kill", "-SIGUSR1", str(process.pid)]) + + print(process.stderr.readline().decode("utf-8").strip()) + # print(process.stdout.readline().decode("utf-8").strip()) + + # Wait for a specified interval + time.sleep(1.9) # Adjust the interval as needed + +except KeyboardInterrupt: + # Handle Ctrl+C to gracefully terminate the script + process.terminate() + +# Wait for the subprocess to complete +process.wait() +``` + +You'll notice I have three different ways to send signals shown, but only one of them is working. Moreover, instead +of the signal not working like expected, the signal actually kills the process. When checked the exit code, +one will find that the status code is the same as the signal number, but negated. + +For example, `SIGUSR1` exits with `-10`, `SIGUSR2` exits with `-12`, and `SIGHUP` exits with `-2`. In fact, +when you look into signals, this is the default behavior for processes exited by signals. + +## Signal Handlers Aren't Instant + +To my surprise, the handlers that programs like `tar` use aren't available instantly - so much so that even Python +can send a signal before they're registered. + +I am still not sure as to how signal handlers are implemented - I would've assumed they are static, unchanging, and +registered at program start, but that doesn't seem to be the case - or at least, Python can beat them to the punch. + +Whatever the case, the issue with my implementation is that the signal is sent before the handler is registered, and +the default behavior of the signal takes over. For many signals, this is to terminate the process. + +## How to wait for Signal Handlers + +```TODO``` + +### Credits + +Credit to [Eryk Sun][python-discuss-solution] for explaining the issue and providing an immaculate solution to signal +handlers in Python. + +[python-discuss-solution]: https://discuss.python.org/t/os-kill-signals-not-being-received-correctly-alternative-is-kill-sigusr1-command/26913/6 + +[checking-tar-progress]: https://www.gnu.org/software/tar/manual/html_section/verbose.html \ No newline at end of file