DEV Community

Cover image for Running PyTorch Jobs on Shared GPUs with HTCondor
Haruki Niwa
Haruki Niwa

Posted on AI-assisted

Running PyTorch Jobs on Shared GPUs with HTCondor

Part 2 of 6 in Building and Operating an HTCondor Pool for Research Computing.

Our November 2026 inventory included a GTX 1080 with 8 GiB of memory and a TITAN RTX with 24 GiB. With our sharing configuration, HTCondor can assign two jobs to the same physical GPU. We’ll submit one PyTorch job, inspect which GPU it sees, and check the result of a small matrix multiplication.

Start with a pool that can return the hostname from a remote shell job, as in Part 1. The machine you submit from can have a different Python installation from the worker. The wrapper below records which worker HTCondor selected, and the Python script records which interpreter that worker started.

What one GPU request means in our pool

Our lab’s setup instructions enable GPU sharing on NVIDIA workers with the two settings below. If you followed the optional GPU setup in Part 1, you can go straight to checking what the worker advertises.

use feature : GPUs
GPU_DISCOVERY_EXTRA = -properties -divide 2
Enter fullscreen mode Exit fullscreen mode

With -divide 2, HTCondor lists each detected GPU ID twice and halves its advertised GlobalMemoryMb for matching. For example, a device that discovery reports as 24,576 MiB would advertise 12,288 MiB per share. Two jobs assigned the same card draw from the same VRAM. This setting alone does not enforce a per-job memory limit.

On the access point, run condor_status -gpus -compact to see the advertised model names. On a worker, use the following command to see the physical devices and their full UUIDs. Keep the output with the worker’s hostname so you can look up the driver version for the machine that runs your test.

nvidia-smi --query-gpu=index,uuid,name,driver_version --format=csv
Enter fullscreen mode Exit fullscreen mode

Choose a PyTorch build for the workers

Our lab’s training tutorial uses PyTorch 2.4.1 with CUDA 12.1 and runs a transferred script with Python on shared storage. This article uses the same shared-interpreter setup for a smaller test that exits if CUDA is unavailable.

The GTX 1080 is a Pascal GPU with compute capability 6.1. The TITAN RTX is a Turing GPU with compute capability 7.5. For Linux x86-64, the official PyTorch 2.8.0 CUDA 12.6 wheel includes support for both architectures. The CUDA 12.8 wheel from the same release dropped Pascal support. The 12.6 wheel is therefore a candidate to test on both cards. Each worker still needs a compatible NVIDIA driver. PyTorch 2.8 architecture support · Official wheel installation commands

Before submitting, prepare a dedicated Python environment that every eligible worker can run at the same absolute path. The example uses /shared/envs/pytorch/bin/python. Substitute your environment’s path in both the install command and the job wrapper. If you already have a suitable PyTorch build in that environment, skip the installation step.

/shared/envs/pytorch/bin/python -m pip install torch==2.8.0 --index-url https://download.pytorch.org/whl/cu126
Enter fullscreen mode Exit fullscreen mode

Run a small job that requires CUDA

On the access point, save this as check_gpu.py in your job directory.

import os
import sys

print("Interpreter:", sys.executable, flush=True)
print("CUDA_VISIBLE_DEVICES:", os.environ.get("CUDA_VISIBLE_DEVICES", "<unset>"), flush=True)

import torch

print("PyTorch:", torch.__version__, flush=True)
print("CUDA version used to build PyTorch:", torch.version.cuda, flush=True)
if not torch.cuda.is_available():
    raise SystemExit("CUDA is unavailable to this PyTorch process")

print("Device: cuda:0", torch.cuda.get_device_name(0), flush=True)
print("Compute capability:", torch.cuda.get_device_capability(0), flush=True)
x = torch.ones((128, 128), device="cuda:0")
result = (x @ x)[0, 0].item()
print("Matrix product [0, 0]:", result)
if result != 128.0:
    raise SystemExit(f"Expected 128.0, got {result}")
Enter fullscreen mode Exit fullscreen mode

Save this wrapper as run_gpu.sh. Replace the example path with the absolute path to your environment’s Python on the worker.

#!/bin/sh
printf 'Execution host: %s\n' "$(hostname)"
exec /shared/envs/pytorch/bin/python check_gpu.py
Enter fullscreen mode Exit fullscreen mode

Save gpu.submit in the same directory.

universe                = vanilla
executable              = run_gpu.sh
should_transfer_files   = YES
when_to_transfer_output = ON_EXIT
transfer_input_files    = check_gpu.py

request_cpus   = 1
request_memory = 2 GB
request_gpus   = 1

output = log/gpu.$(ClusterId).$(ProcId).out
error  = log/gpu.$(ClusterId).$(ProcId).err
log    = log/gpu.$(ClusterId).log
queue 1
Enter fullscreen mode Exit fullscreen mode

HTCondor transfers the wrapper and check_gpu.py. The wrapper starts the Python interpreter already available to the worker. Here request_memory requests host RAM, and request_gpus = 1 requests one advertised GPU resource.

To test a specific card model, copy its advertised name from condor_status -gpus -compact and add a require_gpus line before queue 1. For example, if the worker advertises the name shown here, use:

require_gpus = (DeviceName == "NVIDIA GeForce GTX 1080")
Enter fullscreen mode Exit fullscreen mode

A capability test alone would also allow the lab’s TITAN X (Pascal), which has the same compute capability as the GTX 1080. Submit once for the GTX 1080, then replace the name with the TITAN RTX’s advertised name and submit again. Keep the cluster ID for each run and check the model PyTorch prints before recording a pass for that card. For routine jobs, gpus_minimum_capability sets a lower bound if all eligible models are supported by your build. HTCondor GPU submit options

mkdir -p log
chmod +x run_gpu.sh
condor_submit gpu.submit
condor_q -nobatch
Enter fullscreen mode Exit fullscreen mode

Verify the CUDA calculation

From the submission directory on the access point, replace 123 with the cluster ID for the run you want to check. The submit file queues one process, so its job ID ends in .0. Wait up to three minutes for a completion or removal event.

cluster=123
job="$cluster.0"
event_log="log/gpu.$cluster.log"
condor_wait -wait 180 "$event_log" "$job"
Enter fullscreen mode Exit fullscreen mode

If the wait times out or reports an error, use the troubleshooting table below. A zero exit status from condor_wait can also mean the job was removed. Check the job’s own exit status and read its output.

condor_history "$job" -af JobStatus ExitBySignal ExitCode
cat "log/gpu.$job.out"
Enter fullscreen mode Exit fullscreen mode

For a successful run, look for both of the following:

  • The history shows 4 false 0. The job completed, exited normally, and returned exit code zero.
  • Stdout shows Matrix product [0, 0]: 128.0. Each element sums 128 products of ones. Calling .item() reads the selected element back from the GPU as a Python number.

To identify the card, start with Execution host and CUDA_VISIBLE_DEVICES in stdout. HTCondor sets the latter from the job’s assignment. Inside the process, CUDA numbers visible devices starting at zero, so cuda:0 refers to the first device visible to this job. If HTCondor supplied a full GPU UUID or a unique prefix, match it against the UUIDs from nvidia-smi on that worker. A numeric value alone does not give you a physical UUID. HTCondor job environment · CUDA device enumeration

Record the worker’s driver version alongside the PyTorch version, build CUDA version, model and compute capability for each successful run. The 128.0 result verifies this calculation for one job. Demonstrating simultaneous assignment would require two jobs on the same worker with the same physical GPU UUID and overlapping execution times.

Symptom What to check
The wait times out or reports an error Run condor_q "$job" -af JobStatus HoldReason and read recent events with tail -n 20 "$event_log". For an Idle job, run condor_q -better-analyze "$job" and check the GPU conditions. For a Running job, repeat the wait. For a Held job, read the hold reason. If the queue query returns no row, check the history. If the log is missing, check the cluster ID and working directory.
The wrapper prints a hostname, then Python fails to start Read log/gpu.$job.err. On the named worker, check the interpreter path and execute permissions. If Python starts but importing PyTorch fails, stderr should identify the missing package or library.
The script reports that CUDA is unavailable Check CUDA version used to build PyTorch and CUDA_VISIBLE_DEVICES in stdout. For this NVIDIA setup, a build value of None means PyTorch lacks CUDA support. With a CUDA build, check device visibility and the driver on the named worker.
The model is printed but a GPU operation fails Read log/gpu.$job.err. For a kernel-image error, compare the model’s compute capability with the wheel’s supported architectures. For an out-of-memory error, check other processes and free VRAM on the physical card.

Part 3 runs a compiled MATLAB application on a CPU worker. To investigate GPU contention in Part 5, use the worker name and the full GPU UUID once you’ve matched it to the worker’s inventory. The VRAM time series shows free memory across the card at each sample, including the effects of other processes. With the example’s 30-second collection interval, a short job can start and finish between samples.

References

AI assistance was used in drafting and editing this article.

Top comments (0)