DEV Community

Cover image for Writing HTCondor DAG Files: Job Dependencies, Variables, and Concurrency Limits
Haruki Niwa
Haruki Niwa

Posted on AI-assisted

Writing HTCondor DAG Files: Job Dependencies, Variables, and Concurrency Limits

Part 4 of 6 in Building and Operating an HTCondor Pool for Research Computing.

Our lab’s GPU tutorial defines three PyTorch runs with different training parameters. They share one submit file, and DAGMan admits one of the three nodes to the queue at a time. In a participant analysis, the final job also needs every participant’s result, even when those results come from different workers.

This walkthrough starts with two Linux jobs and one line of data you can follow from worker to worker. It then applies the same file-transfer rules to the MAT-files from Part 3. The commands target HTCondor 25.0 LTS on the Ubuntu access point from Part 1. The outputs described below are what to check in your own run.

1. Pass a result from A to B

A DAG is a directed graph with no cycles. Each JOB line names a node and gives its HTCondor submit file. A PARENT … CHILD … line makes the child eligible for submission after all its parents succeed. The submit files specify what each job runs and which files it receives.

Create a fresh working directory on the access point and save the three files below in it. Keep the absolute path in project for the later exercises. The workers need /bin/echo and /bin/cat at the paths shown.

mkdir project && cd project
project=$PWD
Enter fullscreen mode Exit fullscreen mode

my.dag

JOB A A.submit
JOB B B.submit
PARENT A CHILD B
Enter fullscreen mode Exit fullscreen mode

A.submit

universe = vanilla
executable = /bin/echo
transfer_executable = False
arguments = sample-001
should_transfer_files = YES
when_to_transfer_output = ON_EXIT
output = A.out
error = A.err
log = A.log
queue 1
Enter fullscreen mode Exit fullscreen mode

B.submit

universe = vanilla
executable = /bin/cat
transfer_executable = False
arguments = A.out
should_transfer_files = YES
transfer_input_files = A.out
when_to_transfer_output = ON_EXIT
output = B.out
error = B.err
log = B.log
queue 1
Enter fullscreen mode Exit fullscreen mode

A writes sample-001 to stdout. HTCondor returns stdout to A.out on the access point. After A succeeds, DAGMan submits B, and HTCondor stages A.out in B’s scratch directory. The dependency controls when B can be submitted. The transfer_input_files line gives B the data, so the workers can use separate filesystems. HTCondor file transfer

Submit the DAG once. The cluster number printed here belongs to the DAGMan management job. DAGMan will submit A and B itself.

condor_submit_dag my.dag
Enter fullscreen mode Exit fullscreen mode

While it is running, use the cluster number from that submission in place of 123. The first command reports workflow progress. The second lists only the currently queued node jobs for this DAG. A node waiting on a parent has no queued job yet.

dag_id=123
htcondor dag status "$dag_id"
condor_q -constraint "DAGManJobId == $dag_id" -af DAGNodeName ClusterId ProcId JobStatus
Enter fullscreen mode Exit fullscreen mode

To verify the run, wait on the management job’s event log (.dagman.log). A zero return from condor_wait can also mean removal. Check the final status in the diagnostic log (.dagman.out) and compare both outputs.

if condor_wait -wait 300 my.dag.dagman.log "$dag_id.0"; then
    tail -n 1 my.dag.dagman.out
    if tail -n 1 my.dag.dagman.out | grep -Eq 'EXITING WITH STATUS 0[[:space:]]*$' &&
       printf '%s\n' sample-001 | cmp - A.out &&
       cmp A.out B.out; then
        echo 'Workflow and output verified'
    fi
fi
Enter fullscreen mode Exit fullscreen mode

If the wait times out, repeat the status query with the same ID. For a held node (JobStatus 5), use its own cluster and process ID to read HoldReason, for example condor_q 456.0 -af HoldReason. After a short DAG has already left the queue, its diagnostic log and output files remain available. DAGMan logs and job attributes · condor_wait

2. Choose the unit of work

Give each node a unique name within the DAG. Several nodes can refer to the same submit file. An external node submit file has one queue statement. With queue 1, the node submits one job. With queue 10, it submits one cluster of ten jobs under the default configuration. If one of those jobs fails, DAGMan normally treats the group as failed and removes the remaining jobs in that node. A POST script can affect the final node outcome, as shown in Section 5.

A DAG line such as RETRY Train 2 allows up to two additional attempts for a node named Train. Each attempt repeats the whole node, including its PRE script, jobs, and POST script if defined. Give each participant a separate node if you want to retry participants independently. Node submit files · Node success and retries

For a preprocessing step A, parallel analyses B and C, and an aggregation step D, declare four JOB lines and add these dependencies:

PARENT A CHILD B C
PARENT B C CHILD D
Enter fullscreen mode Exit fullscreen mode

After A succeeds, B and C can be submitted independently. Available resources and submission limits determine whether they overlap. D waits for both parents. If B fails and C succeeds, D stays blocked.

3. Pass per-node arguments with VARS

Two nodes can share one submit file. Create a separate directory for this exercise and save the following three files there.

mkdir "$project/vars" && cd "$project/vars"
Enter fullscreen mode Exit fullscreen mode

runs.dag

JOB RUN1 run_job.submit
JOB RUN2 run_job.submit
VARS RUN1 idx="1" alpha="0.1"
VARS RUN2 idx="2" alpha="0.5"
Enter fullscreen mode Exit fullscreen mode

run_job.submit

universe = vanilla
executable = /usr/bin/python3
transfer_executable = False
arguments = train.py --idx $(idx) --alpha $(alpha)
should_transfer_files = YES
transfer_input_files = train.py
when_to_transfer_output = ON_EXIT
log = logs/run_$(idx).$(ClusterId).log
output = logs/run_$(idx).$(ClusterId).$(ProcId).out
error = logs/run_$(idx).$(ClusterId).$(ProcId).err
queue 1
Enter fullscreen mode Exit fullscreen mode

train.py prints its arguments so you can check the expansion without installing a training package.

import argparse

parser = argparse.ArgumentParser()
parser.add_argument("--idx", type=int, required=True)
parser.add_argument("--alpha", type=float, required=True)
args = parser.parse_args()
print(f"idx={args.idx} alpha={args.alpha}")
Enter fullscreen mode Exit fullscreen mode
mkdir -p logs
condor_submit_dag runs.dag
Enter fullscreen mode Exit fullscreen mode

DAGMan supplies idx and alpha as submit macros. HTCondor transfers the Python script, and the worker supplies /usr/bin/python3. After this DAG succeeds, logs/run_1.*.out should contain idx=1 alpha=0.1, and logs/run_2.*.out should contain idx=2 alpha=0.5. The index identifies the parameter set, and the cluster and process IDs identify the submitted job. DAGMan VARS

4. Limit submitted GPU nodes with CATEGORY and MAXJOBS

Our lab guide gives three GPU jobs different batch sizes through VARS. Its GPU submit file queues one process per node. The following shortened template uses the same scheduling rule. Supply your GPU submit file, request the required GPU resources there, and use $(batch_size) in the training arguments.

JOB G1 gpu_job.submit
JOB G2 gpu_job.submit
JOB G3 gpu_job.submit
VARS G1 batch_size="1024"
VARS G2 batch_size="512"
VARS G3 batch_size="256"

CATEGORY G1 GPUJOB
CATEGORY G2 GPUJOB
CATEGORY G3 GPUJOB
MAXJOBS GPUJOB 1
Enter fullscreen mode Exit fullscreen mode

MAXJOBS GPUJOB 1 allows one submitted node cluster in this category in this DAG. An idle or held job can keep its cluster in the queue and delay the next node. Consider two versions of the same submit file:

In each GPU node’s submit file What one category slot allows
queue 1 and request_gpus = 1 One job from these three nodes can be queued at a time.
queue 4 and request_gpus = 1 The admitted cluster contains four jobs. All four can run if the pool can satisfy their requests.

Other DAGs can submit GPU jobs at the same time. In Part 2’s pool, one advertised GPU resource can be a share of a physical card. This category limit therefore provides no guarantee of exclusive access to a physical GPU. DAGMan category throttling

5. Check a parent’s output with a POST script

Suppose A exits zero but writes an empty file. B’s cat can also exit zero on that file. Run a check on the access point after A’s job finishes and before B becomes eligible.

mkdir "$project/checks" &&
    cp "$project"/{my.dag,A.submit,B.submit} "$project/checks/" &&
    cd "$project/checks"
Enter fullscreen mode Exit fullscreen mode

Append these lines to this copy of my.dag:

SCRIPT DEBUG verify.log STDERR POST A ./verify_a.sh $NODE $RETURN
SCRIPT PRE B /usr/bin/test -r A.out
Enter fullscreen mode Exit fullscreen mode

Save the POST script as verify_a.sh:

#!/bin/sh
node=$1
job_status=$2

if [ "$job_status" != "0" ]; then
    printf '%s: job returned %s\n' "$node" "$job_status" >&2
    exit 1
fi

if ! printf '%s\n' sample-001 | cmp -s - A.out; then
    printf '%s: A.out is missing or contains the wrong sample ID\n' "$node" >&2
    exit 1
fi
Enter fullscreen mode Exit fullscreen mode
chmod +x verify_a.sh
condor_submit_dag my.dag
Enter fullscreen mode Exit fullscreen mode

Both scripts run on the access point. When a POST script runs, its exit status determines whether the node succeeds. A cleanup command that exits zero can mask a failed job. This script checks $RETURN first, so it also rejects a job killed by a signal. Only then does it check the bytes in A.out. The DEBUG clause captures the script’s stderr in verify.log.

B’s PRE script checks that the returned file is still readable immediately before submission. Under the default settings, a failed PRE script marks B failed and skips its jobs and POST script. DAGMan scripts and return values

6. Pause a workflow or resume after a failure

To pause a running workflow, use the cluster number of the active DAGMan job as dag_id. A hold and a halt have different effects on work in progress. Managing a running DAG

Operation What happens How to continue
condor_hold "$dag_id" Already queued node jobs continue. Running PRE and POST scripts are killed. DAGMan submits no new jobs while held. Inspect any partial files those scripts were writing, then use condor_release "$dag_id".
htcondor dag halt "$dag_id" New PRE scripts and node submissions stop. Already submitted jobs continue, and POST scripts can still run. Once work drains, DAGMan writes a Rescue DAG and exits. Use htcondor dag resume "$dag_id" while DAGMan is still running. After it exits, resubmit the original DAG from its run directory.

Fail B, repair it, and check that only B reruns

Start with fresh copies of the definitions from Section 1. This time B will receive A.out but ask cat to open a different file. That makes the failure occur inside B, after its inputs have arrived.

mkdir "$project/rescue-demo" &&
    cp "$project"/{my.dag,A.submit,B.submit} "$project/rescue-demo/" &&
    cd "$project/rescue-demo"
sed -i 's/^arguments = A\.out$/arguments = missing.txt/' B.submit
condor_submit_dag my.dag
Enter fullscreen mode Exit fullscreen mode

Replace 234 with this submission’s DAGMan cluster number. Wait for it to finish, then inspect the failure before repairing anything.

failed_id=234
if condor_wait -wait 300 my.dag.dagman.log "$failed_id.0"; then
    tail -n 1 my.dag.dagman.out
    cat B.err
    grep -E '^DONE[[:space:]]+A([[:space:]]|$)' my.dag.rescue001
    printf '%s\n' sample-001 | cmp - A.out && echo 'A.out is ready to reuse'
fi
Enter fullscreen mode Exit fullscreen mode

In this new directory, expect a nonzero final DAGMan status, an error naming missing.txt, and a DONE A entry in my.dag.rescue001. That entry tells DAGMan to skip A. It does not establish that A’s output is still usable, which is why the last command checks the file itself. If the wait times out, inspect the active jobs before proceeding.

Once you have confirmed those results, restore B’s working definition and resubmit the original DAG in the same directory. By default, DAGMan reads it together with the highest-numbered applicable Rescue DAG.

cp "$project/B.submit" B.submit
condor_submit_dag my.dag
Enter fullscreen mode Exit fullscreen mode

Replace 345 with the new DAGMan cluster number. The history query selects only node jobs belonging to this second submission.

rescue_id=345
if condor_wait -wait 300 my.dag.dagman.log "$rescue_id.0"; then
    tail -n 1 my.dag.dagman.out
    condor_history -constraint "DAGManJobId == $rescue_id" -af DAGNodeName ExitBySignal ExitCode
    printf '%s\n' sample-001 | cmp - A.out && cmp A.out B.out && echo 'Output verified'
fi
Enter fullscreen mode Exit fullscreen mode

Expect EXITING WITH STATUS 0, a single history row (B false 0), and Output verified. A’s original data reached B even though the second DAGMan job never submitted A.

A rescue can reset the retry count for unfinished nodes. With HTCondor 25.0’s default DAGMAN_RESET_RETRIES_UPON_RESCUE = True, an unfinished node gets its configured retry allowance again on rescue. A site that sets it to False deducts retries already used. This exercise defines no RETRY lines. If completed parents’ inputs or code change, use a fresh run directory with fresh result paths so their outputs are regenerated. Rescue DAGs · Retry configuration

7. Hand four MAT-files to one merge job

Part 3’s compiled application accepts a matrix size and writes result.mat. To check the handoff, run four copies with predictable output names. The MERGE node below uses sha256sum to read each returned file. This verifies that all four files reached the child. Part 3 shows how to verify their numerical contents in MATLAB.

In a fresh directory on the access point, copy Part 3’s window_overlap, run_window_overlap.sh, run_job.sh, and matlab.sub. Copy matlab.sub to run.submit and keep the worker requirement and arguments = 64 from your successful Part 3 run. Keep when_to_transfer_output = ON_SUCCESS, success_exit_code = 0, and transfer_output_files = result.mat. Change the output remap in run.submit to:

transfer_output_remaps = "result.mat=results/$(subject).mat"
Enter fullscreen mode Exit fullscreen mode

Save the DAG as participants.dag:

JOB RUN1 run.submit
JOB RUN2 run.submit
JOB RUN3 run.submit
JOB RUN4 run.submit
VARS RUN1 subject="sub-001"
VARS RUN2 subject="sub-002"
VARS RUN3 subject="sub-003"
VARS RUN4 subject="sub-004"

JOB MERGE merge.submit
PARENT RUN1 RUN2 RUN3 RUN4 CHILD MERGE
Enter fullscreen mode Exit fullscreen mode

Each parent writes result.mat in its own worker scratch directory. The remap places the files under four distinct names on the access point. The merge job can therefore declare its inputs before any parent runs.

Four parent jobs return sub-001.mat through sub-004.mat to the access point. MERGE receives all four files after the parents succeed.

Save the following as merge.submit. Its worker needs /usr/bin/sha256sum. The input list uses paths on the access point. The command arguments use the basenames that HTCondor stages in the child’s working directory by default. HTCondor input sandbox layout

universe = vanilla
executable = /usr/bin/sha256sum
transfer_executable = False
should_transfer_files = YES
transfer_input_files = results/sub-001.mat, results/sub-002.mat, results/sub-003.mat, results/sub-004.mat
arguments = sub-001.mat sub-002.mat sub-003.mat sub-004.mat
when_to_transfer_output = ON_EXIT
output = results/manifest.sha256
error = log/merge.err
log = log/merge.log
queue 1
Enter fullscreen mode Exit fullscreen mode
mkdir -p log results
chmod u+x run_job.sh
condor_submit_dag participants.dag
Enter fullscreen mode Exit fullscreen mode

After this DAG exits with status zero, compare the child’s checksums against the four files on the access point:

(cd results && sha256sum --check manifest.sha256)
Enter fullscreen mode Exit fullscreen mode

Expect one OK for each filename from sub-001.mat through sub-004.mat. Each digest compares the child’s copy with the corresponding file on the access point. A file with incorrect matrix values could still pass this check. All four parents used the same matrix size. For a participant analysis, pass each parent its own input data and replace the checksum command with your aggregation program. If preprocessing produces those inputs, add a PREP job and the dependency PARENT PREP CHILD RUN1 RUN2 RUN3 RUN4. Reuse the run directory for a rescue and start each independent study run in a new one.

When a result is missing, follow one filename from the parent’s scratch directory to the access point and then to the child. If the job has yet to start, Part 5 shows how to inspect the pool’s queues and allocations in Grafana.

AI assistance was used in translating, organizing, and editing this article.

Top comments (0)