Add notebook with HTCondor commands

%% Cell type:markdown id:00431e7a-c5fa-4831-a875-a824797006ac tags:

# Introduction

This notebook is a compilation of commands to perform operations with HTCondor through the command line.

## Documentation

* Predefined functions that can be applied to classads: [doc](https://htcondor.readthedocs.io/en/latest/classads/classad-builtin-functions.html)

%% Cell type:markdown id:6777812d-93b5-4f00-bc5b-d98eaa269a5f tags:

# Job querying

This section applies to the functions used to query running jobs (`condor_q`) and job history (`condor_history`)

%% Cell type:markdown id:49fbc530-7105-4b29-8e81-9a2ee57cc1b5 tags:

### command help

%% Cell type:code id:8eb3922b-8743-4598-b7e6-f5f1df4f5262 tags:

``` python

!condor_q -h

```

%% Output

    Usage: condor_q [general-opts] [restriction-list] [output-opts | analyze-opts]

        [general-opts] are

    	-global			 Query all Schedulers in this pool

    	-schedd-constraint	 Query all Schedulers matching this constraint

    	-submitter <submitter>	 Get queue of specific submitter

    	-name <name>		 Name of Scheduler

    	-pool <host>		 Use host as the central manager to query

    	-jobads[:<form>] <file>	 Read queue from a file of job ClassAds

    	           where <form> is one of:

    	       auto    default, guess the format from reading the input stream

    	       long    The traditional -long form

    	       xml     XML form, the same as -xml

    	       json    JSON classad form, the same as -json

    	       new     'new' classad form without newlines

    	-userlog <file>		 Read queue from a user log file

        [restriction-list] each restriction may be one of

    	<cluster>		 Get information about specific cluster

    	<cluster>.<proc>	 Get information about specific job

    	<owner>			 Information about jobs owned by <owner>

    	-factory		 Get information about late materialization job factories

    	-autocluster		 Get information about the SCHEDD's autoclusters

    	-constraint <expr>	 Get information about jobs that match <expr>

    	-unmatchable		 Get information about jobs that do not match any machines

    	-allusers		 Consider jobs from all users

        [output-opts] are

    	-limit <num>		 Limit the number of results to <num>

    	-cputime		 Display CPU_TIME instead of RUN_TIME

    	-currentrun		 Display times only for current run

    	-debug			 Display debugging info to console

    	-dag			 Sort DAG jobs under their DAGMan

    	-expert			 Display shorter error messages

    	-grid			 Get information about grid jobs

    	-goodput		 Display job goodput statistics

    	-help [Universe|State]	 Display this screen, JobUniverses, JobStates

    	-hold			 Get information about jobs on hold

    	-io			 Display information regarding I/O

    	-batch			 Display DAGs or batches of similar jobs as a single line

    	-nobatch		 Display one line per job, rather than one line per batch

    	-idle			 Get information about idle jobs

    	-run			 Get information about running jobs

    	-totals			 Display only job totals

    	-stream-results 	 Produce output as jobs are fetched

    	-version		 Print the HTCondor version and exit

    	-wide[:<width>]		 Don't truncate data to fit in 80 columns.

    				 Truncates to console width or <width> argument.

    	-autoformat[:jlhVr,tng] <attr> [<attr2> [...]]

    	-af[:jlhVr,tng] <attr> [attr2 [...]]

    	    Print attr(s) with automatic formatting

    	    the [jlhVr,tng] options modify the formatting

    	        j   Display Job id

    	        l   attribute labels

    	        h   attribute column headings

    	        V   %V formatting (string values are quoted)

    	        r   %r formatting (raw/unparsed values)

    	        ,   comma after each value

    	        t   tab before each value (default is space)

    	        n   newline after each value

    	        g   newline between ClassAds, no space before values

    	    use -af:h to get tabular values with headings

    	    use -af:lrng to get -long equivalent format

    	-format <fmt> <attr>	 Print attribute attr using format fmt

    	-print-format <file>	 Use <file> to set display attributes and formatting

    				 (experimental, see htcondor-wiki for more information)

    	-long[:<form>]		 Display entire ClassAds in <form> format

    				 See -jobads for <form> choices

    	-xml			 Display entire ClassAds in XML form

    	-json			 Display entire ClassAds in JSON form

    	-attributes X,Y,...	 Attributes to show in -xml, -json, and -long

        [analyze-opts] are

    	-analyze[:<qual>]	 Perform matchmaking analysis on jobs

    	-better-analyze[:<qual>] Perform more detailed match analysis

    	    <qual> is a comma separated list of one or more of

    	    priority	Consider user priority during analysis

    	    summary	Show a one-line summary for each job or machine

    	    reverse	Analyze machines rather than jobs

    	-machine <name>		 Machine name or slot name for analysis

    	-mconstraint <expr>	 Machine constraint for analysis

    	-slotads[:<form>] <file> Read Machine ClassAds for analysis from <file>

    				 <file> can be the output of condor_status -long

    	-userprios <file>	 Read user priorities for analysis from <file>

    				 <file> can be the output of condor_userprio -l

    	-nouserprios		 Don't consider user priority during analysis (default)

    	-reverse-analyze	 Analyze Machine requirements against jobs

    	-verbose		 Show progress and machine names in results

        Only information about jobs owned by the current user will be returned.

    This default is overridden when the restriction list has usernames and/or

    job ids, when the -submitter or -allusers arguments are specified, or

    when the current user is a queue superuser

%% Cell type:markdown id:06c00100-7e5e-4dd9-ab25-e77406914006 tags:

## Job filtering

### Regular expression matching

%% Cell type:code id:2a7199a2-5f61-441d-9b30-c9008086048f tags:

``` python

!condor_history cpardoar -constraint 'regexp(".*dask_worker.*", Arguments) is False' -limit 10

```

%% Output

     ID     OWNER          SUBMITTED   RUN_TIME     ST COMPLETED   CMD

    290882.0 cpardoar       10/23 21:40   4+00:00:13 X              /data/magnesia/

    292050.0 cpardoar       10/25 10:18   0+07:58:29 X              /data/jupyter/s

    291558.0 cpardoar       10/24 08:41   0+12:50:54 X              /data/jupyter/s

    289337.0 cpardoar       10/23 09:23   0+16:39:20 X              /data/jupyter/s

    290116.0 cpardoar       10/23 16:16   0+00:06:14 C  10/23 16:23 /data/magnesia/

    289515.0 cpardoar       10/23 16:02   0+00:09:03 C  10/23 16:12 /data/magnesia/

    289133.0 cpardoar       10/22 11:30   0+08:17:26 X              /data/jupyter/s

    289016.0 cpardoar       10/22 10:18   0+01:11:13 X              /data/jupyter/s

    288442.0 cpardoar       10/21 08:49   0+12:28:14 X              /data/jupyter/s

    287697.0 cpardoar       10/19 18:52   1+23:55:17 C  10/21 18:48 /data/magnesia/

%% Cell type:markdown id:f25921fd-7246-46b0-9e83-3b8fb2255103 tags:

## Output formatting

%% Cell type:markdown id:9b946e33-1d59-45a7-9d9f-d84eb1f8b54f tags:

### Long output format

%% Cell type:code id:39268d2f-5fd2-45c3-901e-09a2f7055b3f tags:

``` python

!condor_history 290882.0 -limit 1 -long

```

%% Output

    AccountingGroup = strcat(AcctGroup,".",AcctGroupUser)

    AcctGroup = strcat(ifThenElse(flavour =?= "boost","group_HIGHPRIO",ifThenElse(regexp("neissner",Owner),"group_ASTRO",ifThenElse(regexp("codefly",Owner),"group_HIGHPRIO",ifThenElse(regexp("pic",OwnerGroup),"group_HIGHPRIO",ifThenElse(regexp("atlas",OwnerGroup),"group_AT3",ifThenElse(regexp("at3",OwnerGroup),"group_AT3",ifThenElse(regexp("paus",OwnerGroup),"group_ASTRO",ifThenElse(regexp("mice",OwnerGroup),"group_ASTRO",ifThenElse(regexp("des",OwnerGroup),"group_ASTRO",ifThenElse(regexp("desi",OwnerGroup),"group_ASTRO",ifThenElse(regexp("magic",OwnerGroup),"group_MAGIC",ifThenElse(regexp("euclid",OwnerGroup),"group_EUCLID",ifThenElse(regexp("ciematdune",OwnerGroup),"group_DUNE","group_OTHER"))))))))))))))

    AcctGroupUser = strcat(ifThenElse(flavour =?= undefined || (flavour != "short" && flavour != "long" && flavour != "xlong" && flavour != "boost"),"medium",flavour),".",OwnerGroup,ifThenElse(RequestCpus > 1,"_mcore","_score"),".",Owner)

    ActivationDuration = 345612

    ActivationExecutionDuration = 345612

    ActivationSetupDuration = 0

    ActivationTeardownDuration = 0

    Arguments = ""

    AssignedGPUs = "GPU-4a16d2e1"

    BlockReadKbytes = 0

    BlockReads = 0

    BlockWriteKbytes = 0

    BlockWrites = 0

    BytesRecvd = 0.0

    BytesSent = 10712856.0

    ClusterId = 290882

    Cmd = "/data/magnesia/common/paper_pardo_araujo_etal_2024/exp_13_ensemble5comp_10rounds_allrounds1k_retrain_from_scratch_7param_updated_atnf/htcondor_submit/wrapper.sh"

    CommittedSlotTime = 0

    CommittedSuspensionTime = 0

    CommittedTime = 0

    ConcurrencyLimits = strcat(Owner,", ",OwnerGroup,", ",flavour)

    CondorPlatform = "$CondorPlatform: x86_64_AlmaLinux9 $"

    CondorVersion = "$CondorVersion: 23.0.10 2024-05-09 BuildID: 731952 PackageID: 23.0.10-1 $"

    CpusProvisioned = 1

    CpusUsage = 0.1035460983223704

    CumulativeRemoteSysCpu = 5474.0

    CumulativeRemoteUserCpu = 26695.0

    CumulativeSlotTime = 345613.0

    CumulativeSuspensionTime = 0

    CurrentHosts = 0

    DiskProvisioned = 42577764

    DiskUsage = 12500

    DiskUsage_RAW = 10499

    EnteredCurrentStatus = 1730058113

    Environment = ""

    Err = "/data/magnesia/common/paper_pardo_araujo_etal_2024/exp_13_ensemble5comp_10rounds_allrounds1k_retrain_from_scratch_7param_updated_atnf/htcondor_output/0-error.txt"

    ExecutableSize = 1

    ExecutableSize_RAW = 1

    ExitBySignal = false

    ExitCode = 0

    ExitStatus = 0

    FileSystemDomain = "pic.es"

    GPUsAverageUsage = 0.003265946433177767

    GPUsMemoryUsage = 4884.0

    GPUsProvisioned = 1

    GlobalJobId = "submit02.pic.es#290882.0#1729712458"

    HoldWallTime = (time() - EnteredCurrentStatus) > JobWallTimeLimit

    HoldWallTimeReason = strcat("Your job walltime ",RemoteWallClockTime - CumulativeSuspensionTime,"s exceeded the Maximum Walltime ",JobWallTimeLimit,"s")

    HoldWallTimeResult = (RemoteWallClockTime - CumulativeSuspensionTime) > JobWallTimeLimit

    ImageSize = 4500000

    ImageSize_RAW = 4457948

    In = "/dev/null"

    Iwd = "/data/magnesia/common/paper_pardo_araujo_etal_2024/exp_13_ensemble5comp_10rounds_allrounds1k_retrain_from_scratch_7param_updated_atnf"

    JobCurrentFinishTransferInputDate = 1729712500

    JobCurrentFinishTransferOutputDate = -1

    JobCurrentStartDate = 1729712500

    JobCurrentStartExecutingDate = 1729712500

    JobCurrentStartTransferInputDate = 1729712500

    JobCurrentStartTransferOutputDate = 1730058112

    JobFinishedHookDone = 1730079810

    JobLeaseDuration = 2400

    JobMemoryLimit = ifThenElse(InteractiveJob =?= true || WN_property =?= "tdmem",3 * RequestMemory,1.5 * RequestMemory)

    JobNotification = 0

    JobPrio = -100

    JobRunCount = 1

    JobStartDate = 1729712500

    JobStatus = 3

    JobSubmitMethod = 0

    JobUniverse = 5

    JobWallTimeLimit = 345600

    LastHoldReason = "Your job walltime 3.456120000000000E+05s exceeded the Maximum Walltime 345600s"

    LastHoldReasonCode = 3

    LastHoldReasonSubCode = 0

    LastJobLeaseRenewal = 1730058112

    LastJobStatus = 5

    LastMatchTime = 1729712500

    LastPublicClaimId = "<192.168.101.82:9618?addrs=192.168.101.82-9618+[2001-67c-1148-301--82]-9618&alias=gpu05.pic.es&noUDP&sock=startd_2973_cf27>#1725599738#22885#..."

    LastRejMatchReason = "no match found "

    LastRejMatchTime = 1729712478

    LastRemoteHost = "slot2_4@gpu05.pic.es"

    LastRemoteWallClockTime = 345613.0

    LastSuspensionTime = 0

    LastVacateTime = 1730058113

    LeaveJobInQueue = false

    MATCH_EXP_SubmitPICScaling = "1.6258"

    MATCH_PICScaling = 1.6258

    MachineAttrCpus0 = 1

    MachineAttrSlotWeight0 = 1

    MaxHosts = 1

    MemoryProvisioned = 32768

    MemoryUsage = ((ResidentSetSize + 1023) / 1024)

    MinHosts = 1

    MyType = "Job"

    NumCkpts = 0

    NumCkpts_RAW = 0

    NumHolds = 3

    NumHoldsByReason = [ JobPolicy = 3 ]

    NumJobCompletions = 0

    NumJobMatches = 1

    NumJobStarts = 1

    NumRestarts = 0

    NumShadowStarts = 1

    NumSystemHolds = 0

    OrigMaxHosts = 1

    Out = "/data/magnesia/common/paper_pardo_araujo_etal_2024/exp_13_ensemble5comp_10rounds_allrounds1k_retrain_from_scratch_7param_updated_atnf/htcondor_output/0-out.txt"

    Owner = "cpardoar"

    OwnerGroup = "magnesia"

    PeriodicHold = (JobStatus == 2 && HoldWallTime)

    PeriodicHoldReason = ifThenElse(HoldWallTimeResult == true,HoldWallTimeReason,undefined)

    ProcId = 0

    QDate = 1729712458

    Rank = 0.0

    RecentBlockReadKbytes = 0

    RecentBlockReads = 0

    RecentBlockWriteKbytes = 0

    RecentBlockWrites = 0

    RecentStatsLifetimeStarter = 1200

    RemoteIwd = "/data/magnesia/common/paper_pardo_araujo_etal_2024/exp_13_ensemble5comp_10rounds_allrounds1k_retrain_from_scratch_7param_updated_atnf"

    RemoteSysCpu = 5474.0

    RemoteUserCpu = 26695.0

    RemoteWallClockTime = 345613.0

    RemoveReason = "The system macro SYSTEM_PERIODIC_REMOVE expression '(JobStatus == 5 && (time() - EnteredCurrentStatus) > 21600) || (JobStatus == 4 && (time() - EnteredCurrentStatus) > 7200)' evaluated to TRUE"

    RequestCpus = 1

    RequestDisk = 16777216

    RequestGPUs = 1

    RequestMemory = 32768

    Requirements = (TARGET.WN_property == ifThenElse(MY.WN_property =?= undefined,"default",MY.WN_property)) && (TARGET.Arch == "X86_64") && (TARGET.OpSys == "LINUX") && (TARGET.Disk >= RequestDisk) && (TARGET.Memory >= RequestMemory) && (TARGET.GPUs >= RequestGPUs) && ((TARGET.FileSystemDomain == MY.FileSystemDomain) || (TARGET.HasFileTransfer))

    ResidentSetSize = 4500000

    ResidentSetSize_RAW = 4407184

    ScratchDirFileCount = 23

    ShouldTransferFiles = "YES"

    SpoolOnEvict = false

    SpooledOutputFiles = ""

    StartdPrincipal = "execute-side@matchsession/192.168.101.82"

    StatsLifetimeStarter = 345612

    StreamErr = false

    StreamOut = false

    SubmitHost = "gpu05.pic.es"

    SubmitPICScaling = "$$(PICScaling:1)"

    TargetType = "Machine"

    TotalSubmitProcs = 1

    TotalSuspensions = 0

    TransferExecutable = false

    TransferIn = false

    TransferInFinished = 1729712500

    TransferInputSizeMB = 0

    TransferInputStats = [  ]

    TransferOutFinished = 1730058112

    TransferOutStarted = 1730058112

    TransferOutput = ""

    TransferOutputStats = [ CedarFilesCountTotal = 2; CedarFilesCountLastRun = 2 ]

    User = "cpardoar@pic.es"

    UserLog = "/data/magnesia/common/paper_pardo_araujo_etal_2024/exp_13_ensemble5comp_10rounds_allrounds1k_retrain_from_scratch_7param_updated_atnf/htcondor_output/0-log.txt"

    WhenToTransferOutput = "ON_EXIT_OR_EVICT"

    flavour = "long"

%% Cell type:markdown id:3aadbf70-44b3-4c60-bca4-deab84b4ebc7 tags:

### Timestamp formatting

%% Cell type:code id:b3346a29-141e-4591-a891-d95a0a70581c tags:

``` python

!condor_history cpardoar \

-limit 10 -af ClusterId Cmd Arguments \

'formatTime(JobStartDate, "%Y-%m-%dT%H:%M:%S")' # output formatting

```

%% Output

    293508 /bin/sh -c /data/magnesia/scratch/conda/envs/pop_syn/bin/python' '-m' 'distributed.cli.dask_worker' 'tcp://192.168.101.82:43325' '--name' 'HTCondorCluster-139' '--nthreads' '1' '--memory-limit' '1.86GiB' '--nanny' '--death-timeout' '60s' '--nthreads' '1 2024-10-28T09:12:33

    293512 /bin/sh -c /data/magnesia/scratch/conda/envs/pop_syn/bin/python' '-m' 'distributed.cli.dask_worker' 'tcp://192.168.101.82:43325' '--name' 'HTCondorCluster-541' '--nthreads' '1' '--memory-limit' '1.86GiB' '--nanny' '--death-timeout' '60s' '--nthreads' '1 2024-10-28T09:12:33

    293474 /bin/sh -c /data/magnesia/scratch/conda/envs/pop_syn/bin/python' '-m' 'distributed.cli.dask_worker' 'tcp://192.168.101.82:43325' '--name' 'HTCondorCluster-326' '--nthreads' '1' '--memory-limit' '1.86GiB' '--nanny' '--death-timeout' '60s' '--nthreads' '1 2024-10-28T09:11:45

    293510 /bin/sh -c /data/magnesia/scratch/conda/envs/pop_syn/bin/python' '-m' 'distributed.cli.dask_worker' 'tcp://192.168.101.82:43325' '--name' 'HTCondorCluster-328' '--nthreads' '1' '--memory-limit' '1.86GiB' '--nanny' '--death-timeout' '60s' '--nthreads' '1 2024-10-28T09:12:33

    293521 /bin/sh -c /data/magnesia/scratch/conda/envs/pop_syn/bin/python' '-m' 'distributed.cli.dask_worker' 'tcp://192.168.101.82:43325' '--name' 'HTCondorCluster-29' '--nthreads' '1' '--memory-limit' '1.86GiB' '--nanny' '--death-timeout' '60s' '--nthreads' '1 2024-10-28T09:12:33

    293524 /bin/sh -c /data/magnesia/scratch/conda/envs/pop_syn/bin/python' '-m' 'distributed.cli.dask_worker' 'tcp://192.168.101.82:43325' '--name' 'HTCondorCluster-520' '--nthreads' '1' '--memory-limit' '1.86GiB' '--nanny' '--death-timeout' '60s' '--nthreads' '1 2024-10-28T09:12:33

    293504 /bin/sh -c /data/magnesia/scratch/conda/envs/pop_syn/bin/python' '-m' 'distributed.cli.dask_worker' 'tcp://192.168.101.82:43325' '--name' 'HTCondorCluster-185' '--nthreads' '1' '--memory-limit' '1.86GiB' '--nanny' '--death-timeout' '60s' '--nthreads' '1 2024-10-28T09:12:33

    293523 /bin/sh -c /data/magnesia/scratch/conda/envs/pop_syn/bin/python' '-m' 'distributed.cli.dask_worker' 'tcp://192.168.101.82:43325' '--name' 'HTCondorCluster-310' '--nthreads' '1' '--memory-limit' '1.86GiB' '--nanny' '--death-timeout' '60s' '--nthreads' '1 2024-10-28T09:12:33

    293519 /bin/sh -c /data/magnesia/scratch/conda/envs/pop_syn/bin/python' '-m' 'distributed.cli.dask_worker' 'tcp://192.168.101.82:43325' '--name' 'HTCondorCluster-270' '--nthreads' '1' '--memory-limit' '1.86GiB' '--nanny' '--death-timeout' '60s' '--nthreads' '1 2024-10-28T09:12:33

    293518 /bin/sh -c /data/magnesia/scratch/conda/envs/pop_syn/bin/python' '-m' 'distributed.cli.dask_worker' 'tcp://192.168.101.82:43325' '--name' 'HTCondorCluster-394' '--nthreads' '1' '--memory-limit' '1.86GiB' '--nanny' '--death-timeout' '60s' '--nthreads' '1 2024-10-28T09:12:33

%% Cell type:markdown id:be6c0a6f-f756-42c3-8220-05fa90e6aa62 tags:

# Job submission

## Setting job requirements

### CPU model

Avoid / Choose specific CPU models: choose AMD or Intel, avoid old CPU models

    condor_submit /path/to/sumbit/file.sub -requirements 'CPU_MODEL =!= "Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz"'

### Host

Avoid / Choose nodes:

* Go to a regular CPU node, avoiding those in immersion cooling==tdsXXX.pic.es

    condor_submit -interactive requirements='regexp("td\d", Name)'

%% Cell type:code id:16740d31-69b7-41ef-8fce-1bf3ce10a61e tags:

``` python

```