Find out the CPU time and memory usage of a slurm job

Question

I suppose it's a pretty trivial question but nevertheless, I'm looking for the (sacct I guess) command that will display the CPU time and memory used by a slurm job ID.

Answer 1

You're right that the sacct command is what you're looking for. The --format switch is the other key element. If you run this command:

sacct -e

you'll get a printout of the different fields that can be used for the --format switch. The details of each field are described in the Job Account Fields section of the man page. For CPU time and memory, CPUTime and MaxRSS are probably what you're looking for. cputimeraw can also be used if you want the number in seconds, as opposed to the usual Slurm time format.

sacct --format="CPUTime,MaxRSS"

Answer 2

sacct is indeed the command to use for finished jobs. For running jobs, you can look at the sstat command.

Answer 3

@aaron.kizmiller is right, sacct is the command to use.

One can fetch all of the following fields by passing them into saact --format="field,field"

Fields:

Account           AdminComment      AllocCPUS         AllocGRES
AllocNodes        AllocTRES         AssocID           AveCPU
AveCPUFreq        AveDiskRead       AveDiskWrite      AvePages
AveRSS            AveVMSize         BlockID           Cluster
Comment           ConsumedEnergy    ConsumedEnergyRaw CPUTime
CPUTimeRAW        DerivedExitCode   Elapsed           ElapsedRaw
Eligible          End               ExitCode          GID
Group             JobID             JobIDRaw          JobName
Layout            MaxDiskRead       MaxDiskReadNode   MaxDiskReadTask
MaxDiskWrite      MaxDiskWriteNode  MaxDiskWriteTask  MaxPages
MaxPagesNode      MaxPagesTask      MaxRSS            MaxRSSNode
MaxRSSTask        MaxVMSize         MaxVMSizeNode     MaxVMSizeTask
McsLabel          MinCPU            MinCPUNode        MinCPUTask
NCPUS             NNodes            NodeList          NTasks
Priority          Partition         QOS               QOSRAW
ReqCPUFreq        ReqCPUFreqMin     ReqCPUFreqMax     ReqCPUFreqGov
ReqCPUS           ReqGRES           ReqMem            ReqNodes
ReqTRES           Reservation       ReservationId     Reserved
ResvCPU           ResvCPURAW        Start             State
Submit            Suspended         SystemCPU         Timelimit
TotalCPU          UID               User              UserCPU
WCKey             WCKeyID           WorkDir

For example, to list all job ids, elapsed time, and max VM size, you can run:

sacct --format='JobID,Elapsed,MaxVMSize'

Answer 4

The other answers all detail formats for output of sacct, which is great for looking at multiple jobs aggregated in a table.

However, sometimes you want to look at a specific job in more detail, so you can tell whether your job efficiently used the allocated resources. For that, seff is very useful. The syntax is simply seff <Jobid>. For example, here's a recent job of mine (that failed):

$ seff 15780625

Job ID: 15780625
Cluster: mycluster
User/Group: myuser/mygroup
State: OUT_OF_MEMORY (exit code 0)
Nodes: 1
Cores per node: 16
CPU Utilized: 12:06:01
CPU Efficiency: 85.35% of 14:10:40 core-walltime
Job Wall-clock time: 00:53:10
Memory Utilized: 1.41 GB
Memory Efficiency: 70.47% of 2.00 GB

Note that the key CPU metric, CPU Utilized, corresponds to the TotalCPU field from sacct, while Memory Utilized corresponds to MaxRSS.