🖥️ HPC Quick Start Guide

1. Introduction

High-Performance Computing (HPC) enables researchers to solve complex computational problems that would be impractical on regular computers. The IISER Mohali HPC cluster provides shared computing resources for the research community.

2. HPC Basics

What is a Node?

A node is an individual computer within the cluster. Think of each node as a separate, powerful workstation. The cluster consists of multiple nodes connected via a high-speed network.

┌─────────────────────────────────────────────────────────────────┐
│                         HPC CLUSTER                             │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│   ┌──────────┐    ┌──────────┐         ┌──────────┐             │
│   │  Login   │    │  Login   │         │  Master  │             │
│   │  Node 1  │    │  Node 2  │         │  Nodes   │             │
│   └────┬─────┘    └────┬─────┘         └────┬─────┘             │
│        │               │                    │                   │
│        └───────────────┴────────────────────┘                   │
│                        │                                        │
│            ┌───────────┴───────────┐                            │
│            │    High-Speed Network │                            │
│            └───────────┬───────────┘                            │
│                        │                                        │
│   ┌────────────────────┼────────────────────┐                   │
│   │                    │                    │                   │
│   ▼                    ▼                    ▼                   │
│ ┌──────────┐      ┌──────────┐        ┌──────────┐              │
│ │ Compute  │      │ Compute  │  ...   │ Compute  │              │
│ │ Node     │      │ Node     │        │ Node     │              │
│ │ (gpc1)   │      │ (gpc2)   │        │ (gpc32)  │              │
│ │          │      │          │        │          │              │
│ │ 52 cores │      │ 52 cores │        │ 52 cores │              │
│ └──────────┘      └──────────┘        └──────────┘              │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘
                

What is a Core?

A core (also called CPU core or processor) is an individual processing unit within a node. Each node contains multiple cores that can work independently or together.

#PBS -l nodes=1:ppn=1

Cluster Architecture Summary

3. Resource Ethics & Responsibilities

❌ What NOT to Do — RESOURCE CRIMES

✅ Best Practices

How to Check if Your Code Uses Multiple Cores

Before requesting multiple cores, TEST your program to see actual CPU usage:

# Step 1: Get an interactive session
qsub -I -l nodes=1:ppn=4 -q short

# Step 2: Once on the compute node, run your program in background
cd /path/to/your/project
./your_program &

# Step 3: Check CPU usage with 'top'
top -u $USER

# Step 4: Look at %CPU column:
#   ~100%  = using 1 core   → use ppn=1
#   ~200%  = using 2 cores  → use ppn=2
#   ~400%  = using 4 cores  → use ppn=4
#   ~5200% = using all 52   → use ppn=52

# Step 5: If %CPU is ~100%, YOUR CODE IS NOT PARALLEL!
# Use ppn=1 only!

4. Job Queues

Jobs are submitted to queues based on their expected runtime and resource requirements. Each queue has different time limits, core limits, and priorities.

CPU Queues

GPU Queues

Check Queue Details

# View detailed information about all queues
qstat -Qf

# Example output (partial):
Queue: long
    queue_type = Execution
    total_jobs = 48
    state_count = Transit:0 Queued:1 Held:12 Waiting:0 Running:35 Exiting:0
    resources_max.ncpus = 416
    resources_max.walltime = 1080:00:00
    max_user_run = 100
    max_user_res.ncpus = 100
    enabled = True
    started = True

5. Writing PBS Job Scripts

Basic Structure

A PBS job script is a shell script with special #PBS directives that tell the scheduler what resources you need.

#!/bin/bash

#===============================================
# PBS DIRECTIVES - Resource requests
#===============================================
#PBS -N my_job_name          # Job name (appears in qstat)
#PBS -l nodes=1:ppn=1        # 1 node, 1 core (for serial jobs!)
#PBS -l walltime=08:00:00    # Maximum runtime: 8 hours
#PBS -q default              # Queue name
#PBS -o output.log           # Standard output file
#PBS -e error.log            # Standard error file

#===============================================
# CHANGE TO WORKING DIRECTORY
#===============================================
cd $PBS_O_WORKDIR

#===============================================
# LOAD REQUIRED MODULES
#===============================================
module load anaconda3

#===============================================
# RUN YOUR PROGRAM
#===============================================
python my_script.py

#PBS -l nodes=1:ppn=1

PBS Directives Explained

Setting Walltime

Walltime is the maximum time your job is allowed to run. Format: HH:MM:SS

# Examples of walltime settings

#PBS -l walltime=01:00:00      # 1 hour
#PBS -l walltime=08:00:00      # 8 hours (max for 'default' queue)
#PBS -l walltime=72:00:00      # 72 hours / 3 days (max for 'short' queue)
#PBS -l walltime=168:00:00     # 168 hours (7 days)
#PBS -l walltime=720:00:00     # 720 hours (30 days)
#PBS -l walltime=1080:00:00    # 1080 hours / 45 days (max for 'long' queue)

Log Files Configuration

# Method 1: Separate output and error files
#PBS -o output.log
#PBS -e error.log

# Method 2: Combined output and error
#PBS -o combined.log
#PBS -j oe

# Method 3: With full path (recommended)
#PBS -o /persistent/data1/username/project/logs/job_output.log
#PBS -e /persistent/data1/username/project/logs/job_error.log

# Method 4: Include job ID in filename (in your script)
exec > "$PBS_O_WORKDIR/output_$PBS_JOBID.log" 2>&1

Targeting Specific Nodes

# RECOMMENDED: Let scheduler choose automatically
#PBS -l nodes=1:ppn=1

# Target a specific node (use only if necessary)
#PBS -l nodes=gpc30:ppn=4

# Request multiple specific nodes
#PBS -l nodes=gpc30:ppn=4+gpc31:ppn=4

# Request GPU nodes
#PBS -l nodes=gpu1:ppn=4

6. Submitting Jobs

Basic Job Submission

# Submit a job script
qsub job.sh

# Output example:
430105.iisermhpc1

Interactive Jobs

Interactive jobs give you a shell on a compute node for testing and debugging.

# Basic interactive job (1 core)
qsub -I -l nodes=1:ppn=1 -q default

# Interactive job with specific walltime
qsub -I -l nodes=1:ppn=4 -l walltime=02:00:00 -q short

# Interactive job on specific node (if needed)
qsub -I -l nodes=gpc30:ppn=1 -q long

Deleting Jobs

# Delete a specific job
qdel 430105.iisermhpc1

# Delete multiple jobs
qdel 430105.iisermhpc1 430106.iisermhpc1

# Delete all your jobs (be careful!)
qdel $(qstat -u $USER | grep $USER | awk '{print $1}')

7. Monitoring Jobs

Check Your Jobs

# View all your jobs
qstat -u $USER

# Example:
qstat -u ms21080

# Sample output:
Job ID            Name       User      Time  S  Queue
----------------  ---------  --------  ----  -  ------
430035.iisermhpc1 simulation ms21080  03:52  R  long
430036.iisermhpc1 test_job   ms21080  02:45  R  default
430045.iisermhpc1 analysis   ms21080  00:00  Q  short
430103.iisermhpc1 e04        ms21080  00:00  H  long

# Status codes:
# R = Running
# Q = Queued (waiting to start)
# H = Held (job has been held, check why)
# E = Exiting
# C = Completed

Detailed Job Information

# Get detailed information about a specific job
qstat -f 430035.iisermhpc1

# Example output:
Job Id: 430104.iisermhpc1
    Job_Name = e04
    Job_Owner = ms21080@login1
    job_state = H
    queue = long
    server = iisermhpc1
    ...
    Resource_List.nodes = gpc25:ppn=2
    Resource_List.walltime = 1080:00:00
    ...
    comment = job held, too many failed attempts to run
    run_count = 21
    Exit_status = -3
    ...

# Key fields to look for:
# - job_state: Current status (R, Q, H, etc.)
# - exec_host: Which node(s) the job is running on
# - resources_used: CPU time, memory, walltime used
# - Resource_List: What was requested
# - comment: Error messages or hold reasons
# - Exit_status: Exit code (0 = success, non-zero = error)
# - run_count: How many times job tried to run

View All Jobs in Queue

# View all jobs in the cluster
qstat

# View all queues with their status
qstat -Q

# View detailed queue information
qstat -Qf

8. Checking Available Resources

Before submitting jobs (especially to specific nodes), you should check what resources are available. This helps you choose the right queue and avoid problematic nodes.

Check Specific Node Status

# View status of a specific node
pbsnodes gpc25

# Example output:
gpc25
     Mom = gpc25
     ntype = PBS
     state = free
     pcpus = 52
     resources_available.arch = linux
     resources_available.host = gpc25
     resources_available.mem = 394860832kb
     resources_available.ncpus = 52
     resources_available.vnode = gpc25
     resources_assigned.accelerator_memory = 0kb
     resources_assigned.hbmem = 0kb
     resources_assigned.mem = 0kb
     resources_assigned.naccelerators = 0
     resources_assigned.ncpus = 0
     resources_assigned.vmem = 0kb
     queue = long
     resv_enable = True
     sharing = default_shared
     last_state_change_time = Thu Jan 29 00:43:49 2026
     last_used_time = Fri Jan  9 12:07:53 2026

# Key fields:
# state = free          → Node is available
# state = job-exclusive → All cores in use
# state = offline       → Node is not available
# state = down          → Node has problems
# pcpus = 52            → Total cores on node
# resources_assigned.ncpus = 0  → Currently used cores
# queue = long          → Which queue this node belongs to

Check All Nodes

# View status of all nodes
pbsnodes -a

Check Free Cores in Each Queue

Use this script to find which nodes have free cores in a specific queue. This is very useful before submitting jobs!

Script for Checking Free Cores (Long Queue Example):

pbsnodes -a | awk '
/^[a-z]/ {node=$1}
/queue = long/ {inqueue=1}
/pcpus/ {if(inqueue) total=$3}
/resources_assigned.ncpus/ {
  if(inqueue){
    used=$3;
    free=total-used;
    printf "%-8s : %3d free cores\n", node, free;
    sum+=free;
    if(free==total) fullfree++;
    inqueue=0
  }}
END {
  print "------------------------";
  print "Total free cores  =", sum;
  print "Fully free nodes  =", fullfree;
}'

Example Output:

gpc27    :  46 free cores
gpc28    :  20 free cores
gpc29    :  12 free cores
gpc30    :  49 free cores
gpc31    :  52 free cores
gpc32    :  20 free cores
gpc25    :  52 free cores
------------------------
Total free cores  = 251
Fully free nodes  = 2

For Other Queues (Change the queue name):

# For 'short' queue - change "long" to "short":
pbsnodes -a | awk '
/^[a-z]/ {node=$1}
/queue = short/ {inqueue=1}
/pcpus/ {if(inqueue) total=$3}
/resources_assigned.ncpus/ {
  if(inqueue){
    used=$3;
    free=total-used;
    printf "%-8s : %3d free cores\n", node, free;
    sum+=free;
    if(free==total) fullfree++;
    inqueue=0
  }}
END {
  print "------------------------";
  print "Total free cores  =", sum;
  print "Fully free nodes  =", fullfree;
}'

# For 'default' queue - change to: /queue = default/
# For 'infinity' queue - change to: /queue = infinity/

Add This Function to Your ~/.bashrc (Recommended)

# Add this to your ~/.bashrc file for easy access
checkfree() {
    local queue=${1:-long}
    echo "=== Free cores in '$queue' queue ==="
    pbsnodes -a | awk -v q="$queue" '
    /^[a-z]/ {node=$1}
    /queue =/ {if($3==q) inqueue=1}
    /pcpus/ {if(inqueue) total=$3}
    /resources_assigned.ncpus/ {
      if(inqueue){
        used=$3;
        free=total-used;
        if(free>0) printf "%-8s : %3d free cores\n", node, free;
        sum+=free;
        if(free==total) fullfree++;
        inqueue=0
      }}
    END {
      print "------------------------";
      print "Total free cores  =", sum;
      print "Fully free nodes  =", fullfree+0;
    }'
}

# After adding to ~/.bashrc, reload it:
source ~/.bashrc

# Now you can use:
checkfree long
checkfree short
checkfree default
checkfree infinity
checkfree gpushort
checkfree gpulong

            

# Option 1: Let scheduler choose (SAFEST)
#PBS -l nodes=1:ppn=1

# Option 2: Target a known working node
#PBS -l nodes=gpc31:ppn=1

9. Complete Examples

Example 1: Simple Python Job (Serial - Single Core)

#!/bin/bash
#PBS -N python_analysis
#PBS -l nodes=1:ppn=1           # ONLY 1 CORE for serial job!
#PBS -l walltime=04:00:00
#PBS -q default
#PBS -o python_out.log
#PBS -e python_err.log

cd $PBS_O_WORKDIR

module load anaconda3

echo "Job started at: $(date)"
echo "Running on node: $(hostname)"

python analysis.py

echo "Job finished at: $(date)"

Example 2: Multi-threaded Job (OpenMP)

#!/bin/bash
#PBS -N openmp_sim
#PBS -l nodes=1:ppn=16          # 16 cores on 1 node
#PBS -l walltime=48:00:00
#PBS -q short
#PBS -o omp_output.log
#PBS -e omp_error.log

cd $PBS_O_WORKDIR

# Set OpenMP threads to match requested cores
export OMP_NUM_THREADS=16

echo "Using $OMP_NUM_THREADS threads"
./my_openmp_program

Example 3: MPI Job (Multiple Nodes)

#!/bin/bash
#PBS -N mpi_simulation
#PBS -l nodes=4:ppn=20          # 4 nodes, 20 cores each = 80 total
#PBS -l walltime=72:00:00
#PBS -q short
#PBS -o mpi_output.log
#PBS -e mpi_error.log

cd $PBS_O_WORKDIR

module load openmpi-4.1.0

# Calculate total processes
NPROCS=$(wc -l < $PBS_NODEFILE)
echo "Running on $NPROCS processors"

mpirun -np $NPROCS ./my_mpi_program

Example 4: Geant4 Simulation

#!/bin/bash
#PBS -N geant4_sim
#PBS -l nodes=1:ppn=2
#PBS -l walltime=168:00:00
#PBS -q long
#PBS -o geant4_out.log
#PBS -e geant4_err.log

cd $PBS_O_WORKDIR

# Load Geant4
module load codes/geant4/11.1

# Set Geant4 data paths
export G4ENSDFSTATEDATA=/gscratch/apps/root/geant4/install/share/Geant4/data/G4ENSDFSTATE2.3
export G4LEVELGAMMADATA=/gscratch/apps/root/geant4/install/share/Geant4/data/PhotonEvaporation5.7
export G4LEDATA=/gscratch/apps/root/geant4/install/share/Geant4/data/G4EMLOW8.2
export G4PARTICLEXSDATA=/gscratch/apps/root/geant4/install/share/Geant4/data/G4PARTICLEXS4.0

# Record timing
START=$(date +%s)
echo "Started at: $(date)"

./sim run.mac

END=$(date +%s)
echo "Finished at: $(date)"
echo "Duration: $((END-START)) seconds"

Example 5: Targeting a Specific Working Node

#!/bin/bash
# Use this when you've checked available nodes and want to avoid a problematic one
# First run: checkfree long (see Section 8)
# Then choose a working node from the output

#PBS -N my_simulation
#PBS -l nodes=gpc31:ppn=1       # Targeting gpc31 specifically
#PBS -l walltime=720:00:00
#PBS -q long
#PBS -o output.log
#PBS -e error.log

cd $PBS_O_WORKDIR

echo "Running on node: $(hostname)"
./my_program

10. Troubleshooting

Job Stuck in Queue (Q status)

Job Held (H status)

# Check why job is held
qstat -f JOB_ID | grep -E "(comment|run_count|Exit_status)"

# Example output:
comment = job held, too many failed attempts to run
run_count = 21
Exit_status = -3

# This means the node has issues! Steps to fix:
# 1. Delete the held job
qdel JOB_ID

# 2. Check which nodes are free (see Section 8)
# 3. Either let scheduler choose or pick a different node
# 4. Resubmit
qsub job.sh

Job Failed (Exit_status ≠ 0)

# Check exit status
qstat -f JOB_ID | grep Exit_status

# Common exit codes:
# 0    = Success
# 1    = General error in your program
# -3   = Job couldn't start (node/environment issue)
# 137  = Killed (memory limit exceeded or SIGKILL)
# 265  = Walltime exceeded

# Check error log
cat error.log

Node Shows "Free" But Job Keeps Failing

# Check the node status
pbsnodes gpc25

# Even if state = free, the node might have issues!
# Check last_used_time - if it's old, node might be problematic

# Solution: Use a different node or let scheduler choose
#PBS -l nodes=1:ppn=1           # Let scheduler choose
# OR
#PBS -l nodes=gpc30:ppn=1       # Target known working node

11. Quick Reference Commands

Job Submission & Control

Job Monitoring

Node & Resource Information

Module Management

PBS Environment Variables

Final Tips