Someone finally approved your account. You have a username, a hostname, and a documentation page with forty links on it. You log in, type the command that works on your laptop, and either nothing happens or a system administrator emails you about it an hour later.
This is the most common place new users get stuck, and it isn't a skills problem. A cluster works differently from a personal computer in a few specific ways, and nobody tends to explain those ways in order. Here they are, with the smallest possible first job at the end.
What a cluster actually is
Picture a room full of ordinary computers, a few hundred of them, wired together and sharing one filesystem. Each one is called a node. You never touch most of them directly. Instead you connect to one or two special machines called login nodes, and from there you ask a program called the scheduler to run your work on the others.
That indirection is the whole idea. Dozens of people are using the same machines, so a piece of software has to referee: it holds requests in a queue, decides who gets what and when, and starts your work as soon as the resources you asked for are free. On most academic systems that referee is Slurm. You'll also meet HTCondor, especially on systems built for many small independent jobs, and older systems may run PBS or Grid Engine. The vocabulary differs, the idea doesn't.
The login node is not where you compute
This is the mistake that gets a polite email. Login nodes are shared by everyone on the system, and they exist for editing files, moving data, compiling code, and submitting jobs. When you run a heavy analysis directly on one, you slow down every other person who's trying to log in, and most systems will kill the process automatically.
The fix isn't complicated. Anything real goes through the scheduler, in one of two ways. A batch job is a script you hand off; it waits in the queue, runs without you, and writes its output to a file. An interactive job gives you a shell on a compute node so you can poke at things by hand, which is useful while you're debugging. Both are one command. The habit of using them is what separates a smooth first month from a frustrating one.
If you remember one thing: log in on the login node, compute somewhere else. Everything below is detail on how to ask.
Your software probably isn't installed the way you expect
You can't just install things system-wide, and the version of Python sitting at /usr/bin/python3 is usually old on purpose, because the operating system depends on it. Clusters solve this with an environment module system: hundreds of versions of hundreds of programs sit on disk, inactive, until you load the one you want.
module avail lists what's there. module load python/3.11 puts that version on your path for the session. module list shows what you've loaded. The important part is that a module you load by hand on the login node does not follow you into your job, so the load command has to go inside your submit script too.
For anything with a dependency stack, a language-level environment on top of the module is worth the ten minutes: a conda environment, a Python venv, or an R library folder in your own space. If the system supports containers (Apptainer, formerly Singularity, is the usual one), a container is even better, because the same image runs on your laptop, on your campus cluster, and on a national system without you rebuilding anything.
There is more than one place to put files
Every cluster gives you at least two storage areas, and treating them as interchangeable is how people lose work.
- Home. Small, backed up, and meant for code, scripts, and configuration. Not for datasets. Quotas here are tight and jobs that write output to home tend to hit them at 3 a.m.
- Scratch or work. Large, fast, and usually not backed up. Often files older than some number of days are deleted automatically. This is where your job reads and writes while it runs.
- Project or archive space. Some systems add a third tier for data you need to keep. Ask, because the answer differs everywhere.
The rule that follows: results you care about get copied out of scratch to somewhere durable as part of your workflow, not whenever you remember. Purge policies don't send warnings, and "I'll move it later" is the most expensive sentence in research computing.
Asking for resources honestly
When you submit a job you tell the scheduler four things: how many CPU cores, how much memory, how long, and sometimes which partition or queue. New users get this wrong in both directions.
Ask for too little time and the scheduler kills your job at the limit, mid-run, with nothing saved. Ask for far too much of everything and you sit in the queue for days, because a request for 32 cores and 48 hours has to wait for a big enough gap while a request for 2 cores and 2 hours slides into the next opening. Small, accurate requests start sooner. That surprises people.
You won't know your numbers the first time, and you're not supposed to. Run something deliberately tiny, one input file, a short time limit, then look at what it actually used. Slurm's seff <jobid> reports the CPU and memory efficiency of a finished job, and it's the fastest way to learn that the analysis you reserved 64 GB for peaked at 3. Size the real run from that.
One more thing worth knowing early: asking for more cores does not make a program faster unless the program was written to use them. A single-threaded R script given 16 cores uses one and wastes fifteen. Check the documentation for whatever you're running before you scale the request.
A submit script, start to finish
A batch job is a shell script with scheduler directives at the top. In Slurm they're the lines beginning with #SBATCH. The scheduler reads them, everything else runs on the compute node.
#!/bin/bash
#SBATCH --job-name=test-run
#SBATCH --time=00:20:00
#SBATCH --cpus-per-task=1
#SBATCH --mem=4G
#SBATCH --output=logs/test-%j.out
#SBATCH --error=logs/test-%j.err
module load python/3.11
source ~/envs/myproject/bin/activate
cd $SLURM_SUBMIT_DIR
python analyze.py --input data/sample01.csv --out results/sample01.jsonSave it as run.sh, make the logs and results folders first (the job fails if the output directory doesn't exist, which catches nearly everyone once), and submit with sbatch run.sh. You get a job ID back. squeue --me shows whether it's pending or running, and the reason column tells you what it's waiting for. When it finishes, read both the .out and the .err file, even if it looked like it worked.
The exact flags vary by system, so copy your center's example script rather than mine and adjust it. What carries over is the shape: request resources, load software, go to the right directory, run one command, write output somewhere findable.
Then do it many times
Once one job works, the reason you came to a cluster shows up: you need that same thing run across two thousand inputs. Don't write a loop that submits two thousand separate scripts. Slurm has job arrays (--array=1-2000) that launch one job per index from a single submission, and each task can pick its input from a file list using the array index. HTCondor systems have their own version of the same idea, and it's the pattern the OSPool is built around.
This is also where workflow tools earn their keep. Snakemake and Nextflow describe the steps and dependencies once and handle submission, retries, and restarting from where a failed run stopped. If your pipeline has more than about three stages, learning one of them will save you more time than it costs.
The errors you'll hit in week one
- Job dies instantly, empty output. Usually a path that doesn't exist, or an output directory you forgot to create. Read the
.errfile. - "OOM" or killed with no message. Out of memory. Raise
--memand check whether your code is loading a whole dataset it could stream. - "CANCELLED DUE TO TIME LIMIT". Your time request was short. Also a nudge to add checkpointing so a long job can resume.
- Works on the login node, fails in the job. Almost always a module or environment you loaded by hand and didn't put in the script.
- Job sits pending forever. Look at the reason in
squeue. It's often a request bigger than anything in the partition, or a wrong partition name.
Ask for help earlier than feels reasonable
Most research computing centers employ facilitators whose entire job is helping researchers get running, and they'd rather answer a fifteen-minute question than see a project stall for a month. If your campus doesn't have one, the Campus Champions network and the support teams behind the national systems fill the same role. Nobody scores you on how long you struggled first.
The takeaway
A cluster isn't a bigger laptop, it's a shared queue with rules, and once the rules are visible the rest is ordinary work. Log in, compute elsewhere. Load your software inside the script. Know which disk is backed up. Ask small, measure, then scale. Get one twenty-minute job through the queue today and the two-thousand-job version is a variation on it, not a new skill.
Getting researchers and teams from "I have an account" to "my pipeline runs" is a lot of what I do. If your campus has people stuck at this step, get in touch and we can talk about a workshop or a short onboarding pilot.