Running on a Cluster¶

When local laptop time isn't enough, you can take the same project to a SLURM HPC system. There's no separate configuration to learn — the same lc run command works inside an allocation, just with more hardware to spread across.

The big picture¶

lc run always dispatches through a Dask cluster. Four branches:

1. On your laptop → a LocalCluster sized to the machine.
2. Inside a SLURM allocation → an in-process scheduler bound to the driver's hostname, with one dask worker per allocated node launched via srun.
3. On a lightcone JupyterHub deployment → attach to your running Dask Gateway cluster (see the JupyterHub section below — don't use DASK_SCHEDULER_ADDRESS with a gateway:// address, it cannot be dialled directly).
4. With DASK_SCHEDULER_ADDRESS set → connect to whatever scheduler you've pointed at.

You don't pick — lc run detects which case applies. The only thing you do differently on a cluster is request the nodes.

Pre-flight: pick the right container runtime¶

On most HPC sites, docker isn't available on compute nodes. Most SLURM systems (including NERSC Perlmutter) provide podman-hpc. On a login node:

$EDITOR ~/.lightcone/config.yaml

container:
  runtime: podman-hpc

Then build and migrate the images for your project:

cd my-analysis
lc build

lc build runs podman-hpc build and then podman-hpc migrate, which copies the image into the per-node container cache. Compute nodes can read it without registry access.

If your site has only apptainer / singularity, the Lightcone toolchain doesn't ship explicit support for those today — you can run without containers (runtime: none) for the moment, with the caveat that the manifest's container_image field will record what was declared, not what executed. (See lc run for the provenance warning.)

A typical SLURM workflow¶

1. Get an allocation¶

salloc -N 4 -t 02:00:00 -C gpu                       # interactive
# or
sbatch run.sbatch                                    # batch

run.sbatch looks like:

#!/bin/bash
#SBATCH -N 4
#SBATCH -t 02:00:00
#SBATCH -C gpu

cd $HOME/my-analysis
source .venv/bin/activate
lc run -j 16

#!/bin/bash
#SBATCH -A <your_project>
#SBATCH -q regular
#SBATCH -C gpu
#SBATCH -N 4
#SBATCH -t 04:00:00

cd $SCRATCH/your-analysis

# make `lc` available — pick the line that matches your install:
export PATH=$HOME/.local/bin:$PATH                # uv tool install
# source ~/.conda/envs/your-env-name/bin/activate # conda env

lc run -j 16

2. lc run inside the allocation¶

Once SLURM_JOB_ID is set in your environment, lc run does the rest:

• Starts an in-process Dask scheduler bound to the SLURM node hostname.
• Launches one dask worker per node via srun.
• Each worker advertises the node's CPU, memory, and GPU resources.
• Snakemake submits each rule via the Dask executor; rules with per-recipe resources: constraints land on workers that can hold them.

3. Per-recipe resource hints¶

Add resource hints in your astra.yaml recipe blocks:

outputs:
  - id: heavy_fit
    type: metric
    recipe:
      command: python scripts/fit.py --output {output[0]}
      resources:
        cpus_per_task: 32
        mem_mb: 64000
        gpus_per_task: 1

The Snakemake-via-Dask executor maps these to per-task resource requests, so a rule that needs a GPU only schedules on nodes that advertise one.

Interactive: agent-driven runs¶

During development you're usually iterating — ask the agent to build something, check the result, adjust the spec, repeat. For that loop you want to run the agent itself from inside a SLURM allocation so that lc run executes on the compute node rather than the login node.

salloc -A <your_project> -q interactive -C gpu --nodes=1 -t 00:30:00
# salloc drops you onto a compute node; from there:
cd /path/to/your-analysis
claude                   # or whichever agent CLI you prefer

Everything the agent triggers (lc run, scripts, etc.) now executes on the allocated node. When you're done iterating and want a hands-off sweep of all universes, submit lc run as a batch job instead (the sbatch template above).

What about login-node-only operations?¶

Build images, dry-run, look at status — all fine on a login node without an allocation:

lc build                       # build images (uses podman-hpc on login node)
lc status                      # offline; reads only manifests

The actual lc run should happen inside an allocation, since that's where the worker nodes are.

External Dask schedulers¶

If you have a long-lived Dask cluster (Slurm jobqueue, k8s, etc.) that you'd rather attach to:

export DASK_SCHEDULER_ADDRESS=tcp://my-scheduler:8786
lc run

lc run notices the env var and connects rather than starting its own scheduler. It does not tear the scheduler down on exit.

JupyterHub / Dask Gateway¶

On a lightcone JupyterHub deployment (where the DASK_GATEWAY__* env vars are ambient in every pod), the model is: you create the cluster, lc run attaches to it.

1. Create a Dask Gateway cluster from JupyterLab — the Dask sidebar's + NEW button, or a notebook:

from dask_gateway import Gateway
# shutdown_on_close=False keeps the cluster alive when this kernel
# exits — otherwise a kernel restart kills any lc run attached to it.
# Idle clusters are reaped by the deployment after 30 min.
cluster = Gateway().new_cluster(shutdown_on_close=False)
cluster.adapt(minimum=1, maximum=8)

1. Run:

lc run

The Gateway API only shows your clusters, so with a single cluster running lc run attaches with zero configuration — no cluster name to copy. It never changes the cluster's scaling and leaves it running on exit, mirroring the DASK_SCHEDULER_ADDRESS convention, so you can iterate lc run against the same warm cluster with its dashboard panels docked. With no cluster running, lc run tells you how to create one; with several, pick one:

export LIGHTCONE_GATEWAY_CLUSTER=<cluster-name>   # e.g. hub.a1b2c3...
lc run

A Gateway scheduler's gateway:// address cannot be used with DASK_SCHEDULER_ADDRESS — attachment is always by name.

Requires the optional gateway extra: pip install lightcone-cli[gateway] (preinstalled on the hub image).

Containers on the hub: the pod is the runtime¶

There is no docker or podman inside a pod — Kubernetes itself is the container runtime (container_runtime: kubernetes, declared by the site, no warning fired). Your recipes run unwrapped inside the worker pod, so the worker image is the project environment:

• A container: spec naming a registry image is used directly: create your Gateway cluster with that image.
• A container: Containerfile spec resolves to a content-addressed ref in the deployment registry: $LIGHTCONE_REGISTRY/lc-<project>:<hash> — the same hash the local lc build tag carries, so the environment is provably the same artifact on every path. lc build on the hub checks whether that ref exists in the registry and, when it doesn't, prints the publish commands: run lc build from a clone on any machine with docker (this builds from the hash-attested, filtered build context — don't substitute a raw docker build .), then docker tag lc-<project>-<hash> <ref> and docker push <ref>.

For the image to work as a Gateway worker it must contain dask, distributed, dask-gateway, and lightcone-cli at versions matching the hub — the simplest way is to base your Containerfile on the deployment's worker image (FROM <registry>/lightcone-worker-default:<tag>) and add your science deps on top. That one Containerfile then serves every path: built locally it wraps recipes on your laptop; pushed to the registry it runs them as pods on the hub.

At attach time lc run verifies the cluster's actual worker image against the project's resolved image and warns on mismatch. Manifests record the image the worker pod actually ran (worker_image), so provenance stays truthful even if you knowingly run on a stale cluster.

NERSC Perlmutter: site-specific notes¶

Storage: keep Snakemake state on $SCRATCH¶

lc redirects state automatically when it detects Perlmutter, so this usually just works. To pin explicitly at project creation:

lc init your-analysis --scratch '$SCRATCH'   # kept verbatim, expanded at run time

Or, after the fact, edit <project>/.lightcone/lightcone.yaml:

scratch_root: $SCRATCH

Further reading¶

• NERSC interactive jobs — salloc patterns and reservation queues
• Perlmutter system overview — node types and partitions
• NERSC queue policy — QoS options for GPU and CPU partitions

Troubleshooting¶

• dask CLI is not on PATH inside the SLURM allocation. Install lightcone-cli into the venv that your sbatch script activates; dask ships with distributed, which is a transitive dep.
• Workers never register. Usually means the SLURM node hostnames aren't resolvable from each other; check SLURMD_NODENAME / gethostname() and confirm the workers can reach the scheduler.
• Image not found on compute nodes. Re-run lc build on the login node — the migrate step is the one that actually publishes the image to the per-node cache.
• Snakemake locking errors or silent rule-rerun loops on Perlmutter. .snakemake/ ended up on DVS-mounted storage — set scratch_root: $SCRATCH in the project's .lightcone/lightcone.yaml.
• pip install hangs or times out. Compute nodes have no public internet — always install from a login node.
• PermissionError reading another user's symlinked results/. Cross-user scratch path without group ACLs — request access from the data owner, or copy the manifests into your own scratch.

For the wiring detail, see engine/dask_cluster in the maintainer docs.