Running Databases

LAVA provides a Python script (setup.py) for setting up and running databases (parametric sweeps) of simulations. It reads a setup file and a run matrix, creates per-run sub-directories with parameterized input files and PBS job scripts, and optionally submits the jobs.

Usage

setup.py <setup_file>          # Generate directories and PBS scripts only
setup.py <setup_file> run      # Generate and submit jobs via qsub
setup.py <setup_file> debug    # Generate PBS scripts as if on a NAS front-end (useful for local testing)

setup.inputs

The setup file specifies the run configuration. All parameters have defaults and can be omitted if the default is acceptable. An example:

run_matrix       = "run_matrix.txt"
run_path         = "runs/"
inputs_template  = "run.inputs"
grids_path       = "/path/to/grids"
sym_links        = ["../../grid.cgns"]
copy_files       = []
exec_name        = "lavacart.3D.NS1"
num_nodes        = 10
pbs_queue        = "long"
pbs_gid          = "your_project_gid"
pbs_model        = "mil_ait"
pbs_walltime     = "8:00:00"
pbs_source       = "module use /swbuild/LAVA/LAVA_modules\nmodule load LAVA"

All Parameters

Parameter	Default	Description
`run_matrix`	`run_matrix.txt`	Path to the run matrix file
`run_path`	`runs/`	Output directory where all run sub-directories are created
`inputs_template`	`run.inputs`	Template input file for LAVA
`grids_path`	(empty)	Path to a directory containing grid files; a symlink is created per run using the `grid` column from the run matrix
`sym_links`	`["../grid.cgns"]`	List of files/patterns to symlink into every run directory
`copy_files`	`[]`	List of files/directories to copy into every run directory
`exec_name`	`lavacart.3D.NS1`	Base executable name; architecture suffix (e.g. `AVX2`, `AVX512`) is appended automatically per node type
`num_nodes`	`10`	Number of nodes to request
`num_mpi`	(auto)	MPI ranks per node; defaults to a per-model value if not set
`num_omp`	(auto)	OpenMP threads per MPI rank; derived from `hyperthreadfac` and cores/node if not set
`hyperthreadfac`	`1`	Hyperthreading multiplier used to compute `num_omp` automatically
`pbs_queue`	`devel`	PBS queue name
`pbs_gid`	`a1455`	PBS group/project ID
`pbs_model`	(required)	Node model (see table below)
`pbs_walltime`	`2:00:00`	Requested wall-clock time
`pbs_shell`	`/bin/bash`	Shell for the PBS script (`/bin/bash` or `/bin/csh`)
`pbs_source`	`module use .../LAVA_modules` `module load LAVA`	Commands written into the PBS script after `cd $PBS_O_WORKDIR`, used to set up the environment. Override with a custom `source` command if needed.
`pbs_pinning`	`mbind`	Process pinning tool for CPU nodes: `mbind` or `omplace`
`pbs_email`	`False`	Send PBS email notifications (`True`/`False`)
`pbs_emailaddress`	`$USER@nas.nasa.gov`	Address for PBS email notifications
`pbs_lock`	`False`	Write a `RUNNING`/`FINISHED` lock-file guard into the PBS script
`pbs_multi_model`	`[]`	List of additional node models to generate PBS scripts for; SBU-equivalent node counts are computed automatically
`pbs_filesystems`	(empty)	Value for `#PBS -l site=needed=<filesystems>` (e.g. `lustre`)

| do_interactive | False | Keep the job alive after the solver finishes (for interactive debugging) |

| do_several_tries | False | Wrap the executable with /u/scicon/tools/bin/several_tries |

Supported Node Models (NAS)

The script automatically selects the correct MPI launch command, architecture suffix, and select-line format for each node type.

Model	System	Cores	GPUs	Mem (GB)	SBU	MPI Launch
`bro_ele`	Electra (pfe)	28	—	128	1.00	`mpirun -npernode`
`sky_ele`	Electra (pfe)	40	—	192	1.59	`mpirun -npernode`
`cas_ait`	Aitken (afe)	40	—	192	1.64	`mpirun -npernode`
`rom_ait`	Aitken (afe)	128	—	512	4.06	`mpirun -npernode`
`mil_ait`	Aitken (afe)	128	—	512	4.38	`mpirun -npernode`
`tur_ath`	Athena (athfe)	512	—	1024	12.19	`mpiexec -ppn --cpu-bind none`
`mil_a100`	Cabeus (cfe)	128	4× A100	500	37.86	`mpiexec -N --bind-to none`
`gh200`	Cabeus GH200 (cghfe)	72	1× GH200	460	18.34	`mpiexec -N + numactl`

pbs_multi_model

Setting pbs_multi_model generates one PBS script per listed node type. The number of nodes for each model is scaled automatically to maintain an equivalent SBU cost relative to the reference pbs_model. When more than one model is listed, pbs_lock is enabled automatically so that only one script can run at a time.

pbs_model       = "mil_ait"
pbs_multi_model = ["mil_ait", "tur_ath", "mil_a100"]

This produces run_mil_ait.pbs, run_tur_ath.pbs, and run_mil_a100.pbs in each run directory.

run_matrix.txt

The run matrix is a whitespace-delimited text file. The first row is the header (column names); subsequent rows are parameter values. Each row produces one run sub-directory.

Column names that match setup parameters (num_nodes, pbs_model, pbs_queue, pbs_walltime, etc.) override the corresponding setup-file value for that run. Any other column names are treated as template substitution variables.

Append * to a column name to exclude it from the sub-directory name.

Example:

dt*       num_steps*  levels    num_nodes*
8.00e-5        3125       5          5
4.00e-5        6250       6          5
2.00e-5       12500       7          5
1.00e-5       25000       8         10
5.00e-6       50000       9         25
2.50e-6      100000      10         50
1.25e-6      200000      11        400

This creates directories named run_levels5/, run_levels6/, …, one per row.

inputs_template

The template input file is a standard LAVA input file with substitution placeholders of the form ${PARAM}, where PARAM matches a column header in run_matrix.txt. The script replaces all placeholders with the corresponding value for each row.

dt = ${dt}
num_steps = ${num_steps}
levels = ${levels}

Example Directory Structure

After running setup.py setup.inputs, the output looks like:

runs/
  run_levels5/
    run.inputs        ← substituted input file
    run.pbs           ← PBS job script
    grid.cgns -> ...  ← symlinks from sym_links
  run_levels6/
    ...