Research Computing Handbook
Published:
Day-to-day notes on conda, Python, R, Jupyter, and cluster jobs. Server administration notes (users, Docker, proxy, shared SSH) moved to Linux Server Setup.
Conda and mamba
Basics
Install mamba:
conda install mamba -n base -c conda-forge
Create a conda environment from an environment.yml file:
conda env create -f environment.yml
Remove environment:
ENV_NAME=r_env
conda remove --name "$ENV_NAME" --all -y
Intel packages on Apple silicon
For MacOS, override your default platform osx-arm64 and replace it with osx-64:
ENV_NAME=renv
CONDA_SUBDIR=osx-64 mamba create -n "$ENV_NAME"
mamba activate "$ENV_NAME"
conda config --env --set subdir osx-64
TensorFlow GPU
Tensorflow-gpu in conda (Reference):
export XLA_FLAGS=--xla_gpu_cuda_data_dir=$CONDA_PREFIX
R through conda
R_ENV=r_env
mamba create -n "$R_ENV" -y
conda activate "$R_ENV"
conda config --add channels bioconda
conda config --add channels conda-forge
# conda config --set channel_priority strict
# conda config --set channel_priority false
mamba install r-base=4.1.1 -y
mamba install -c conda-forge -c bioconda r-seurat r-seuratdisk r-signac -y
mamba install -c bioconda bioconductor-genomeinfodbdata -y
mamba install -c bioconda bioconductor-genomeinfodbdata --force-reinstall
Jupyter kernels
Python kernel
Add a new kernel KERNEL from a conda environment ENV (Reference):
ENV=env
KERNEL=kernel_name
conda activate "$ENV"
mamba install ipykernel -y
python -m ipykernel install --name "$KERNEL"
conda deactivate
R kernel
Add a new R kernel KERNEL from a conda environment R_ENV to another conda environment ENV:
CONDA_ROOT=/raid6/Jinhong/anaconda3
R_ENV=r_env
ENV=env
KERNEL=kernel_name
conda activate "$R_ENV"
mamba install r-irkernel -y
conda activate "$ENV"
"$CONDA_ROOT/envs/$R_ENV/bin/R" -e "IRkernel::installspec(name='$KERNEL', displayname='$KERNEL')"
Running jobs
Background scripts
nohup python -u foo.py 0 > foo.out 2>&1 &
nohup Rscript foo.R 0 > foo.out 2>&1 &
Slurm
Check GPU usage on a node:
NODE=n01
ssh "$NODE" 'nvidia-smi -l 1'
Check the status of your jobs ($USER is your login name):
# all jobs
squeue -u "$USER"
# pending jobs
squeue --start -u "$USER"
Check the CPU and memory usage of a job:
JOB_ID=123456
# for running jobs
sacct -e
sacct --format="CPUTime,MaxRSS"
sacct -a -j "$JOB_ID" --format=user%10,jobname%10,node%10,start%10,end%10,elapsed%10,MaxRSS
# for finished jobs
seff "$JOB_ID"
Package versions
Python
Print the versions of Python packages in the current conda environment:
conda list -f "python|numpy|numba|scipy|pandas|scikit-learn|h5py|anndata|scanpy|statsmodels|matplotlib|joblib|cvxpy|tqdm" | awk '{ print $1" | "$2 }'
R
Print the versions of R packages in the current environment:
Rscript -e 'for (p in c("Matrix", "Seurat", "Signac")) cat(p, "|", as.character(packageVersion(p)), "\n")'
Shell utilities
Files
Delete all files with a certain pattern in the current directory and subdirectories (Reference):
find . -type f -name '*.o' -delete
find . -type f -name '*.o' -exec rm {} \;
rm -- **/*.o
Compress all files in the current directory and subdirectories with a certain pattern:
zip -r archive.zip results/* -i "*gears*"
Create a symbolic link for a folder:
TARGET=/path/to/original/directory
LINK=/path/to/link/location/link_name
ln -s "$TARGET" "$LINK"
rsync
Set the remote host and the two directories once:
REMOTE=jinhongd@hydra1.stat.cmu.edu
FROM=/path/on/remote
TO=/path/on/local
Sync files from remote server:
rsync -avu --info=progress2 "$REMOTE:$FROM" "$TO"
Sync only the .sh scripts:
rsync -zarv --include="*/" --include="*.sh" --exclude="*" "$REMOTE:$FROM" "$TO"
SSH keys
Set a custom key name and the server once:
KEY=~/.ssh/id_ed25519_myserver
REMOTE=user@remote_host
Generate key:
ssh-keygen -t ed25519 -f "$KEY"
Authorize it on the server (Requires password once):
ssh-copy-id -i "$KEY.pub" "$REMOTE"
SSH only tries the default key names on its own, so point it to the custom key, either per command:
ssh -i "$KEY" "$REMOTE"
or once in ~/.ssh/config:
Host myserver
HostName remote_host
User user
IdentityFile ~/.ssh/id_ed25519_myserver
IdentitiesOnly yes
and then connect with ssh myserver.
AI coding CLIs
claude --dangerously-skip-permissions
codex --dangerously-bypass-approvals-and-sandbox
Second Codex account
mkdir -p ~/.codex-team
# Login personal
CODEX_HOME=~/.codex-team codex login
# or add to ~/.bashrc
echo 'alias codex-team="CODEX_HOME=$HOME/.codex-team codex"' >> ~/.bashrc
source ~/.bashrc
Second Claude account
mkdir -p ~/.claude-team
# Find the exact path to your global claude binary
type -a claude
# for example, $HOME/.local/bin/claude
# add to ~/.bashrc
echo 'alias claude-team="CLAUDE_CONFIG_DIR=~/.claude-team $HOME/.local/bin/claude"' >> ~/.bashrc
source ~/.bashrc