Python Handbook

10 minute read

Published: April 30, 2024

Conda

Install mamba:

conda install mamba -n base -c conda-forge

Remove environment:

conda remove --name r_env --all -y

For MacOS, override your default platform osx-arm64 and replace it with osx-64:

CONDA_SUBDIR=osx-64 mamba create -n renv
mamba activate renv
conda config --env --set subdir osx-64

Tensorflow-gpu in conda (Reference):

export XLA_FLAGS=--xla_gpu_cuda_data_dir=$CONDA_PREFIX

Create a conda environment from an environment.yml file:

conda env create -f environment.yml

Setup of R environment

(Reference)

mamba create -n r_env -y

conda activate r_env
conda config --add channels bioconda
conda config --add channels conda-forge
# conda config --set channel_priority strict
# conda config --set channel_priority false

mamba install r-base=4.1.1 -y
mamba install -c conda-forge -c bioconda r-seurat r-seuratdisk r-signac -y
mamba install -c bioconda bioconductor-genomeinfodbdata -y
mamba install -c bioconda bioconductor-genomeinfodbdata --force-reinstall

Jupyter Notebook

Python kernel

Add a new kernel kernel_name from a conda environment env (Reference):

conda activate env
mamba install ipykernel -y
python -m ipykernel install --name kernel_name
conda deactivate

R kernel

(Reference)

Add a new R kernel kernel_name from a conda environment r_env to another conda environment env:

conda activate r_env
mamba install r-irkernel -y

conda activate env
/raid6/Jinhong/anaconda3/envs/r_env/bin/R -e 'IRkernel::installspec(name="kernel_name", displayname="kernel_name")'

Bash commands

Running scripts

nohup python -u foo.py 0 > foo.out 2>&1 &

nohup Rscript foo.R 0 > foo.out 2>&1 &

Python dependencies

Print the versions of Python packages in the current conda environment:

conda list -f "python|numpy|numba|scipy|pandas|scikit-learn|h5py|anndata|scanpy|statsmodels|matplotlib|joblib|cvxpy|tqdm" | awk '{ print $1" | "$2 }'

R dependencies

Linux

System monitoring

System information:

# for basic system info
landscape-sysinfo

# for detailed system info
top
htop

System update:

sudo apt update && sudo apt upgrade -y && sudo reboot

File management

Delete all files with a certain pattern in the current directory and subdirectories (Reference):

find . -type f -name '*.o' -delete
find . -type f -name '*.o' -exec rm {} \;
rm -- **/*.o

Compress all files in the current directory and subdirectories with a certain pattern:

zip -r archive.zip results/* -i "*gears*"

Create a symbolic link for a folder:

ln -s /path/to/original/directory /path/to/link/location/link_name

Sync files from remote server:

rsync -avu --info=progress2 jinhongd@hydra1.stat.cmu.edu:/path /path

# Example: sync all .sh scripts from source to destination
rsync -zarv --include="*/" --include="*.sh" --exclude="*" "$from" "$to" 

User management

On most Linux servers, create a user with:

sudo adduser USERNAME

Set or reset the password:

sudo passwd USERNAME

Give access to a shared folder:

sudo mkdir -p /data/USERNAME
sudo chown -R USERNAME:USERNAME /data/USERNAME
sudo chmod 711 /data
sudo chmod 700 /data/USERNAME

711 on /data allows the user to enter the directory without listing everyone else’s files.

conda/mamba environment for multiple users:

sudo groupadd micromamba
sudo chgrp -R micromamba /data/miniforge3
sudo chmod 770 -R /data/miniforge3

# Each user who wants to use anaconda must be added to that group. Note that you need to log out or otherwise refresh your shell to see your new membership.
sudo adduser USERNAME micromamba

# Last, each user who wants to use conda needs to add it to their path. This can be done following the instructions in the installation doc:
source /data/miniforge3/bin/activate
conda init

Docker

Dependencies: https://stackoverflow.com/questions/76876492/unable-to-install-docker-on-ubuntu-22-04-2

Permission for apt: https://askubuntu.com/questions/908800/what-does-this-apt-error-message-download-is-performed-unsandboxed-as-root

Docker Desktop: https://docs.docker.com/desktop/setup/install/linux/ubuntu/

Enable lingering for the user: This ensures that the user's systemd services, including the rootless Docker daemon, can continue running even after the user logs out. Code

sudo loginctl enable-linger $(whoami)

Replace $(whoami) with the actual username if performing this for a different user.

VPN / Proxy Checking and Node Selection on Linux Server

This server uses mihomo as a Clash-compatible proxy backend.

Default local endpoints:

# HTTP + SOCKS mixed proxy
http://127.0.0.1:7890

# Mihomo external controller API
http://127.0.0.1:9000

The proxy group used for manual node switching is usually:

Proxy

If the group name is different, list all groups first.

1. Check whether mihomo is running

systemctl status mihomo --no-pager

Expected status:

Active: active (running)

Check whether the proxy and controller ports are listening:

ss -lntp | grep -E '7890|9000'

Expected output should include something like:

127.0.0.1:7890
127.0.0.1:9000

If 7890 is missing, the proxy port is not running.

If 9000 is missing, the external controller is not running, and node switching by curl will not work.

2. Check mihomo config path

The systemd service usually starts mihomo with:

/usr/local/bin/mihomo -d /etc/mihomo

So the active config file should be:

/etc/mihomo/config.yaml

Check important fields:

sudo grep -nE 'mixed-port|external-controller|secret|proxies:|proxy-groups:|rules:' /etc/mihomo/config.yaml

The config should include something like:

mixed-port: 7890
external-controller: 127.0.0.1:9000
secret: ""

It should also include:

proxies:
proxy-groups:
rules:

If the config only has something like:

mixed-port: 7890

then mihomo is running with an incomplete config and has no usable nodes.

3. Test the proxy endpoint

Check whether traffic can go through the proxy:

curl -x http://127.0.0.1:7890 https://ipinfo.io

A successful response should show the outbound IP information.

Test Claude API connectivity:

HTTPS_PROXY=http://127.0.0.1:7890 curl -I https://api.anthropic.com

Test OpenAI / Codex connectivity:

HTTPS_PROXY=http://127.0.0.1:7890 curl -I https://api.openai.com

HTTP status codes such as 401, 403, or 404 usually mean the network path is reachable but authentication or endpoint details are missing.

Network-level failures look like:

Connection refused
Could not resolve host
Operation timed out

4. Check whether the controller API works

curl -s http://127.0.0.1:9000/proxies | jq

If this fails with:

Connection refused

then the controller is not listening on port 9000.

Check logs:

sudo journalctl -u mihomo -n 100 --no-pager

Look for messages related to config parsing, controller startup, or listening ports.

5. List all proxy groups

curl -s http://127.0.0.1:9000/proxies | jq -r '.proxies | keys[]'

Typical groups may include:

Proxy
Auto
DIRECT
GLOBAL

The manually switchable group is usually Proxy.

6. Show available nodes in the `Proxy` group

curl -s http://127.0.0.1:9000/proxies | jq -r '.proxies["Proxy"].all[]'

Show the currently selected node:

curl -s http://127.0.0.1:9000/proxies | jq -r '.proxies["Proxy"].now'

If this returns null, the group name may not be Proxy.

List all groups again:

curl -s http://127.0.0.1:9000/proxies | jq -r '.proxies | keys[]'

Then replace Proxy with the correct group name.

7. Manually switch node

Switch the Proxy group to Auto:

curl -X PUT 'http://127.0.0.1:9000/proxies/Proxy' \
  -H 'Content-Type: application/json' \
  --data '{"name":"Auto"}'

Switch to a specific node:

curl -X PUT 'http://127.0.0.1:9000/proxies/Proxy' \
  -H 'Content-Type: application/json' \
  --data '{"name":"NODE_NAME_HERE"}'

Example:

curl -X PUT 'http://127.0.0.1:9000/proxies/Proxy' \
  -H 'Content-Type: application/json' \
  --data '{"name":"日本-PRO-FW-JP1-流量倍率:0.2"}'

Confirm the selected node:

curl -s http://127.0.0.1:9000/proxies | jq -r '.proxies["Proxy"].now'

8. Useful helper script: `switch-node`

Create a command-line helper:

sudo tee /usr/local/bin/switch-node > /dev/null <<'EOF'
#!/usr/bin/env bash
set -euo pipefail

API="http://127.0.0.1:9000"
GROUP="${GROUP:-Proxy}"

if [ $# -eq 0 ]; then
  echo "Current node:"
  curl -s "$API/proxies" | jq -r ".proxies[\"$GROUP\"].now"

  echo
  echo "Available nodes:"
  curl -s "$API/proxies" | jq -r ".proxies[\"$GROUP\"].all[]"
  exit 0
fi

NODE="$*"

curl -s -X PUT "$API/proxies/$GROUP" \
  -H 'Content-Type: application/json' \
  --data "{\"name\":\"$NODE\"}" >/dev/null

echo "Switched to:"
curl -s "$API/proxies" | jq -r ".proxies[\"$GROUP\"].now"
EOF

sudo chmod +x /usr/local/bin/switch-node

Usage:

# Show current node and available nodes
switch-node

# Switch to Auto
switch-node Auto

# Switch to a specific node
switch-node 日本-PRO-FW-JP1-流量倍率:0.2

If the proxy group is not named Proxy, set GROUP manually:

GROUP='YourGroupName' switch-node
GROUP='YourGroupName' switch-node Auto

9. Enable proxy variables for Claude / Codex / CLI tools

For the current shell:

export HTTP_PROXY=http://127.0.0.1:7890
export HTTPS_PROXY=http://127.0.0.1:7890
export ALL_PROXY=http://127.0.0.1:7890

export http_proxy=http://127.0.0.1:7890
export https_proxy=http://127.0.0.1:7890
export all_proxy=http://127.0.0.1:7890

export NO_PROXY=localhost,127.0.0.1,::1,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16
export no_proxy=$NO_PROXY

For all SSH users, add these variables to:

/etc/profile.d/proxy.sh

Example:

sudo tee /etc/profile.d/proxy.sh > /dev/null <<'EOF'
export HTTP_PROXY=http://127.0.0.1:7890
export HTTPS_PROXY=http://127.0.0.1:7890
export ALL_PROXY=http://127.0.0.1:7890

export http_proxy=http://127.0.0.1:7890
export https_proxy=http://127.0.0.1:7890
export all_proxy=http://127.0.0.1:7890

export NO_PROXY=localhost,127.0.0.1,::1,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16
export no_proxy=$NO_PROXY
EOF

After re-login:

env | grep -i proxy

11. Minimal daily workflow

# Check service
systemctl status mihomo --no-pager

# Check ports
ss -lntp | grep -E '7890|9000'

# List available nodes
switch-node

# Switch node
switch-node Auto

# Test outbound IP
curl -x http://127.0.0.1:7890 https://ipinfo.io

# Test Claude
HTTPS_PROXY=http://127.0.0.1:7890 curl -I https://api.anthropic.com

# Test OpenAI / Codex
HTTPS_PROXY=http://127.0.0.1:7890 curl -I https://api.openai.com


# If config changes, restart mihomo
sudo systemctl restart mihomo

SOCKS5-based VPN on Linux

Suppose a VPN/proxy client, such as a Trojan client, provides a local SOCKS5 proxy at:

127.0.0.1:1080

Test whether the SOCKS5 proxy works:

curl -v --socks5-hostname 127.0.0.1:1080 https://ifconfig.me
curl -v --socks5-hostname 127.0.0.1:1080 https://api.anthropic.com

Use --socks5-hostname rather than --socks5 so that DNS resolution is also sent through the proxy.

Some command-line tools do not support SOCKS5 proxies directly. For example, Claude Code respects HTTP_PROXY and HTTPS_PROXY, but does not directly support SOCKS5 proxies. If Claude Code shows errors such as ECONNRESET, bridge the SOCKS5 proxy to a local HTTP proxy using privoxy.

Install privoxy:

sudo apt update
sudo apt install privoxy

Edit the config:

sudo nano /etc/privoxy/config

Add the following lines:

listen-address 127.0.0.1:8118
forward-socks5t / 127.0.0.1:1080 .

Restart privoxy:

sudo systemctl restart privoxy

Then export HTTP proxy variables:

unset ALL_PROXY all_proxy
unset HTTP_PROXY HTTPS_PROXY http_proxy https_proxy

export HTTP_PROXY=http://127.0.0.1:8118
export HTTPS_PROXY=http://127.0.0.1:8118
export NO_PROXY=localhost,127.0.0.1,::1

Test the HTTP proxy bridge:

curl -v --proxy http://127.0.0.1:8118 https://api.anthropic.com
curl -v --proxy http://127.0.0.1:8118 https://claude.ai

Run Claude Code:

claude

If it still fails, inspect the proxy-related environment variables:

env | grep -i proxy

A common bad setup is:

HTTPS_PROXY=socks5://127.0.0.1:1080
HTTP_PROXY=socks5://127.0.0.1:1080

For Claude Code, use the HTTP proxy bridge instead:

HTTPS_PROXY=http://127.0.0.1:8118
HTTP_PROXY=http://127.0.0.1:8118

For browsers, make sure SOCKS5 uses remote DNS. In Firefox, set:

network.proxy.socks_remote_dns = true

For Chrome or Chromium, disabling QUIC may help if HTTPS traffic behaves inconsistently:

chrome://flags/#enable-quic

LaTeX on Linux

sudo apt install texlive-full texlive-latex-recommended

Slurm

Check GPU usage on the node n01:

ssh n01 'nvidia-smi -l 1'

Check the status of the jobs:

# all jobs
squeue -u jinhongd

# pending jobs
squeue --start -u jinhongd

Check the CPU and memory usage of a job:

# for running jobs
sacct -e
sacct --format="CPUTime,MaxRSS"
sacct -a -j <job_id> --format=user%10,jobname%10,node%10,start%10,end%10,elapsed%10,MaxRS

# for finished jobs
seff <job_id>

LLM

claude --dangerously-skip-permissions

codex --dangerously-bypass-approvals-and-sandbox

codex alias

mkdir -p ~/.codex-jing

# Login personal
CODEX_HOME=~/.codex-jing codex login

alias codex-jing='CODEX_HOME=$HOME/.codex-jing codex'


# or add to ~/.bashrc
export CODEX_HOME_WORK="$HOME/.codex-jing"
source ~/.bashrc

Share on

Twitter Facebook LinkedIn

Python Handbook

Conda

Setup of R environment

Jupyter Notebook

Python kernel

R kernel

Bash commands

Running scripts

Python dependencies

R dependencies

Linux

System monitoring

File management

User management

Docker

VPN / Proxy Checking and Node Selection on Linux Server

1. Check whether mihomo is running

2. Check mihomo config path

3. Test the proxy endpoint

4. Check whether the controller API works

5. List all proxy groups

6. Show available nodes in the `Proxy` group

7. Manually switch node

8. Useful helper script: `switch-node`

9. Enable proxy variables for Claude / Codex / CLI tools

11. Minimal daily workflow

SOCKS5-based VPN on Linux

LaTeX on Linux

Slurm

LLM

Share on

You May Also Enjoy

Academic Job Interview

Screening interview questions

Norm Inequality

Divergence Measures

Reproducing Kernel Hilbert Spaces

Conda

Setup of R environment

Jupyter Notebook

Python kernel

R kernel

Bash commands

Running scripts

Python dependencies

R dependencies

Linux

System monitoring

File management

User management

Docker

VPN / Proxy Checking and Node Selection on Linux Server

1. Check whether mihomo is running

2. Check mihomo config path

3. Test the proxy endpoint

4. Check whether the controller API works

5. List all proxy groups

6. Show available nodes in the Proxy group

7. Manually switch node

8. Useful helper script: switch-node

9. Enable proxy variables for Claude / Codex / CLI tools

11. Minimal daily workflow

SOCKS5-based VPN on Linux

LaTeX on Linux

Slurm

LLM

Share on

You May Also Enjoy

Academic Job Interview

Screening interview questions

Norm Inequality

Divergence Measures

Reproducing Kernel Hilbert Spaces

6. Show available nodes in the `Proxy` group

8. Useful helper script: `switch-node`