Linux Server Setup
Published:
Notes on administering a shared Linux (Ubuntu) server. Day-to-day notes on conda, Python, R, and Slurm are in the Research Computing Handbook.
System basics
Monitoring
# for basic system info
landscape-sysinfo
# for detailed system info
top
htop
Updates
sudo apt update && sudo apt upgrade -y && sudo reboot
LaTeX
sudo apt install texlive-full texlive-latex-recommended
Users and shared storage
Set the username once; the commands in this section and the next use it:
USERNAME=alice
On most Linux servers, create a user with:
sudo adduser "$USERNAME"
Set or reset the password:
sudo passwd "$USERNAME"
Give the user a private folder on the shared disk:
sudo mkdir -p "/data/$USERNAME"
sudo chown -R "$USERNAME:$USERNAME" "/data/$USERNAME"
sudo chmod 711 /data
sudo chmod 700 "/data/$USERNAME"
711 on /data allows the user to enter the directory without listing everyone else’s files.
Shared conda for many users
Install conda/mamba once (here /data/miniforge3) and share it through a group:
CONDA_DIR=/data/miniforge3
sudo groupadd micromamba
sudo chgrp -R micromamba "$CONDA_DIR"
sudo chmod 770 -R "$CONDA_DIR"
# Add each user who needs conda to the group (USERNAME as set above).
# The user must log out and back in to pick up the new membership.
sudo adduser "$USERNAME" micromamba
Each user then adds conda to their own PATH:
source /data/miniforge3/bin/activate
conda init
Docker
References:
- Installing Docker dependencies on Ubuntu 22.04
- apt warning "Download is performed unsandboxed as root"
- Docker Desktop on Ubuntu
For rootless Docker, enable lingering for the user. This keeps the user's systemd services, including the rootless Docker daemon, running after the user logs out:
sudo loginctl enable-linger $(whoami)
Replace $(whoami) with the username when doing this for another user.
Network proxy
mihomo
This server uses mihomo as a Clash-compatible proxy backend, started by systemd as /usr/local/bin/mihomo -d /etc/mihomo. Default local endpoints:
# HTTP + SOCKS mixed proxy
http://127.0.0.1:7890
# mihomo external controller API
http://127.0.0.1:9000
The proxy group used for manual node switching is usually Proxy.
1. Check that the service and ports are up
systemctl status mihomo --no-pager # expect: Active: active (running)
ss -lntp | grep -E '7890|9000' # expect 127.0.0.1:7890 and 127.0.0.1:9000
If 7890 is missing, the proxy port is not running. If 9000 is missing, the external controller is not running, and switching nodes with curl will not work.
2. Check the config
The active config is /etc/mihomo/config.yaml. Check the important fields:
sudo grep -nE 'mixed-port|external-controller|secret|proxies:|proxy-groups:|rules:' /etc/mihomo/config.yaml
The config should contain:
mixed-port: 7890
external-controller: 127.0.0.1:9000
secret: ""
proxies:
proxy-groups:
rules:
If it only has mixed-port: 7890, mihomo is running with an incomplete config and has no usable nodes.
3. Test the proxy
# Outbound IP
curl -x http://127.0.0.1:7890 https://ipinfo.io
# Claude API
HTTPS_PROXY=http://127.0.0.1:7890 curl -I https://api.anthropic.com
# OpenAI / Codex
HTTPS_PROXY=http://127.0.0.1:7890 curl -I https://api.openai.com
HTTP status codes such as 401, 403, or 404 usually mean the network path works and only authentication or endpoint details are missing. Network-level failures look like Connection refused, Could not resolve host, or Operation timed out.
4. Check the controller API
curl -s http://127.0.0.1:9000/proxies | jq
If this fails with Connection refused, the controller is not listening on port 9000. Check the logs for messages about config parsing, controller startup, or listening ports:
sudo journalctl -u mihomo -n 100 --no-pager
5. List groups and nodes
Set the controller address and proxy group once; steps 5 and 6 use them:
API=http://127.0.0.1:9000
GROUP=Proxy
# All proxy groups (typically Proxy, Auto, DIRECT, GLOBAL)
curl -s "$API/proxies" | jq -r '.proxies | keys[]'
# Available nodes in the group
curl -s "$API/proxies" | jq -r ".proxies[\"$GROUP\"].all[]"
# Currently selected node
curl -s "$API/proxies" | jq -r ".proxies[\"$GROUP\"].now"
If the last command returns null, the group is not called Proxy; set GROUP to the right name from the group list.
6. Switch node manually
Set the target node (Auto, or a name from the list above), then switch:
NODE=Auto
curl -X PUT "$API/proxies/$GROUP" \
-H 'Content-Type: application/json' \
--data "{\"name\":\"$NODE\"}"
Confirm the selection with the .now query from step 5.
7. Helper script: switch-node
sudo tee /usr/local/bin/switch-node > /dev/null <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
API="http://127.0.0.1:9000"
GROUP="${GROUP:-Proxy}"
if [ $# -eq 0 ]; then
echo "Current node:"
curl -s "$API/proxies" | jq -r ".proxies[\"$GROUP\"].now"
echo
echo "Available nodes:"
curl -s "$API/proxies" | jq -r ".proxies[\"$GROUP\"].all[]"
exit 0
fi
NODE="$*"
curl -s -X PUT "$API/proxies/$GROUP" \
-H 'Content-Type: application/json' \
--data "{\"name\":\"$NODE\"}" >/dev/null
echo "Switched to:"
curl -s "$API/proxies" | jq -r ".proxies[\"$GROUP\"].now"
EOF
sudo chmod +x /usr/local/bin/switch-node
Usage:
switch-node # show current and available nodes
switch-node Auto # switch to Auto
switch-node "$NODE" # switch to the node set in step 6
# If the group is not called Proxy (the script reads GROUP from the environment)
export GROUP=YourGroupName
switch-node
switch-node Auto
8. Proxy variables for Claude, Codex, and other CLI tools
Put the variables in /etc/profile.d/proxy.sh so every SSH user gets them at login:
sudo tee /etc/profile.d/proxy.sh > /dev/null <<'EOF'
export HTTP_PROXY=http://127.0.0.1:7890
export HTTPS_PROXY=http://127.0.0.1:7890
export ALL_PROXY=http://127.0.0.1:7890
export http_proxy=http://127.0.0.1:7890
export https_proxy=http://127.0.0.1:7890
export all_proxy=http://127.0.0.1:7890
export NO_PROXY=localhost,127.0.0.1,::1,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16
export no_proxy=$NO_PROXY
EOF
For the current shell only, run source /etc/profile.d/proxy.sh. After logging in again, check with:
env | grep -i proxy
9. Daily checklist
systemctl status mihomo --no-pager # service
ss -lntp | grep -E '7890|9000' # ports
switch-node # list nodes
switch-node Auto # switch node
curl -x http://127.0.0.1:7890 https://ipinfo.io # outbound IP
HTTPS_PROXY=http://127.0.0.1:7890 curl -I https://api.anthropic.com # Claude
HTTPS_PROXY=http://127.0.0.1:7890 curl -I https://api.openai.com # OpenAI / Codex
# After changing the config
sudo systemctl restart mihomo
SOCKS5 client with a privoxy bridge
Suppose a VPN/proxy client, such as a Trojan client, provides a local SOCKS5 proxy at 127.0.0.1:1080. Test it:
curl -v --socks5-hostname 127.0.0.1:1080 https://ifconfig.me
curl -v --socks5-hostname 127.0.0.1:1080 https://api.anthropic.com
Use --socks5-hostname rather than --socks5 so that DNS resolution also goes through the proxy.
Some command-line tools do not support SOCKS5 proxies. Claude Code, for example, respects HTTP_PROXY and HTTPS_PROXY but not SOCKS5; with a SOCKS5 proxy it fails with errors such as ECONNRESET. Bridge the SOCKS5 proxy to a local HTTP proxy with privoxy:
sudo apt update
sudo apt install privoxy
sudo nano /etc/privoxy/config
Add:
listen-address 127.0.0.1:8118
forward-socks5t / 127.0.0.1:1080 .
Restart privoxy and point the HTTP proxy variables at it:
sudo systemctl restart privoxy
unset ALL_PROXY all_proxy
unset HTTP_PROXY HTTPS_PROXY http_proxy https_proxy
export HTTP_PROXY=http://127.0.0.1:8118
export HTTPS_PROXY=http://127.0.0.1:8118
export NO_PROXY=localhost,127.0.0.1,::1
Test the bridge, then run Claude Code:
curl -v --proxy http://127.0.0.1:8118 https://api.anthropic.com
curl -v --proxy http://127.0.0.1:8118 https://claude.ai
claude
If it still fails, check env | grep -i proxy. A common mistake is pointing the HTTP variables at the SOCKS5 port:
HTTPS_PROXY=socks5://127.0.0.1:1080 # wrong for Claude Code
HTTPS_PROXY=http://127.0.0.1:8118 # use the privoxy bridge instead
For browsers, make sure SOCKS5 uses remote DNS. In Firefox, set:
network.proxy.socks_remote_dns = true
For Chrome or Chromium, disabling QUIC may help if HTTPS traffic behaves inconsistently:
chrome://flags/#enable-quic
Shared SSH access
Goal: let members of the hpcusers group reach remote servers (hku, wright) through one account (jinhongd), without giving them that account's private keys.
Layout:
/home/jinhongd/.ssh/config-shared # server aliases, usernames, keys
/usr/local/bin/shared-ssh # runs ssh as jinhongd
/usr/local/bin/remote # convenience command
/etc/sudoers.d/remote-access # permissions
The two wrapper scripts:
sudo tee /usr/local/bin/shared-ssh > /dev/null <<'EOF'
#!/usr/bin/env bash
exec sudo -u jinhongd /usr/bin/ssh -F /home/jinhongd/.ssh/config-shared "$@"
EOF
sudo tee /usr/local/bin/remote > /dev/null <<'EOF'
#!/usr/bin/env bash
exec /usr/local/bin/shared-ssh "$@"
EOF
sudo chmod 755 /usr/local/bin/shared-ssh /usr/local/bin/remote
Allow the group to run exactly these ssh commands as jinhongd, one pair of rules per remote alias:
sudo visudo -f /etc/sudoers.d/remote-access
%hpcusers ALL=(jinhongd) NOPASSWD: /usr/bin/ssh -F /home/jinhongd/.ssh/config-shared hku
%hpcusers ALL=(jinhongd) NOPASSWD: /usr/bin/ssh -F /home/jinhongd/.ssh/config-shared hku *
%hpcusers ALL=(jinhongd) NOPASSWD: /usr/bin/ssh -F /home/jinhongd/.ssh/config-shared wright
%hpcusers ALL=(jinhongd) NOPASSWD: /usr/bin/ssh -F /home/jinhongd/.ssh/config-shared wright *
The * variants are needed because rsync runs SSH with a remote command after the host name.
Fix the file's permissions and validate it:
sudo chmod 0440 /etc/sudoers.d/remote-access
sudo chown root:root /etc/sudoers.d/remote-access
sudo visudo -c
Test:
remote hku
remote wright
rsync -av -e /usr/local/bin/shared-ssh ./data/ hku:~/data/
rsync -av -e /usr/local/bin/shared-ssh ./data/ wright:~/data/