Commands

What you type when it matters

A cheat sheet lists flags. This one gives the reason: why this command rather than the obvious alternative, and what the output actually tells you. Most of them exist because something was broken and nobody knew where to look.

126 commands · 10 toolboxes · 18 worth committing to memory

126 of 126

Shell and files

The interview that happens over a screen share. Text processing and process control on a box you do not control. These are the commands that separate people who live in a terminal from people who open one when something breaks.

```
rg -n --hidden -g '!.git' 'pattern' path/
```
pick
Recursive search with line numbers, including dotfiles, skipping the git directory.
ripgrep respects gitignore by default and is roughly an order of magnitude faster than grep on a large tree. The --hidden flag is the one people miss, since config files are usually what you were looking for.
- search
- ripgrep
```
rg -t py -A 3 -B 3 'def embed'
```
Search only Python files, printing three lines of context either side.
Type filters beat glob patterns for speed and for not matching the same string inside node_modules or a lockfile.
- search
- context
```
fd -e parquet -x du -h {} \;
```
Find every Parquet file and print its size.
fd is the ergonomic find. The -x form runs a command per result in parallel, which is how you spot the small files problem in a partitioned dataset.
- files
- find
```
jq -r '.items[] | select(.score > 0.8) | .id' results.json
```
pick
Filter a JSON array by a field and print one id per line.
The default tool for reading eval output and API responses without writing a script. The -r flag strips quotes so the output pipes cleanly.
- json
- filtering
```
jq -c 'select(.status == "error")' traces.jsonl
```
Filter a JSON Lines file, one compact object per line.
JSONL is the usual trace and eval format precisely because it streams. Reach for this before loading a multi-gigabyte log into pandas.
- json
- logs
```
awk -F',' '{s+=$3} END {print s/NR}' metrics.csv
```
Average the third column of a CSV.
Faster than opening a notebook when you want one number from a file, and it works over ssh on a box with no Python.
- text
- arithmetic
```
cut -d' ' -f7 access.log | sort | uniq -c | sort -rn | head -20
```
Top twenty most frequent values in a field.
The oldest analytics pipeline there is, and still the fastest way to find the endpoint or the tenant generating the traffic.
- text
- analysis
```
watch -n 2 'nvidia-smi --query-gpu=memory.used --format=csv'
```
Re-run a command every two seconds and show only the latest output.
For watching a number move during a load test or a model load, without a scrolling wall of output.
- monitoring
```
lsof -i :8000
```
Show which process holds a port.
The first command when a server refuses to start because the address is in use. Investigate the holder before killing it, since it may be work in progress.
- processes
- ports
```
ps aux --sort=-%mem | head
```
Processes ranked by memory.
For the box that started swapping. On a GPU host the answer is often a data loader rather than the model.
- processes
- memory
```
tmux new -s train && tmux attach -t train
```
Start or reattach a named session that survives a dropped connection.
Any training or indexing run over ssh. Losing an eight hour job to a closed laptop lid is a mistake you make once.
- sessions
- remote
```
rsync -avzP --exclude '.git' ./corpus/ user@host:/data/corpus/
```
Sync a directory to a remote host, compressed, resumable, showing progress.
Moving a corpus or checkpoints. -P makes an interrupted transfer resume rather than restart, which matters at tens of gigabytes.
- transfer
- remote
```
du -h --max-depth=1 . | sort -h
```
Directory sizes one level down, ordered.
Finding what filled the disk. Usually a model cache, a checkpoint directory or a log nobody rotated.
- disk
```
python train.py 2>&1 | tee -a run.log
```
Show output live and append it to a file, capturing stderr too.
Any long run you might want to grep afterwards. The 2>&1 is essential because tracebacks and progress bars go to stderr.
- logging
```
cat urls.txt | xargs -P 8 -I{} curl -sO {}
```
Run eight downloads in parallel from a list.
Cheap parallelism without writing concurrency code. Keep the count modest against an API or you will discover its rate limit.
- parallel
- download
```
timeout 300 python eval.py || echo 'timed out'
```
Kill a command after five minutes.
Wrapping anything that calls a model in a batch job, since a hung request otherwise holds a worker until someone notices.
- reliability

Git

History as a debugging tool. Everyone can commit and push. The commands worth knowing are the ones for the day something regressed and nobody remembers when: bisect, blame with line ranges, log with a pickaxe, and the reflog that gets your work back.

```
git bisect start HEAD v1.2.0 && git bisect run ./check.sh
```
pick
Binary search the history for the commit that broke something, automated by a script that exits non-zero on failure.
When a metric regressed and the range is a hundred commits. Ten checks find it. The script can be an eval run, not just a test.
- debugging
- history
```
git log -S 'chunk_size' --oneline
```
Find commits that changed the number of occurrences of a string.
The pickaxe. It answers when did this constant change and who changed it, which grep over the current tree cannot.
- search
- history
```
git blame -L 40,60 -- src/retrieval.py
```
Attribution for a specific line range.
Reading a suspicious block. Add -w to ignore whitespace-only changes so a reformat does not hide the real author.
- attribution
```
git reflog
```
pick
Every position HEAD has held, including states no branch points at any more.
After a reset or a rebase went wrong. Commits are still there for the garbage collection window, and this is how you find them.
- recovery
```
git worktree add ../review-branch feature/x
```
Check out another branch into a second directory sharing one repository.
Reviewing or running a second branch without stashing your work. Better than a clone because history and objects are shared.
- branches
- workflow
```
git diff main...HEAD --stat
```
Summary of what a branch changed relative to where it diverged.
Before opening a pull request. Three dots means since the merge base, which is what you want. Two dots compares tips and will mislead you.
- review
```
git add -p
```
Stage individual hunks interactively.
When one working tree holds two unrelated changes. It is also a review pass over your own diff, which catches stray debug prints.
- staging
```
git stash push -u -m 'wip retrieval'
```
Stash tracked and untracked changes with a label.
The -u matters because new files are otherwise left behind and then get committed to the wrong branch.
- workflow
```
git restore --source=HEAD~3 -- path/to/file
```
Bring one file back from an earlier commit without touching anything else.
Recovering a deleted config or comparing an old implementation. Safer than checking out the whole commit.
- recovery
```
git lfs track '*.safetensors' && git add .gitattributes
```
Route large binaries through Git LFS instead of the object store.
Before the first commit of weights or a large index. Retrofitting means a history rewrite, so decide early.
- large-files
- models

Python and environments

uv, packaging, and the profiler. uv replaced the pip and venv and pyenv stack for most new work because it resolves and installs an order of magnitude faster and manages the interpreter too. The rest of this shelf is the tooling that finds out why a job is slow or leaking.

```
uv init myproject && cd myproject && uv add fastapi
```
pick
Create a project with a pyproject and lockfile, then add a dependency.
uv resolves and installs an order of magnitude faster than pip and manages the interpreter as well, so there is no separate pyenv step.
- packaging
- uv
```
uv run --with pandas python analyse.py
```
Run a script in an ephemeral environment with an extra dependency, without installing it into the project.
One-off analysis. Nothing is added to the lockfile, so the project stays clean.
- uv
- ephemeral
```
uvx ruff check .
```
Run a tool from PyPI without installing it.
Linters, formatters and CLI utilities you use occasionally. Equivalent to pipx run, and much faster.
- uv
- tools
```
uv pip compile requirements.in -o requirements.txt
```
Resolve loose requirements into pinned versions.
Keeping a legacy requirements workflow while getting deterministic builds. Pinning is what makes a container reproducible next month.
- uv
- reproducibility
```
uv python install 3.12 && uv venv --python 3.12
```
Install an interpreter version and create a virtual environment on it.
Reproducing a bug that only appears on one Python version, or matching what the deployment image runs.
- uv
- versions
```
python -m venv .venv && source .venv/bin/activate
```
The standard library environment, no extra tooling.
On a locked-down box where you cannot install uv. Worth knowing because it is always available.
- environments
```
uv pip list --outdated
```
Show installed packages with newer releases available.
Before a dependency bump. Pair it with a lockfile diff so the review shows exactly what moved.
- dependencies
```
ruff check --fix . && ruff format .
```
Lint with autofix, then format.
Replaces flake8, isort and black in one binary, fast enough to run on save and in a pre-commit hook.
- lint
- format
```
mypy --strict src/
```
Static type check.
Most valuable at the boundaries of a pipeline, where a wrong shape or a None reaches production silently. Strict mode on a new module, incremental on an old one.
- types
```
pytest -k 'retrieval and not slow' -x --lf
```
Run a matching subset, stop at the first failure, prioritise last-failed.
The debugging loop. --lf reruns what broke last time, which is almost always what you are fixing.
- testing
```
pytest --durations=10
```
Report the ten slowest tests.
When the suite has crept past the point where anyone runs it locally. Usually two fixtures account for most of it.
- testing
- performance
```
py-spy top --pid 12345
```
pick
Live sampling profiler attached to a running process, no code change and no restart.
A production worker is hot or hung and you cannot reproduce it locally. py-spy dump gives the stack of every thread, which is how you find the deadlock.
- profiling
- production
```
python -m cProfile -s cumtime script.py | head -30
```
Deterministic profile sorted by cumulative time.
For a script you can rerun. High overhead compared with sampling, so do not point it at a live service.
- profiling
```
memray run -o out.bin script.py && memray flamegraph out.bin
```
Track allocations and render a flame graph.
A job whose memory grows over hours. It attributes allocations to lines, including inside native extensions, which tracemalloc cannot.
- memory
- profiling
```
python -W error::DeprecationWarning script.py
```
Turn deprecation warnings into exceptions.
Before a major library upgrade. It surfaces every call site that will break, instead of finding them one at a time in production.
- upgrades

Containers

Build small, run reproducible. Image size and layer caching decide your CI time and your cold start. The debugging commands matter more than the build ones, since production problems arrive as a container that exited with no useful log.

```
docker build --progress=plain -t app:dev .
```
Build with full unfolded log output.
When you need to see which layer missed the cache. The default compact output hides exactly that.
- build
- cache
```
docker buildx build --platform linux/amd64,linux/arm64 -t app:1.0 --push .
```
pick
Build and push a multi-architecture image.
Developing on Apple silicon and deploying to x86. Without this the image runs under emulation, or does not run at all.
- build
- multiarch
```
docker run --gpus all --rm -it -v $(pwd):/work -w /work nvcr.io/nvidia/pytorch:24.10-py3 bash
```
Interactive GPU container with the working directory mounted.
The standard shape for reproducible training and inference. --gpus all needs the NVIDIA container toolkit on the host.
- gpu
- run
```
docker exec -it <container> sh
```
Shell into a running container.
First move when a service is up but wrong. If the image is distroless there is no shell, which is when you use debug below.
- debugging
```
docker run --rm -it --pid=container:<id> --network=container:<id> nicolaka/netshoot
```
Attach a toolbox container to another container's process and network namespaces.
Debugging a minimal image that has no shell, curl or dig. You get the tools without rebuilding the image.
- debugging
- network
```
docker logs --tail 100 -f --timestamps <container>
```
Follow the last hundred lines with timestamps.
Timestamps matter more than they look, because correlating a container log with a trace is how you find the slow step.
- logs
```
docker stats --no-stream
```
One-shot CPU, memory and network per container.
Checking whether a container is near its memory limit before it gets killed with an unhelpful exit code 137.
- monitoring
```
docker history --no-trunc app:1.0
```
Per-layer sizes and the commands that created them.
Finding the layer that made a two gigabyte image. Usually a pip install of CUDA wheels or a COPY that pulled in the whole context.
- size
- optimization
```
printf '.git\n.venv\ndata/\n*.safetensors\n' >> .dockerignore
```
Keep large and irrelevant paths out of the build context.
A slow build that starts with sending build context is this problem. It also stops secrets in a local env file from entering an image layer.
- build
- security
```
docker system df && docker system prune -a --filter 'until=168h'
```
Report reclaimable space, then remove images and caches older than a week.
Check the report before pruning. On a machine with model images the cache is large but expensive to rebuild.
- cleanup
```
docker compose up -d --build && docker compose logs -f api
```
Rebuild and start the stack detached, then follow one service.
Local development with a vector store, a database and an app. The dependency ordering it gives you is the reason to prefer it over several run commands.
- compose
- local

Kubernetes

Triage under pressure. You are unlikely to be asked to write a manifest from memory. You are quite likely to be asked what you would type first when a GPU pod is stuck in Pending, and there is a correct sequence.

```
kubectl describe pod <pod>
```
pick
Full state including the event list at the bottom.
The first command for a pod that is not running. The events explain Pending, ImagePullBackOff and OOMKilled, and logs will not.
- triage
```
kubectl logs <pod> -c <container> --previous --tail=200
```
pick
Logs from the instance before the last restart.
A crash-looping pod. The current instance has nothing useful. --previous holds the traceback that caused the restart.
- triage
- logs
```
kubectl get pods -o wide --sort-by=.status.startTime
```
Pods with node placement and IPs, newest last.
Working out whether failures cluster on one node, which is the usual shape of a bad GPU or a full disk.
- triage
```
kubectl top pod --sort-by=memory
```
Live resource usage per pod.
Comparing actual usage against requests and limits. Requests that are far above usage are what makes a GPU cluster look full when it is not.
- resources
```
kubectl get events --sort-by=.lastTimestamp -A | tail -40
```
Recent cluster events across all namespaces, newest last.
When something is wrong but you do not yet know which workload. Scheduling failures and evictions show here first.
- triage
```
kubectl port-forward svc/vllm 8000:8000
```
Tunnel a cluster service to a local port.
Testing a model endpoint without exposing it. It bypasses the ingress, which is how you determine whether a problem is the ingress.
- access
- debugging
```
kubectl exec -it <pod> -- nvidia-smi
```
Run a command inside a running pod.
Confirming the container actually sees the GPU it was scheduled onto. A missing device plugin produces a pod that runs and never uses the accelerator.
- debugging
- gpu
```
kubectl rollout status deploy/api --timeout=5m
```
Block until a rollout completes or the timeout expires.
In a deploy script, so a broken release fails the pipeline instead of silently leaving old pods serving.
- deploy
```
kubectl rollout undo deploy/api
```
Roll back to the previous revision.
The command you want to have typed once before you need it at three in the morning. Confirm the revision with rollout history first.
- deploy
- rollback
```
kubectl scale deploy/worker --replicas=0
```
Scale a deployment, including to zero.
Stopping a runaway consumer or freeing GPUs without deleting the deployment and its config.
- scaling
```
kubectl diff -f manifest.yaml
```
Show what applying a manifest would change.
Before every apply against a shared cluster. It catches the field you did not mean to change and the drift someone applied by hand.
- safety
- review
```
kubectl get nodes -o custom-columns=NAME:.metadata.name,GPU:.status.allocatable.'nvidia\.com/gpu'
```
Allocatable GPUs per node.
A pod stuck in Pending with an insufficient GPU message. This tells you whether capacity exists or the device plugin is not reporting.
- gpu
- capacity

Cloud and infrastructure

Storage, identity, and the bill. Enough of each provider CLI to move a corpus, check who can read it, and find out what a workload cost. Plus the Terraform verbs, because infrastructure changes get reviewed as diffs.

```
aws s3 sync ./corpus s3://bucket/corpus --exclude '*.tmp' --storage-class INTELLIGENT_TIERING
```
Upload only what changed, with a storage class chosen at write time.
Moving a corpus. Setting the storage class on write is far cheaper than a lifecycle transition later.
- aws
- storage
```
aws s3 ls s3://bucket/prefix/ --recursive --summarize --human-readable | tail -3
```
Object count and total size under a prefix.
Sizing a reindex or explaining a storage bill. The count matters as much as the bytes, because request charges track objects.
- aws
- cost
```
aws sts get-caller-identity
```
pick
Which identity the current credentials resolve to.
The first command when something returns access denied. Usually you are in the wrong profile or an assumed role expired.
- aws
- identity
```
aws logs tail /aws/lambda/fn --follow --since 10m --filter-pattern ERROR
```
Stream CloudWatch logs filtered to errors.
Live debugging without the console. --since is what keeps it from replaying a week of logs.
- aws
- logs
```
aws ce get-cost-and-usage --time-period Start=2026-07-01,End=2026-08-01 --granularity MONTHLY --metrics BlendedCost --group-by Type=DIMENSION,Key=SERVICE
```
Monthly spend grouped by service.
Answering where the money went with a number rather than a guess. Group by tag instead when workloads are tagged per team.
- aws
- cost
```
gcloud config list && gcloud auth list
```
Active project, region and account.
Same role as get-caller-identity. Most confusing gcloud errors are a wrong active project.
- gcp
- identity
```
gcloud storage rsync -r ./data gs://bucket/data
```
Recursive sync to Cloud Storage.
gcloud storage replaced gsutil and is substantially faster on many small files, which is the shape of a chunked corpus.
- gcp
- storage
```
az account show --output table
```
Current subscription and tenant.
Before anything that writes. Azure CLI happily targets the wrong subscription if you have several.
- azure
- identity
```
az storage blob upload-batch -d container -s ./corpus --account-name acct
```
Batch upload a directory to blob storage.
The Azure equivalent of s3 sync. Use azcopy for large transfers, it parallelises better.
- azure
- storage
```
terraform plan -out=tf.plan && terraform apply tf.plan
```
pick
Save a plan, then apply exactly that plan.
Applying a saved plan removes the gap between what was reviewed and what runs. Applying without a saved plan re-resolves state and can do something else.
- terraform
- safety
```
terraform state list | grep gpu
```
Enumerate managed resources.
Finding out what Terraform believes it owns before an import or a targeted destroy.
- terraform
```
terraform fmt -recursive && terraform validate
```
Format and check syntax and internal consistency.
In CI before plan. It fails in seconds on the errors that would otherwise fail after a slow provider refresh.
- terraform
- ci

Data and query

Answering a question about a file. DuckDB reads Parquet, CSV and JSON from local disk or object storage with no server, which makes it the fastest path from a file to an answer. The rest is psql and the JSON and CSV tools you reach for before writing a script.

```
duckdb -c "SELECT count(*), avg(score) FROM 'data/*.parquet'"
```
pick
Query a directory of Parquet files directly, no import and no server.
The fastest path from files to an answer. Glob patterns work, and only the referenced columns are read.
- duckdb
- parquet
```
duckdb -c "INSTALL httpfs; LOAD httpfs; SELECT * FROM 's3://bucket/events/*.parquet' LIMIT 10"
```
Query object storage in place.
Inspecting a remote dataset without downloading it. Credentials come from the environment or a SET commands block.
- duckdb
- s3
```
duckdb -c "DESCRIBE SELECT * FROM 'file.parquet'"
```
Inferred schema and types.
Before writing the real query. Catches the id column that arrived as a string and would have silently broken a join.
- duckdb
- schema
```
duckdb -c "COPY (SELECT * FROM 'big.csv') TO 'big.parquet' (FORMAT PARQUET, COMPRESSION ZSTD)"
```
Convert CSV to compressed Parquet.
First step on any dataset you will query more than once. Expect a large size reduction and much faster scans.
- duckdb
- conversion
```
docker run --rm -it -v "$(pwd):/workspace" -w /workspace duckdb/duckdb
```
DuckDB in a container with the current directory mounted.
On a machine where you cannot install it, or to pin a version in CI.
- duckdb
- containers
```
psql -c 'EXPLAIN (ANALYZE, BUFFERS) SELECT ...'
```
pick
Real execution plan with timings and buffer counts.
The answer to why is this query slow. ANALYZE runs it, so be careful on writes. Buffers show whether it hit cache or disk.
- postgres
- performance
```
psql -c "CREATE INDEX ON items USING hnsw (embedding vector_cosine_ops) WITH (m = 16, ef_construction = 64)"
```
Build an HNSW index in pgvector with explicit graph parameters.
m controls links per node and therefore memory, ef_construction controls build quality and time. Defaults are conservative, and both should be chosen against a measured recall target.
- postgres
- vectors
- index
```
psql -c "SELECT pid, state, wait_event, query FROM pg_stat_activity WHERE state != 'idle'"
```
What is running right now and what it is waiting on.
During an incident. It identifies the blocking query and the lock, which is usually a migration someone started.
- postgres
- triage
```
psql -c "\copy items FROM 'items.csv' WITH (FORMAT csv, HEADER)"
```
Bulk load a CSV from the client machine.
Orders of magnitude faster than row-by-row inserts. The backslash form runs client side, so it works without server filesystem access.
- postgres
- loading
```
mlr --icsv --ojson head -n 5 data.csv
```
Convert and inspect structured text between CSV, TSV and JSON.
Miller is awk that understands headers. Useful for a quick reshape before loading, without a pandas import.
- csv
- conversion
```
wc -l traces.jsonl && head -1 traces.jsonl | jq keys
```
Row count and the field names of a JSONL file.
The two things to know before writing any processing for an unfamiliar export.
- json
- inspection

Models and inference

Pull, serve, quantize, measure. The hub CLI, the three serving engines, and the benchmark commands that turn a claim about throughput into a number. The hf CLI replaced the deprecated huggingface-cli, so read older tutorials with that in mind.

```
hf download meta-llama/Llama-3.1-8B-Instruct --local-dir ./models/llama-8b
```
pick
Download a model repository from the Hub with resumable parallel transfers.
The hf CLI replaced the deprecated huggingface-cli. One package gives you the CLI and the huggingface_hub library, so older tutorials showing huggingface-cli still map one to one.
- huggingface
- download
```
hf download org/model --include '*.safetensors' '*.json' --exclude '*.bin'
```
Fetch only the files you need.
Many repositories ship both safetensors and legacy bin weights. Filtering halves the download and the disk.
- huggingface
- efficiency
```
hf auth login
```
Store a Hub token for gated repositories and uploads.
Required for gated model families. In CI, pass the token by environment variable rather than running an interactive login.
- huggingface
- auth
```
hf upload org/my-adapter ./out --repo-type model
```
Push a local directory to a Hub repository.
Publishing a LoRA adapter or a dataset. Check the repository is private first if the training data was not public.
- huggingface
- publish
```
hf cache scan
```
Report what the local Hub cache holds and its size.
The usual answer to a full disk on a workstation. Pair with hf cache delete to remove specific revisions.
- huggingface
- disk
```
vllm serve meta-llama/Llama-3.1-8B-Instruct --port 8000 --max-model-len 8192
```
pick
Start an OpenAI-compatible server with continuous batching and paged attention.
The default self-hosted serving path. Capping max-model-len is what stops the KV cache reservation from eating the memory you wanted for concurrency.
- vllm
- serving
```
vllm serve <model> --tensor-parallel-size 4 --gpu-memory-utilization 0.92
```
Shard across four GPUs and set the fraction of device memory the engine may claim.
Tensor parallelism is for models that do not fit on one device. The utilisation figure is a trade between KV cache room and headroom against fragmentation.
- vllm
- distributed
```
vllm serve <model> --enable-prefix-caching --max-num-seqs 256
```
Reuse the KV cache for shared prompt prefixes and set the concurrency ceiling.
Any workload with a long shared system prompt or repeated documents. max-num-seqs is the throughput and latency dial, and the honest way to set it is to measure goodput.
- vllm
- cost
- latency
```
vllm serve --help=max
```
Keyword search the flag reference instead of paging through all of it.
The CLI supports --help=all and section forms such as --help=ModelConfig, plus keyword matching. Faster than the documentation site for finding the exact flag name.
- vllm
- reference
```
vllm serve TheBloke/model-AWQ --quantization awq --dtype half
```
Serve a pre-quantized checkpoint.
Fitting a larger model on the same card. Run your own evaluation against the unquantized version rather than trusting a reported benchmark average.
- vllm
- quantization
```
ollama run llama3.1:8b
```
Pull if needed and start an interactive session.
The lowest-friction local path. Good for prompt iteration, not for serving concurrent users.
- ollama
- local
```
ollama ps
```
Show which models are currently loaded and how much memory they hold.
Working out why the GPU is full. Models stay resident after a session ends until the keep-alive expires.
- ollama
- memory
```
OLLAMA_HOST=0.0.0.0:11434 ollama serve
```
Run the API server bound to all interfaces.
Reaching it from a container or another machine on the network. Do not expose it to the internet, there is no authentication.
- ollama
- serving
- security
```
ollama ls && ollama rm <model>
```
List local models and delete one.
Quantized weights accumulate fast. This is the first place to look when a laptop runs out of space.
- ollama
- disk
```
llama-server -hf ggml-org/gemma-3-1b-it-GGUF -c 4096 --port 8080
```
Start llama.cpp's OpenAI-compatible server, pulling GGUF weights from the Hub.
CPU and Apple silicon inference, or a single-user box with no CUDA. -c sets context length and directly sets the KV cache size.
- llama.cpp
- local
```
llama-cli -hf <repo> -t 8 -ngl 99
```
One-shot generation with eight CPU threads and as many layers as possible offloaded to the GPU.
Hybrid CPU and GPU execution is llama.cpp's distinguishing feature. -ngl is the dial for how much of the model fits on the accelerator.
- llama.cpp
- hybrid
```
llama-quantize model-f16.gguf model-q4_k_m.gguf Q4_K_M
```
Quantize a GGUF to a mixed 4-bit variant.
Q4_K_M is the usual balance point for local use. Measure on your own prompts, because the loss shows up on long-tail inputs before it shows up on an average.
- llama.cpp
- quantization
```
vllm bench serve --model <model> --dataset-name random --request-rate 8 --num-prompts 500
```
pick
Load test a running server and report TTFT, TPOT and throughput percentiles.
Turning a claim about capacity into a number. Sweep the request rate and find where p99 TTFT crosses your SLO, since that point is the real capacity.
- benchmark
- capacity
```
lm_eval --model hf --model_args pretrained=<model> --tasks mmlu,gsm8k --batch_size auto
```
Run standard academic benchmarks against a local model.
Sanity-checking that a quantization or a fine-tune did not break general capability. It does not tell you anything about your task, so it complements a golden set rather than replacing one.
- evaluation
- benchmark

GPU and profiling

Finding out what is actually saturated. Utilisation percentage is the most misread number in the stack: it says a kernel was resident, not that the device was busy. These commands get you to memory, bandwidth, interconnect and the kernel timeline.

```
nvidia-smi
```
pick
Devices, driver and CUDA version, memory used, utilisation, and the processes holding each GPU.
The first command on any GPU box. Utilisation percentage means a kernel was resident during the sample, not that the device was saturated, so do not tune against it alone.
- monitoring
```
nvidia-smi --query-gpu=timestamp,memory.used,utilization.gpu,power.draw --format=csv -l 1
```
Machine-readable sampling once per second.
Logging a training or benchmark run for later analysis. Power draw is a better saturation signal than utilisation.
- monitoring
- logging
```
nvidia-smi pmon -c 30
```
Per-process GPU usage sampled over thirty intervals.
Two jobs share a device and you need to know which one is doing the work.
- monitoring
- processes
```
nvtop
```
Interactive multi-GPU monitor with history graphs.
Watching a training run. The graph shape reveals input-pipeline stalls that a point-in-time reading hides.
- monitoring
```
nvidia-smi topo -m
```
Interconnect matrix showing which GPU pairs share NVLink and which cross a PCIe bridge.
Before setting tensor parallel size. Sharding across a slow link makes a model slower than it was, and this is where you find out.
- topology
- distributed
```
python -c 'import torch; print(torch.__version__, torch.cuda.is_available(), torch.cuda.get_device_name(0))'
```
Confirm the framework sees the device.
The mismatch between the driver, CUDA and the wheel is the most common setup failure, and it presents as silent CPU execution.
- setup
- pytorch
```
python -c 'import torch; print(torch.cuda.memory_summary())'
```
Allocator statistics including reserved versus allocated and fragmentation.
An out-of-memory error with apparently free memory is fragmentation. The gap between reserved and allocated is the number to read.
- memory
- pytorch
```
python -m torch.utils.bottleneck script.py
```
Combined CPU and CUDA profile identifying whether the bottleneck is the data pipeline or the model.
Low GPU utilisation during training. The answer is usually the data loader, not the model.
- profiling
- pytorch
```
nsys profile -o report python train.py
```
System-wide timeline of kernels, memory transfers and CPU activity.
When you need to see gaps between kernels. Those gaps are the launch overhead and synchronisation stalls that a summary profiler averages away.
- profiling
- nsight
```
dcgmi diag -r 1
```
Quick health diagnostic on the installed GPUs.
Suspected hardware fault after repeated ECC errors or a job that fails on one node only. Run before blaming the code.
- diagnostics
- hardware

Network and APIs

Proving where the latency is. Timing breakdowns, streaming responses, load generation and the DNS and TLS checks that resolve the argument about whose service is slow.

```
curl -o /dev/null -s -w 'dns %{time_namelookup} tcp %{time_connect} tls %{time_appconnect} ttfb %{time_starttransfer} total %{time_total}\n' https://api.example.com/v1/health
```
pick
Break a request into DNS, TCP, TLS and first-byte phases.
The command that settles an argument about whose service is slow. If DNS or TLS dominates, the API is not the problem.
- latency
- diagnosis
```
curl -N -H 'Content-Type: application/json' -d '{"model":"m","messages":[{"role":"user","content":"hi"}],"stream":true}' http://localhost:8000/v1/chat/completions
```
Post a streaming chat completion with buffering disabled.
-N is what makes server-sent events arrive as they are produced. Without it you see the whole response at once and cannot observe TTFT.
- llm-api
- streaming
```
curl --retry 5 --retry-delay 2 --retry-all-errors --max-time 60 <url>
```
Retry with a delay and an overall timeout.
Scripted calls in a pipeline. A total timeout matters as much as the retry count, since retries multiply the worst case.
- reliability
```
hey -n 500 -c 20 -m POST -H 'Content-Type: application/json' -D body.json http://localhost:8000/v1/chat/completions
```
Five hundred requests at twenty concurrent, reporting a latency distribution.
A quick capacity check. Read p95 and p99 and ignore the mean, since the tail is what users experience and what an SLO is written against.
- load-testing
```
dig +short api.example.com && dig +trace api.example.com | tail -20
```
Resolve a name, then show the full delegation path.
Intermittent connection failures after a DNS change. The trace shows which resolver in the chain is still serving the old record.
- dns
```
openssl s_client -connect api.example.com:443 -servername api.example.com </dev/null 2>/dev/null | openssl x509 -noout -dates -subject
```
Certificate subject and validity dates.
A client failing TLS verification. Also worth checking before an expiry causes an outage nobody planned for.
- tls
```
ss -tunap | grep 8000
```
Sockets on a port with the owning process.
Replaces netstat. A large receive queue on a listener means the application is not accepting fast enough, which is a backpressure signal.
- sockets
- diagnosis
```
mitmproxy --mode reverse:https://api.openai.com -p 8081
```
Inspect requests and responses passing through a local proxy.
Seeing exactly what an SDK sends, including headers and the token counts in the response. Never point it at traffic carrying production credentials you do not own.
- debugging
- api
```
sudo tcpdump -i any -nn port 8000 -c 100 -w capture.pcap
```
Capture a hundred packets on a port to a file.
Last resort when logs on both sides disagree about whether a request arrived. Read the capture in Wireshark rather than the terminal.
- packets
- diagnosis
```
until curl -sf http://localhost:8000/health; do sleep 2; done
```
Block until a service reports healthy.
In a script that starts a model server before running an evaluation. Weights take minutes to load, and a fixed sleep is either too short or wasteful.
- scripting
- readiness