🎯 Objective: explore/clean a huge text/CSV file from the command line without loading it into memory ➔ chain small tools with pipes, then hand the reduced file to R/Python.
⚡ Key Constraint: the pipe | is buffered + line-at-a-time — memory stays bounded so it scales past RAM; >overwrites a file, |feeds the next program — never confuse them.
🧩 Pipeline Anatomy (execution order)
cat big.csv.gz | gunzip | awk -F',' 'NR>1 {print $6,$14}' | sort -n | head# └─source──┘ └decomp┘ └──select cols, skip header──┘ └numeric┘ └peek┘# reads L→R; each stage streams rows to the next as needed (nothing fully in memory)
Order ➔ source → transform → filter → sort → view/save; put cheap filters early (less data flows downstream).
Save instead of view ➔ replace the final head/less with > out.txt to persist the result.
Only the LAST stage sets the exit status ➔ a mid-pipeline failure is invisible; set -o pipefail in a script propagates it.
🧭 Location & pathnames
Tool
Micro-syntax
Job / gotcha
pwd
pwd
print working directory — first move when a command “mysteriously” fails
absolute path
/Users/student/fit2109/week1
starts at root / → start-independent
relative path
data/raw
resolved from the cwd → meaning changes with location
. / .. / ~
cd .. · cd ../.. · cd ~
current dir · parent · home
man
man ls · ls --help
look up every option for a command
shortcuts
Tab · Ctrl-C · ↑
complete name · kill running process (not copy) · recall last command
echo $SHELL
echo "$SHELL"
which interactive shell (Bash $ vs Zsh %); a #!/usr/bin/env bash script runs in Bash regardless
- file, d directory; a - in a slot = permission absent
rwx
read · write · execute
on a directory, x means traverse (needed to cd in)
chmod
chmod u+x file
add execute for owner; changes metadata only, never file contents
ls -l
ls -l file
⚠ run this before guessing why a script won’t run
$PATH
echo $PATH
colon-separated dirs, scanned left to right; . is not on it
bare name
run.sh
a lookup request → command not found if not on $PATH
explicit path
./scripts/run.sh
contains / → skips lookup, used literally
builtin vs external
cd (builtin) vs ls (program on disk)
cd must be builtin — it changes the shell’s own cwd
shebang
#!/usr/bin/env bash
script runs in Bash whatever your interactive shell is
ps / $$
ps · echo $$
process status (pid, ppid, comm) · PID of the current shell
exit status
echo $?
0 = success, nonzero = failure — independent of what was printed
Three signature errors = three stages ➔ command not found (lookup) · permission denied (found, but no x) · no such file or directory (path resolves to nothing). The message is the diagnosis.
Key move:grep -c counts in one step; > persists the filtered rows (use >> to append instead of overwrite).
Practice 3: From app.log, save the six most frequent ERROR messages to top_errors.txt while keeping any error output from the pipeline on screen — then prove the pipeline succeeded.
Reference solution
grep "ERROR" app.log | sort | uniq -c | sort -nr | head -n 6 > top_errors.txtecho $? # 0 -> but this is HEAD's status, not grep'sset -o pipefail # (in a script) make any failing stage propagate
Key move: four tools combined — grep filters, sort makes duplicates adjacent for uniq -c, sort -nr ranks by count, head truncates. > retargets stdout only, so stderr stays visible; and $? reports only the last stage unless pipefail is set.
Practice 4: ./tools/clean.sh prints Permission denied. Diagnose it in two commands and fix it in one.
Reference solution
pwd # 1. am I where I think I am?ls -l tools/clean.sh # 2. -rw-r--r-- <- no 'x' anywherechmod u+x tools/clean.sh # fix: owner gains execute -> -rwxr--r--
Key move:permission denied means the file was found — so this is not a path or lookup problem. chmod edits metadata; the script’s contents are irrelevant to the failure.
⚠️ Common Mistakes
💡 > overwrites, | chains ➔ redirect replaces the target file; pipe feeds the next program.
💡 Mind the delimiter ➔ cut -f assumes tab; for CSV use awk -F',' or cut -d','. Check the real delimiter first (/<tab> search in less).
💡 wc/sort/uniq read the whole file ➔ slow + memory-heavy on huge files; less/head only read the start → instant. Put cheap filters before them.
💡 sort is alphabetical by default ➔ add -n for numbers, and uniq only collapses adjacent duplicates (so sort | uniq).
💡 > does not capture errors ➔ it retargets stdout only; stderr still hits the screen and needs 2>.
💡 Empty output ≠ success ➔ check echo $?; and a pipeline’s status is only its last stage’s.
💡 permission denied ≠ file missing ➔ that is no such file or directory. Match the message to the stage before acting.