Linux — Basics

Linux — Basics for Backend / DevOps

Filesystem hierarchy (FHS)

Path	Purpose
/etc	system config
/var/log	logs
/var/lib	persistent app state
/usr/bin, /usr/sbin	binaries
/usr/local	locally installed (not from package)
/opt	third-party apps
/home/<user>	home dirs
/tmp, /var/tmp	temp
/proc, /sys	virtual fs (kernel state)
/dev	device files

Permissions

-rwxr-xr-- = file (or d directory) + owner/group/other rwx.

• Octal: 4=r, 2=w, 1=x. chmod 755, 644, 600.
• chown user:group path.
• setuid/setgid/sticky (4/2/1 in front: 4755). Sticky on dir = only owner can delete (e.g. /tmp).
• ACLs (getfacl/setfacl) for fine-grained beyond user/group/other.

Processes

• Each process: PID, PPID, uid, gid, working dir, fds, env.
• Signals: HUP(1), INT(2), QUIT(3), KILL(9), TERM(15), STOP(19), CONT(18), USR1(10), USR2(12).
• Fork/exec: child created via fork (copy-on-write), then exec replaces image.
• Foreground / background (&), jobs (jobs, fg, bg).
• nohup / disown / setsid / systemd-run for daemonizing.

File descriptors

• 0=stdin, 1=stdout, 2=stderr.
• Redirection: > file stdout, 2> err stderr, &> all, < in.
• 2>&1 merges. tee splits.
• /proc/<pid>/fd shows process fds.
• ulimit -n controls max.

Networking

ip a; ip route; ip link
ss -tnlp; ss -tnp
ping host; traceroute host; mtr host
dig name; nslookup name
curl -v http://...
nc -zv host port           # port check
iptables -L; nft list ruleset

Init systems

• systemd (default everywhere): units (.service, .timer, .socket).
• systemctl start/stop/status/enable/restart unit.
• journalctl -u unit -f — logs.
• Timer units replace cron.

Service file

[Unit]
Description=API
After=network.target

[Service]
ExecStart=/usr/local/bin/api --port 8080
Restart=on-failure
User=api
Group=api
Environment=NODE_ENV=production
LimitNOFILE=65536
ProtectSystem=strict
ProtectHome=true
ReadOnlyPaths=/

[Install]
WantedBy=multi-user.target

Cron

# m h dom mon dow command
0 2 * * * /usr/local/bin/backup.sh
*/5 * * * * /usr/local/bin/check.sh

Disk + filesystem

df -h; du -sh *; du -h --max-depth=1
lsblk; blkid
mount; /etc/fstab
e2fsck, xfs_repair (offline FS check)
fallocate -l 1G file        # preallocate

Resource limits

ulimit -n      # max FDs
ulimit -u      # max processes
ulimit -m      # memory
ulimit -c      # core dump size

In /etc/security/limits.conf for persistent.

Performance tools (one-line)

Tool	What
top/htop	overview
vmstat 1	CPU/IO/swap by second
iostat -x 1	per-disk IO
mpstat -P ALL 1	per-CPU
pidstat 1	per-process
iotop	per-process IO
nethogs/iftop	per-process net
ss -s	socket summary
dstat	one-stop
perf top	sampled CPU profile
strace -p PID	syscalls
ltrace -p PID	library calls
lsof -p PID	open files
bpftrace / bcc	eBPF tracing

Logging

• journalctl — systemd journal.
• /var/log/syslog, /var/log/messages, /var/log/auth.log.
• logrotate config in /etc/logrotate.d/.
• Centralize: rsyslog → ELK / Loki / CloudWatch.

Package mgmt

• Debian/Ubuntu: apt, apt-cache, dpkg.
• RHEL/Fedora: dnf, yum, rpm.
• Alpine: apk.
• Arch: pacman.

SSH

ssh -i ~/.ssh/id_ed25519 user@host
ssh -L 5432:db.internal:5432 jump.host       # local forward
ssh -R 8080:localhost:8080 jump.host         # reverse forward
ssh-add; ssh-agent
~/.ssh/config:
  Host bastion
    HostName bastion.example.com
    User ubuntu
    IdentityFile ~/.ssh/id_ed25519
  Host db
    ProxyJump bastion
    User postgres

Useful one-liners

# top processes by mem
ps -eo pid,user,%mem,%cpu,cmd --sort=-%mem | head

# count requests by status from log
awk '{print $9}' access.log | sort | uniq -c | sort -rn

# tail huge log without loading
tail -f --retry log

# replace in many files
grep -rl 'old' . | xargs sed -i 's/old/new/g'

# find big files
find / -type f -size +500M 2>/dev/null

# what's listening
sudo ss -tnlp

# which process opened a file
sudo lsof /path/to/file

Common interview Qs

1. Difference between hard and soft link. Hard = same inode. Soft = pointer; works across fs.
2. A pod is OOMKilled — how to investigate on the node? dmesg | grep -i oom, journalctl -k, container memory metrics.
3. Disk full — what to look for? du -sh /var/log/*, journalctl --vacuum-size=500M, deleted-but-open files lsof | grep deleted.
4. High load avg — what does it mean? Mean number of runnable + uninterruptible processes. Compare to core count.
5. Process hung — what’s it doing? cat /proc/PID/wchan, strace -p PID, cat /proc/PID/stack.
6. You see lots of TIME_WAIT — concern? Client-side ephemeral port exhaustion possible. Mitigations: persistent conns, tcp_tw_reuse.
7. What is kswapd? Kernel swapper; high CPU = under memory pressure → swap activity.