Automation and scripting: Python, Bash and PowerShell

In short

Scripting means writing short interpreted programs that drive the operating system, files, processes and other programs instead of being compiled applications. Python is the general-purpose glue language, Bash the native shell of Linux and Unix servers, and PowerShell the object-oriented shell of Windows and Azure administration. Indian university syllabi examine this as "Scripting Languages", "Shell Programming" or lab work under Operating Systems and Python Programming; GATE CS touches it only through process/file concepts, but placement rounds at TCS, Infosys, Wipro, Cognizant, Accenture, Capgemini, HCLTech and every DevOps or SRE role test it directly. It is the day-one skill for system administration, CI/CD, data cleaning, test automation and RPA.

Explained

What a script is and how it runs

• An interpreted program: the interpreter (python, bash, pwsh) reads source text and executes it statement by statement - no separate compile-link step, so edit-run cycles are fast.
• Shebang line (#!/usr/bin/env python3, #!/bin/bash) in the first line tells the Unix kernel which interpreter to use; the file must be executable (chmod +x script.sh).
• Windows has no shebang: a .ps1 file is run by PowerShell, and the execution policy (Restricted by default on clients) must allow it - Set-ExecutionPolicy -Scope CurrentUser RemoteSigned, or run once with powershell -ExecutionPolicy Bypass -File s.ps1.
• Exit status: 0 means success, 1-255 failure. Read it as $? in Bash, $LASTEXITCODE (numeric) or $? (boolean) in PowerShell, and in Python as subprocess.run(...).returncode - sys.exit(code) *sets* your own script's status. Schedulers and CI pipelines branch on this.
• Streams: stdin (0), stdout (1), stderr (2). Redirect with >, >>, 2>, 2>&1, <; connect with the pipe |.
• Arguments: Bash $0 $1 ... $# $@; PowerShell param() block or $args; Python sys.argv or the argparse module.
• Environment variables: $HOME, $PATH in Bash; $env:PATH in PowerShell; os.environ in Python. Secrets belong here or in a vault, never hardcoded in the script.

Bash (Linux/Unix shell scripting)

• Variables: name="Ravi" (no spaces around =), used as $name or ${name}; quoting matters - "$var" preserves spaces, '$var' is literal.
• Command substitution: today=$(date +%F); arithmetic: $(( a + b )) (integers only - use bc -l or awk for decimals).
• Conditionals: if [ "$n" -gt 10 ]; then ... elif ... else ... fi; [[ ]] is the Bash-only safer test with pattern matching and &&/||. Numeric operators -eq -ne -lt -le -gt -ge; string = != -z -n; file tests -f -d -e -r -w -x -s.
• Loops: for f in *.csv; do ... done, while read -r line; do ... done < file, until, case ... esac; break, continue.
• Functions: backup() { ... }, called as backup arg1; arguments are $1 $2, return value is an exit status (return 0), output is captured via $( ).
• Core text toolchain: grep (search by regex), sed (stream edit/substitute), awk (field-wise reports), cut, sort, uniq -c, tr, wc, head, tail -f, find, xargs, tee.
• Robustness: start with set -euo pipefail (exit on error, on unset variable, and catch failures mid-pipeline); trap 'cleanup' EXIT INT TERM; getopts for -f value style options.
• Jobs: & background, nohup/disown to survive logout, jobs, fg, bg, kill -TERM pid, ps aux, df -h, crontab -e.

PowerShell (Windows and cross-platform)

• Cmdlets follow a Verb-Noun pattern: Get-Process, Copy-Item, Stop-Service, Get-ChildItem. Aliases (ls, dir, cat, cd) ease the transition.
• Object pipeline - the key difference from Bash: cmdlets pass .NET objects, not text, so no parsing is needed: Get-Process | Where-Object CPU -gt 100 | Sort-Object CPU -Descending | Select-Object Name, CPU.
• Variables and types: $n = 5, [int]$x = "7", arrays @(1,2,3), hashtables @{Name="A"; Age=2}; $null, $true, $false.
• Operators are worded: -eq -ne -gt -lt -ge -le -like -match -contains -in -notmatch; -match applies regex and fills $matches.
• Control flow: if/elseif/else, switch, for, foreach ($i in $list), while, do..until, ForEach-Object (%) in the pipeline.
• Functions and scripts: function Get-Report { param([string]$Path, [switch]$Verbose) ... }; [CmdletBinding()] gives common parameters; modules (.psm1) and Install-Module from the PowerShell Gallery.
• Error handling: try { } catch { } finally { } with -ErrorAction Stop; $ErrorActionPreference; Write-Error, Write-Verbose, throw.
• Admin reach: WMI/CIM (Get-CimInstance Win32_LogicalDisk), registry (Get-ItemProperty HKLM:\...), Active Directory, Exchange, Azure (Az module), remoting (Invoke-Command -ComputerName), scheduled tasks (Register-ScheduledTask), Export-Csv/ConvertTo-Json.
• Versions: Windows PowerShell 5.1 ships with Windows (Desktop .NET); PowerShell 7.x (pwsh) is the current cross-platform open-source edition and runs on Linux and macOS too.

Python for automation

• Standard library modules: os, os.path, pathlib (object-oriented paths), shutil (copy/move/archive), glob, sys, subprocess (run external commands), re (regex), csv, json, datetime, time, logging, argparse, configparser, zipfile, smtplib, sqlite3, unittest.
• Running commands: subprocess.run(["ls","-l"], capture_output=True, text=True, check=True) - pass a list, avoid shell=True (command-injection risk).
• File walking: for p in pathlib.Path("data").rglob("*.csv"): ... replaces nested os.walk loops.
• Context managers: with open(f, newline="", encoding="utf-8") as fh: guarantees the file closes even on exception.
• Third-party workhorses: requests (HTTP/REST), pandas + openpyxl (Excel/CSV), BeautifulSoup/Scrapy (scraping), selenium/playwright (browser automation), paramiko/fabric (SSH), boto3 (AWS), pywin32/pyautogui (Windows GUI/COM), schedule/APScheduler, pytest, psutil.
• Packaging the job: python -m venv .venv, pip install -r requirements.txt, pyinstaller for a single .exe for non-technical users.
• Logging over printing: logging.basicConfig(filename="run.log", level=logging.INFO) so unattended runs leave a trail.

Scheduling, triggering and orchestration

• cron (Linux): five fields minute hour day-of-month month day-of-week then the command. */15 * * * * = every 15 minutes. @reboot, @daily shortcuts. crontab -l/-e/-r; use absolute paths because cron's environment is minimal. systemd timers are the modern alternative; at runs a one-off.
• Windows Task Scheduler: triggers (time, logon, event, idle) + actions; scriptable via schtasks or Register-ScheduledTask. Set "Run whether user is logged on or not" for unattended jobs.
• Event-driven: file-system watchers (inotify, FileSystemWatcher, watchdog), webhooks, message queues, Git hooks.
• CI/CD: Jenkins, GitLab CI, GitHub Actions and Azure Pipelines all execute Bash/PowerShell/Python steps in YAML; non-zero exit fails the stage.
• Configuration management and IaC: Ansible (YAML playbooks over SSH, Python-based), Terraform, Docker ENTRYPOINT scripts, Kubernetes jobs.
• RPA: UiPath, Automation Anywhere and Blue Prism (all with large Indian delivery centres) embed Python/VBScript for logic that GUI robots cannot express.

Comparison, strengths and limits

• Bash is unbeatable for gluing command-line tools on Linux in five lines, but has no real data types, fragile quoting, integer-only arithmetic and poor error handling for anything over ~200 lines.
• PowerShell gives typed objects, structured errors and deep Windows/Azure APIs; cost is verbosity, Windows-centric history and execution-policy/signing friction.
• Python has data structures, exceptions, testing, libraries and portability - best once logic, parsing or APIs are involved; cost is interpreter/dependency installation and slower startup than a shell builtin.
• Rule of thumb used in industry: if it is mostly invoking commands and filters, script the shell; if it has loops over structured data, conditions and parsing, write Python.
• All three are slow for CPU-bound work (interpreted, GIL in Python) - they orchestrate fast programs rather than replace them.

Security, compliance and good practice in Indian IT

• Never hardcode passwords, API keys or Aadhaar/PAN-type personal data in scripts; use environment variables, .env excluded by .gitignore, Windows Credential Manager, HashiCorp Vault or AWS Secrets Manager.
• Quote every variable in Bash and avoid eval/shell=True to prevent command injection; validate input before passing it to the OS.
• Personal data handled by automation falls under the Digital Personal Data Protection Act, 2023 (rules notified in phases from 2025) - minimise, mask in logs, and delete per retention policy. Unauthorised access or data theft via scripts is punishable under the Information Technology Act, 2000 (sections 43 and 66); CERT-In's 2022 directions require 180-day log retention by service providers and incident reporting within six hours.
• Make scripts idempotent (safe to re-run), use dry-run flags, back up before destructive operations, and never run rm -rf "$DIR"/ without checking $DIR is set.
• Keep scripts in Git with comments, a header block, versioning and a README - the deliverable is reproducible on another machine.

Key points

• Script - interpreted program that automates the OS, files and other programs; no compile step.
• Shebang - #!interpreter on line 1 selects the interpreter on Unix; irrelevant on Windows.
• Exit code - 0 = success, non-zero = failure; read via $?, $LASTEXITCODE, returncode; set via sys.exit()/exit.
• Pipe - Bash pipes bytes/text; PowerShell pipes .NET objects; Python uses subprocess pipes.
• set -euo pipefail - the single most valuable line in a production Bash script.
• Quoting - "$var" expands safely, '$var' is literal, backticks/$( ) substitute command output.
• Text trio - grep finds, sed substitutes, awk computes on fields.
• cron fields - minute, hour, day-of-month, month, day-of-week (*/5 = every 5 units).
• Verb-Noun - every PowerShell cmdlet name; Get-Help, Get-Command, Get-Member are the discovery trio.
• Object pipeline - avoids text parsing: Where-Object, Select-Object, Sort-Object, Measure-Object.
• Execution policy - PowerShell's script-running guard, not a security boundary; set per-scope.
• pathlib/shutil/subprocess - Python's file, copy and process-launch workhorses.
• argparse / getopts / param() - the standard way to accept options in each language.
• Idempotence - re-running the script must not double the effect; essential for schedulers.
• Logging - unattended automation must write timestamped logs, not print/echo to a lost console.
• Secrets - from environment or vault, never in source or Git history.
• Choose by content - commands and filters → shell; data, parsing, APIs → Python; Windows/Azure estate → PowerShell.
• Interpreted ≠ fast - scripts orchestrate; heavy computation belongs in compiled code or a database.

Formulas and examples

Example 1 - cron frequency and monthly runs. A log-rotation script is scheduled as */20 * * * 1-5.

• Runs per hour = 60/20 = 3; runs per day = 3 × 24 = 72.
• Weekdays only (Mon-Fri). In a month with 22 working days: 72 × 22 = 1,584 runs.
• If each run takes 4 s of CPU, total = 6,336 s ≈ 1.76 hours of machine time per month.

Example 2 - automation payback in rupees. A junior analyst spends 25 minutes daily consolidating 30 branch CSVs; the Python script does it in 30 s (0.5 min). Development took 6 hours. Loaded cost ₹300/hour.

• Time saved per run = 25 − 0.5 = 24.5 min.
• Payback = 6 × 60 ÷ 24.5 = 14.7 → 15 runs ≈ 3 working weeks.
• Monthly saving (22 runs) = 24.5 × 22 ÷ 60 = 8.98 hours ≈ ₹2,695; annual (264 runs) = 107.8 hours ≈ ₹32,340 against a one-time ₹1,800 build cost.

Example 3 - Bash: disk check with alert.

#!/bin/bash
set -euo pipefail
THRESHOLD=${1:-80}
LOG=/var/log/diskwatch.log
df -hP | awk 'NR>1 {gsub(/%/,"",$5); print $5, $6}' |
while read -r use mount; do
  if [ "$use" -ge "$THRESHOLD" ]; then
    echo "$(date +'%F %T') WARN $mount at ${use}%" | tee -a "$LOG"
  fi
done

• df -hP keeps one line per filesystem; awk strips % and prints use and mount point.
• ${1:-80} = first argument, defaulting to 80. Scheduled by 0 * * * * /opt/bin/diskwatch.sh 85.
• Trace a failing script with bash -x diskwatch.sh.

Example 4 - PowerShell: top memory consumers to CSV.

Get-Process |
  Where-Object { $_.WorkingSet64 -gt 200MB } |
  Select-Object Name, Id, @{n='MB';e={[math]::Round($_.WorkingSet64/1MB,1)}} |
  Sort-Object MB -Descending |
  Export-Csv -Path .\mem.csv -NoTypeInformation

• A process using 314,572,800 bytes → 314,572,800 / 1,048,576 = 300.0 MB, so it passes the 200 MB filter.
• The calculated property @{n=;e=} adds a computed column; nothing is text-parsed.

Example 5 - Python: consolidate CSVs and log.

#!/usr/bin/env python3
import csv, logging, pathlib, sys
logging.basicConfig(filename="merge.log", level=logging.INFO,
                    format="%(asctime)s %(levelname)s %(message)s")
src, out = pathlib.Path(sys.argv[1]), pathlib.Path(sys.argv[2])
rows, header = [], None
for f in sorted(src.glob("*.csv")):
    with f.open(newline="", encoding="utf-8") as fh:
        r = csv.reader(fh)
        h = next(r)
        header = header or h
        if h != header:
            logging.error("header mismatch in %s", f.name); sys.exit(2)
        rows.extend(r)
    logging.info("read %s", f.name)
with out.open("w", newline="", encoding="utf-8") as fh:
    w = csv.writer(fh); w.writerow(header); w.writerows(rows)
logging.info("wrote %d rows to %s", len(rows), out)

• 30 files × 1,200 data rows = 36,000 rows written, plus one header row = 36,001 lines.
• Exit code 2 on a header mismatch lets the scheduler or CI stage flag the failure.

Where it is used

• IT services and infra support (TCS, Infosys, Wipro, HCLTech, Tech Mahindra, Capgemini India): server health checks, patching, user provisioning, batch job monitoring in L1/L2 roles.
• DevOps/SRE and cloud (Flipkart, Swiggy, Zomato, PhonePe, Jio, startups): Bash in Dockerfiles and Jenkins/GitHub Actions stages, Python with boto3/Terraform for AWS and Azure provisioning.
• BFSI and fintech back-office: nightly reconciliation of NPCI/UPI and bank statement files, Razorpay/PayU settlement parsing, regulatory report generation for RBI/SEBI filings.
• Data engineering and analytics: Python ETL from Excel/CSV/API into SQL, pandas cleaning, Airflow DAGs; replaces manual MIS work in operations and WFM teams.
• QA and RPA: pytest/Selenium/Playwright suites, UiPath and Automation Anywhere bots with Python activities in Indian BPO and insurance processing.
• Government and research computing: NIC and state e-governance portals, HPC job scripts at IITs/C-DAC, bulk data pulls from data.gov.in, GSTN/Income-tax e-filing utility automation.

Interview questions

• Q: What is the difference between a shell script and a compiled program? A: A script is read and executed line by line by an interpreter at run time, so it is portable and quick to change; a compiled program is translated to machine code once and runs faster.
• Q: What does the shebang do, and what happens without it? A: It names the interpreter for an executable text file; without it the file is run by the calling shell's default, which may break Bash-only or Python syntax.
• Q: Why is set -euo pipefail recommended? A: -e aborts on the first failing command, -u on an undefined variable, and pipefail makes a pipeline fail if any stage fails - otherwise a broken script carries on with bad data.
• Q: Difference between [ ] and [[ ]] in Bash? A: [ is the POSIX test command and needs careful quoting; [[ ]] is a Bash keyword supporting pattern matching, regex =~ and &&/|| without word-splitting surprises.
• Q: Why is the PowerShell pipeline considered more powerful than the Bash pipeline? A: It passes typed .NET objects with properties, so you filter and sort on real fields instead of parsing text with awk/sed.
• Q: How do you pass arguments and options to a script in each language? A: $1..$n with getopts in Bash, a param() block in PowerShell, and sys.argv or argparse in Python.
• Q: When would you choose Python over Bash? A: When there is real logic - data structures, JSON/Excel parsing, API calls, exceptions, tests - or the script must run on Windows and Linux alike.
• Q: Why avoid shell=True in subprocess? A: The string is handed to the shell, so untrusted input can inject extra commands; passing an argument list bypasses the shell entirely.
• Q: Explain */10 * * * 1-5 in cron. A: Every 10 minutes, every hour, Monday to Friday - 144 runs per weekday.
• Q: A cron job works manually but fails under cron. Why? A: cron runs with a minimal environment and a different working directory, so PATH, virtualenv and relative paths must be set explicitly inside the script.
• Q: What is idempotence and why does a scheduled script need it? A: Re-running must leave the same end state; schedulers retry and overlap, so a non-idempotent script duplicates rows, emails or charges.
• Q: How do you handle credentials in an automation script? A: Read them from environment variables, a secrets manager or the OS credential store, keep them out of Git, and mask them in logs - personal data handling must also satisfy the DPDP Act, 2023.
• Q: Difference between Windows PowerShell 5.1 and PowerShell 7? A: 5.1 is the Windows-only edition built on .NET Framework and shipped with Windows; 7.x (pwsh) is the open-source, cross-platform edition on modern .NET and is the current line.

Common mistakes

• Writing name = "value" in Bash - spaces make the shell treat name as a command; assignment must have no spaces.
• Leaving variables unquoted, so filenames with spaces split into several arguments and rm deletes the wrong thing.
• Trying decimal arithmetic in Bash $(( )), which truncates - needing awk or bc -l instead.
• Confusing PowerShell's -eq/-match with C-style ==/=~, or expecting Bash-style text output from cmdlets.
• Hardcoding absolute paths, drive letters and credentials, so the script dies on another machine or leaks secrets into Git.
• Assuming cron inherits your interactive environment - missing PATH, virtualenv or working directory is the classic "works in terminal, fails at 2 a.m." bug.
• Using print/echo instead of logging with timestamps, leaving no evidence of what an unattended run actually did.
• Testing destructive commands (rm -rf, Remove-Item -Recurse, DROP) directly on production instead of with a dry-run flag and a backup.

Related topics

Python essentials · Operating systems: processes, files and permissions · Linux system administration · DevOps and CI/CD pipelines · Regular expressions and text processing · Cloud computing and infrastructure as code