Malware analysis and reverse engineering

In short

Malware analysis is the systematic examination of malicious software to determine what it does, how it does it, how to detect it and how to remove it; reverse engineering is the technique of recovering a program's design and logic from its binary when no source code is available. It combines file-format knowledge (PE/ELF), x86/x64 assembly, debugging, sandboxing and signature writing (YARA), and feeds detection rules and indicators of compromise back to defenders. Indian employers hiring for this include antivirus and product companies (Quick Heal/Seqrite, K7 Computing, Trend Micro, Palo Alto Networks, CrowdStrike, Zscaler India R&D), CERT-In and NCIIPC, the big SOC/MSSP practices (TCS, Infosys, Wipro, HCLTech, Accenture, LTIMindtree) and DFIR teams in banks under RBI cyber-security directions. It appears in AICTE/CBCS "Malware Analysis", "Cyber Forensics" and "Information Security" papers, in M.Tech cyber-security entrance interviews, and in certification syllabi (GREM, CEH, OSCP-adjacent).

Explained

Malware types and classification

• Virus: needs a host file; executes when the host runs and infects other files. Worm: self-propagating over a network without user action (e.g. mass SMBv1 exploitation in 2017).
• Trojan: disguised as legitimate software; RAT (remote access trojan) adds remote control, keylogging, screen capture.
• Backdoor: covert access channel; rootkit: hides processes/files/registry keys by hooking APIs (user-mode) or modifying kernel structures/SSDT or using DKOM (kernel-mode); bootkit/UEFI implant survives OS reinstall.
• Ransomware: encrypts files and demands payment. Modern families use hybrid crypto - a per-file symmetric key (AES-256) wrapped with the attacker's public RSA key, so offline decryption without the private key is infeasible. "Double extortion" adds data theft.
• Spyware, adware, infostealer (browser credentials, crypto wallets), banking trojan (web-inject, overlay attacks on UPI/netbanking apps), cryptominer, wiper (destruction, no ransom), botnet client, downloader/dropper/loader, logic bomb.
• Fileless malware: lives in memory, abuses PowerShell, WMI, registry Run/WMI subscriptions, and LOLBins (legitimate signed binaries - rundll32, mshta, regsvr32) so nothing malicious is written to disk.
• PUA/greyware: unwanted but not clearly illegal - classification is a vendor policy question.
• Classification axes: propagation (virus/worm/trojan), objective (steal/encrypt/destroy/mine), persistence location, platform (Windows PE, Linux ELF, Android APK/DEX, macOS Mach-O), and delivery (phishing attachment, drive-by, supply chain, USB/autorun).

Analysis lab and safety

• Isolated environment: a hypervisor VM (VMware Workstation, VirtualBox, KVM) with host-only or no networking, snapshots taken clean before every run, no shared folders, no clipboard sharing.
• Fake internet: INetSim or FakeNet-NG answers DNS/HTTP/SMTP so the sample reveals its C2 behaviour without reaching the real host; REMnux (Linux) plus FLARE-VM (Windows) are the standard toolkits.
• Risks to respect: VM-escape bugs, malware that detects virtualisation and goes dormant, worms that scan the LAN, and samples that destroy the host on detection. Bare-metal lab with network tap for high-risk samples.
• Legality in India: possessing and analysing samples for research is legitimate work, but deliberately writing or releasing malicious code attracts IT Act, 2000 s.43 (unauthorised access/damage - civil compensation), s.66 (computer-related offences), s.66F (cyber terrorism) and Bharatiya Nyaya Sanhita, 2023 cheating/forgery provisions; always analyse only on systems you own or are authorised to use.
• CERT-In Directions of 28 April 2022: reportable cyber incidents (including ransomware and malware infections) must be reported to CERT-In within 6 hours of noticing; logs to be kept for 180 days within India. Keeping a signed chain of custody matters if the sample becomes evidence (see digital forensics).

Static analysis (without executing)

1. Identify the file: file, magic bytes (MZ for PE, 0x7F "ELF", PK for APK/JAR), hashes MD5/SHA-1/SHA-256 for lookup; ssdeep/TLSH fuzzy hashes to find near-identical variants; imphash (hash of the import table) to cluster same-compiler/same-family builds.
2. Strings: strings (ASCII and UTF-16) reveals URLs, IPs, registry keys, mutex names, ransom-note text, PDB paths, embedded commands. Few meaningful strings usually means packing or encryption.
3. PE structure: DOS header → NT headers (IMAGE_FILE_HEADER, IMAGE_OPTIONAL_HEADER) → section table. Useful fields: AddressOfEntryPoint, ImageBase, Subsystem, TimeDateStamp (can be forged), sections .text .data .rdata .rsrc .reloc.
4. Imports/exports: suspicious imports tell the story - CreateRemoteThread, WriteProcessMemory, VirtualAllocEx (injection); SetWindowsHookEx, GetAsyncKeyState (keylogging); CryptEncrypt/BCryptGenRandom (ransomware); InternetOpenUrl/WinHttpSendRequest (C2); LoadLibrary+GetProcAddress only = dynamic resolution to hide intent.
5. Resources and overlays: .rsrc often hides a second-stage PE, a decoy PDF or a configuration blob; data appended after the last section (overlay) is a common payload store.
6. Digital signature: check Authenticode - a valid signature from a stolen certificate is a strong IOC; an invalid/absent signature on something claiming to be from a known vendor is suspicious.
7. Disassembly/decompilation: IDA Pro, Ghidra (free, NSA), Binary Ninja, radare2/rizin, objdump; Ghidra's decompiler gives readable C-like pseudocode for quick triage. For .NET use dnSpy/ILSpy (IL is nearly source); for Java jadx; for Python-compiled EXEs pyinstxtractor + uncompyle/decompyle3.
8. Advantages: safe, complete code coverage including branches never taken at runtime, no network exposure. Limits: defeated by packing and obfuscation, cannot see runtime-decrypted configuration, slow and skill-intensive.

Dynamic and behavioural analysis

• Monitoring tools: Procmon (file/registry/process), Process Explorer and Process Hacker (handles, injected memory), Autoruns (persistence), Regshot (before/after registry diff), TCPView, Wireshark, and API-trace tools (API Monitor, Frida).
• Automated sandboxes: Cuckoo/CAPE, Joe Sandbox, ANY.RUN, Hybrid Analysis - give a behaviour report, dropped files, PCAP and screenshots in minutes; Indian SOCs typically pair a commercial sandbox with a local CAPE instance for data that must stay in-country.
• What to record: processes created, files written/encrypted, registry keys (HKCU\...\Run, services, Scheduled Tasks, WMI subscriptions), mutex (shows the "already infected" check), DNS queries and DGA patterns, HTTP beacons with user-agent and URI structure, TLS JA3 fingerprint.
• Debugging: x64dbg/OllyDbg (user mode) and WinDbg (kernel) for breakpoints, single-stepping, memory dumps; typical workflow is to break on VirtualAlloc, let the packer decrypt, then dump the unpacked image from memory and fix the import table (Scylla/ImpREC).
• Memory forensics: Volatility 3 / Rekall on a RAM dump - plugins pslist, psscan (hidden processes), malfind (injected RWX pages), cmdline, netscan, dlllist, handles. Essential for fileless malware.
• Advantages: reveals real behaviour, decrypted strings and live C2 fast. Limits: only the executed path is observed; evasive samples need a specific trigger (a command-line argument, a particular date, a domain-joined machine, a Cyrillic keyboard layout).

Obfuscation, packing and anti-analysis

• Packers/crypters: UPX (trivial - upx -d), Themida, VMProtect, ASPack, and custom crypters. Indicators of packing: very few imports, high entropy (near 8 bits/byte), RawSize near 0 with large VirtualSize, non-standard section names (UPX0, .vmp0), entry point outside .text.
• Obfuscation: string encryption (XOR with a key, RC4, base64 layers), API hashing instead of names, control-flow flattening, opaque predicates, junk instructions, dead-code insertion, self-modifying code, polymorphic (changing decryptor each copy) and metamorphic (rewriting the body) engines.
• Anti-debugging: IsDebuggerPresent, CheckRemoteDebuggerPresent, NtQueryInformationProcess(ProcessDebugPort), PEB BeingDebugged/NtGlobalFlag, timing checks (RDTSC deltas), INT 3/INT 2D tricks, exception-based checks, OutputDebugString abuse.
• Anti-VM/anti-sandbox: querying CPUID hypervisor bit, MAC address prefixes (00:0C:29 = VMware, 08:00:27 = VirtualBox), registry artefacts, VM tool processes, disk size and RAM below a threshold, few recent documents, no mouse movement, long sleeps (defeated by sleep-patching/time acceleration).
• Anti-disassembly: jumps into the middle of instructions, overlapping instructions, opaque conditional jumps that always take one branch, bogus exception handlers - these break linear-sweep disassemblers; recursive-descent plus manual patching fixes them.
• Counter-measures: patch checks to NOP, use ScyllaHide/TitanHide, hardware breakpoints instead of software, unpack by running to the OEP, or work entirely from a memory dump.

Code injection, persistence and the ATT&CK view

• Injection techniques: classic DLL injection (VirtualAllocEx + WriteProcessMemory + CreateRemoteThread), reflective DLL loading, process hollowing (create suspended process, unmap its image, write the malicious image, resume), APC injection, thread hijacking, AtomBombing, and DLL search-order hijacking/sideloading.
• Persistence: Run/RunOnce keys, Startup folder, Windows Services, Scheduled Tasks, WMI event subscription, COM hijacking, Image File Execution Options, LSASS/SSP providers, bootkit; on Linux cron/systemd units/LD_PRELOAD/.bashrc.
• Privilege escalation and defence evasion: UAC bypass, token theft, disabling AMSI/ETW patching, timestomping, clearing event logs, signed-driver abuse (BYOVD).
• Mapping output: analysis findings are expressed as MITRE ATT&CK techniques (e.g. T1055 Process Injection, T1486 Data Encrypted for Impact), which makes a report comparable across teams - the techniques catalogue itself belongs to the threat-intelligence note; what matters here is that every observed behaviour is tagged.

Detection artefacts: IOCs, YARA and reporting

• IOCs: file hashes (fragile - change with one byte), C2 domains/IPs/URLs, mutex names, service names, registry paths, file paths, certificate thumbprints, JA3/JA3S, ransom-note filenames.
• Pyramid of pain: hashes are easiest for the attacker to change; TTPs are hardest - so behavioural rules outlive hash blocklists.
• YARA is the standard signature language: meta, strings (text, hex, regex, with modifiers nocase wide ascii fullword xor) and a condition combining them with PE module fields, entropy and counts. Sigma does the same for log/event detections; Snort/Suricata for network signatures.
• Report structure: executive summary, sample identification (name, size, hashes, file type), delivery, static findings, dynamic behaviour, capability summary, network IOCs, persistence, ATT&CK mapping, detection rules, containment and removal steps, confidence and limitations.

Key points

• Malware analysis - determining a sample's capability, behaviour, detection signature and removal method, without trusting its own claims.
• Static vs dynamic - static reads the binary (safe, full coverage, blocked by packing); dynamic runs it (real behaviour, only the executed path).
• Basic vs advanced - basic = hashes, strings, imports, sandbox report; advanced = disassembly, debugging, unpacking, memory forensics.
• PE identification - MZ at offset 0 and PE\0\0 at the offset in e_lfanew; ELF starts with 0x7F 'E' 'L' 'F'.
• Imphash - hash of the ordered import table; same imphash across samples strongly suggests the same family/build chain.
• Fuzzy hashing (ssdeep/TLSH) - ssdeep gives a 0-100 similarity score; TLSH gives a distance score (0 = identical, larger = more different); both find variants a cryptographic hash cannot.
• Entropy - Shannon entropy per byte, 0-8; sections above about 7.2 suggest compression/encryption/packing.
• Packing - compressing/encrypting the real code with a stub that unpacks it in memory; unpack by running to the OEP and dumping.
• Process hollowing - a legitimate process's image is replaced in memory, so the process name on the task list is innocent.
• Mutex check - a named mutex prevents double infection; its name is a durable IOC and can be used as a vaccine.
• Anti-VM - CPUID hypervisor bit, MAC prefixes, VM tool processes, tiny disks, no user activity.
• Hybrid cryptography in ransomware - AES per file, RSA-wrapped key; recovery without the private key is not feasible, so backups are the control.
• Fileless malware - executes from memory via PowerShell/WMI/LOLBins; found by memory forensics and command-line logging, not file scanning.
• YARA - text/hex/regex strings plus a condition; the practical output of an analysis for detection at scale.
• Volatility plugins - pslist, psscan, malfind, netscan, cmdline are the standard first five on a RAM dump.
• Pyramid of pain - block TTPs over hashes; hashes churn with every rebuild.
• CERT-In 2022 Directions - 6-hour incident reporting and 180-day log retention in India.
• Legal boundary - analysis is lawful on authorised systems; writing or spreading malicious code attracts IT Act s.43/66/66F.

Formulas and examples

Example 1 - Entropy of a packed section. A .text section of 64 KB has, after counting, 256 distinct byte values each occurring with almost equal frequency (pᵢ ≈ 1/256).

• H = − Σ (1/256) log₂(1/256) = − 256 × (1/256) × (−8) = 8.00 bits/byte.
• Compare a normal x86 code section where 0x00, 0x8B, 0xE8, 0xFF dominate: typical H ≈ 6.0-6.6.
• Conclusion: 8.00 is the theoretical maximum - the content is compressed or encrypted, so the sample is packed and static disassembly of this section is pointless until it is unpacked.

Example 2 - Two-symbol entropy check (hand calculation). A 1,000-byte blob contains only two byte values, 0x00 (750 times) and 0xFF (250 times).

• p₁ = 0.75, p₂ = 0.25. H = −(0.75 log₂0.75 + 0.25 log₂0.25).
• log₂0.75 = −0.415, log₂0.25 = −2.
• H = −(0.75 × −0.415 + 0.25 × −2) = −(−0.311 − 0.500) = 0.811 bits/byte.
• Very low entropy - padding or sparse data, not encrypted content.

Example 3 - Reading a PE header by hand. First bytes of a sample: 4D 5A 90 00 ..., and the 4 bytes at offset 0x3C are E0 00 00 00.

• 4D 5A = "MZ" → DOS/PE file. e_lfanew (little-endian) = 0x000000E0 = 224 decimal.
• At offset 0xE0 we expect 50 45 00 00 ("PE\0\0"); the IMAGE_FILE_HEADER follows at 0xE4, and Machine = 0x8664 would mean x64 (0x014C = x86).
• If SizeOfRawData of .text = 0x0200 (512 bytes) but VirtualSize = 0x20000 (131,072 bytes), ratio = 131072/512 = 256 → code is generated at runtime; classic packer/loader layout.

Example 4 - Ransomware impact estimate for an Indian hospital. 1.8 TB of records on one file server; the sample encrypts at a measured 90 MB/s in the lab, and the bill for downtime is ₹4.5 lakh per hour.

• Data = 1.8 TB = 1,800,000 MB (decimal). Time = 1,800,000 / 90 = 20,000 s = 5 h 33 min.
• If detection takes 45 minutes, encrypted share ≈ 45/333 ≈ 13.5% ≈ 243 GB before containment.
• Restoring from a nightly backup at a 200 MB/s restore rate for that 243 GB = 243,000/200 ≈ 1,215 s ≈ 20 minutes of restore, versus 5.5 hours of full-disk damage if undetected - which is the argument for behaviour-based detection (mass file-rename, shadow-copy deletion) over hash blocklists.
• Downtime cost at 6 hours ≈ 6 × ₹4.5 lakh = ₹27 lakh, excluding regulatory and reputational cost. Report to CERT-In within 6 hours of noticing.

Example 5 - Evaluating a classifier. An ML-based static detector is tested on 10,000 files: 2,000 malicious, 8,000 benign. It flags 1,860 of the malicious and 120 of the benign.

• TP = 1,860; FN = 2,000 − 1,860 = 140; FP = 120; TN = 8,000 − 120 = 7,880.
• TPR (detection) = 1,860/2,000 = 93.0%; FPR = 120/8,000 = 1.5%.
• Precision = 1,860/(1,860+120) = 1,860/1,980 = 93.94%; Accuracy = (1,860+7,880)/10,000 = 97.4%.
• Interpretation: 1.5% FPR on an endpoint fleet of 50,000 machines means ~750 false alerts - unacceptable for auto-quarantine, so the rule runs in alert-only mode and is tightened.

Where it is used

• Antivirus and EDR product R&D: signature and behaviour-rule writing, unpacker development and detection QA at Quick Heal/Seqrite (Pune), K7 (Chennai), Sophos, Trend Micro, CrowdStrike, SentinelOne and Zscaler India centres.
• National agencies: CERT-In malware advisories and the Cyber Swachhta Kendra (botnet cleaning centre) with its free bot-removal tools; NCIIPC for critical-infrastructure sectors.
• Bank and NBFC DFIR teams: analysing Android banking trojans and UPI/APK-based fraud campaigns; findings feed RBI-mandated incident reporting and customer advisories.
• MSSP/SOC tier-3 roles: TCS, Infosys, Wipro, HCLTech, Accenture, LTIMindtree - sandbox triage, IOC extraction, YARA/Sigma authoring, retro-hunts across the fleet.
• Law-enforcement and forensic labs: state cyber cells and CFSL/FSL malware examination supporting prosecution; analysis notes become expert evidence, so reproducibility and chain of custody matter.
• Threat-research publishing and bug bounty-adjacent work: firmware and IoT reverse engineering, APK analysis for fake loan/investment apps, and vulnerability research in Indian product companies.

Interview questions

• Q: Difference between static and dynamic malware analysis? A: Static examines the file without running it (hashes, strings, imports, disassembly) and covers all code but is blocked by packing; dynamic runs it in an isolated VM and shows real behaviour but only along the executed path.
• Q: How do you tell at a glance that a binary is packed? A: Very few imports (often just LoadLibrary/GetProcAddress), high section entropy near 8, odd section names like UPX0/.vmp0, VirtualSize far larger than SizeOfRawData, and almost no readable strings.
• Q: How would you unpack a custom-packed sample? A: Load it in x64dbg, set breakpoints on VirtualAlloc/VirtualProtect/WriteProcessMemory, run until the stub transfers control to the original entry point, then dump the image from memory and rebuild the import table with Scylla.
• Q: What is an imphash and why is it useful? A: An MD5 of the ordered import-table entries; identical imphashes across samples suggest the same family or build toolchain, so it clusters variants that differ by file hash.
• Q: Why is a SHA-256 blocklist a weak defence? A: One byte changes the hash, so recompiled or repacked variants evade it; behavioural and TTP-level detections sit higher on the pyramid of pain.
• Q: How does malware detect that it is in a sandbox? A: CPUID hypervisor bit, VM MAC prefixes and registry artefacts, VM tool processes, small RAM/disk, no mouse movement or recent documents, and long sleeps that a sandbox may truncate.
• Q: Explain process hollowing. A: The malware starts a legitimate process suspended, unmaps its image, writes its own PE into that address space, fixes the entry point via SetThreadContext, and resumes - so a trusted process name runs malicious code.
• Q: Can you decrypt files hit by modern ransomware without paying? A: Generally no, because a random AES key per file is wrapped with the attacker's RSA public key; recovery depends on backups, shadow copies if not deleted, a leaked private key, or a key-generation flaw.
• Q: What does malfind in Volatility do? A: It walks a process's VAD entries and flags private regions (not backed by any mapped file) whose protection is executable - typically PAGE_EXECUTE_READWRITE - and dumps/disassembles them: the signature of injected or reflectively loaded code.
• Q: Write the condition idea of a YARA rule for a ransom note. A: Match two or three distinctive note strings wide ascii nocase plus a file-size bound and, for a PE, the MZ magic in the condition - e.g. all of the note strings and filesize < 2MB.
• Q: A sample runs, exits instantly and does nothing. What next? A: Suspect an evasion or a missing trigger: check for required command-line arguments or exports (run via rundll32), anti-debug/anti-VM checks, domain-join or language-locale checks, a mutex already present, or a C2 that must answer - then patch the check or fake the condition.
• Q: What is your legal and reporting obligation in India after confirming a ransomware infection at a client? A: Report the incident to CERT-In within 6 hours of noticing under the 28 April 2022 Directions, preserve logs (180-day retention requirement) and the sample with chain of custody, and advise the client on its sectoral obligations such as RBI reporting for a regulated entity.

Common mistakes

• Running a sample on the host machine, a VM with a bridged network, or a VM with shared folders and no clean snapshot.
• Treating a sandbox report as the complete answer - evasive samples show benign behaviour, and "no malicious activity" is not "benign".
• Confusing a packer with a crypter or calling every high-entropy file malicious - legitimate installers are also packed and signed.
• Reading TimeDateStamp or a PDB path as fact; both are trivially forged, and timestamps are often zeroed or faked.
• Reporting only hashes as IOCs and omitting the durable artefacts - mutex names, registry paths, C2 patterns, ATT&CK techniques.
• Mixing up virus, worm and trojan definitions in exams, or claiming a rootkit and a backdoor are the same thing.
• Forgetting to fix the import table after dumping an unpacked image, then concluding the dump is corrupt.
• Analysing a .NET or Android sample with a native disassembler instead of dnSpy/jadx, and losing near-source clarity.

Related topics

Digital forensics · Incident response and threat hunting · Threat intelligence and MITRE ATT&CK · Security operations: SOC and SIEM · Operating systems and memory management · Assembly language and computer architecture