Python for data analysis: NumPy and pandas

In short

NumPy gives Python a fixed-type, contiguous n-dimensional array (ndarray) with vectorised arithmetic in compiled C, and pandas builds labelled one-dimensional Series and two-dimensional DataFrame structures on top of it for tabular work. Together they are the working layer of almost every Indian data-science job: loading CSV/Excel/SQL extracts, selecting and filtering rows, grouping, joining and aggregating, then handing clean arrays to scikit-learn. University AI&DS syllabi (AICTE model curriculum, UGC CBCS "Python for Data Science" / "Data Analytics Lab") examine array creation, indexing, slicing, broadcasting, aggregation, groupby, merges and reshaping, and GATE DA (Data Science & AI, a paper introduced in 2024) tests the same reasoning as pseudocode and output-prediction. Placement tests at TCS, Infosys, Wipro, Accenture, Cognizant, Mu Sigma, Fractal, ZS, Flipkart and bank analytics teams routinely ask "write this in pandas without a loop" and "what does this print". Expect both coding and explain-the-concept questions on axis semantics, views versus copies, and loc versus iloc.

Explained

Why NumPy: the ndarray and vectorisation

• A Python list stores pointers to boxed objects of any type; an ndarray stores raw values of one dtype in one contiguous memory block, so it is far smaller and faster to scan.
• Attributes of an array a: a.ndim (number of axes), a.shape (tuple of lengths), a.size (total elements = product of shape), a.dtype, a.itemsize (bytes per element), a.nbytes = size × itemsize.
• Common dtypes: int8/16/32/64, float32/float64, bool, complex128, object, datetime64[ns]. Default integer on 64-bit Linux is int64; on Windows NumPy ≥ 2.0 also defaults to int64 (older NumPy on Windows gave int32) — state the assumption in an exam answer.
• Vectorisation means applying an operation to the whole array in one compiled loop: a * 2 + b runs element-wise with no Python-level for. Typical speed-up over a Python loop is one to two orders of magnitude.
• Creation: np.array([...]), np.zeros((3,4)), np.ones, np.full, np.empty (uninitialised garbage), np.eye(n), np.arange(start, stop, step) (stop excluded), np.linspace(start, stop, num) (both ends included by default), np.random.default_rng(42) for reproducible random data (the modern Generator API; the legacy np.random.seed + np.random.rand still appears in syllabi).
• Reshaping: reshape(r, c) needs r*c == size; one dimension may be -1 to be inferred. ravel() returns a flattened view where possible, flatten() always a copy, .T transposes, np.newaxis/None inserts an axis, np.concatenate, np.vstack, np.hstack, np.split combine and cut.

Indexing, slicing, views and copies

• Basic slicing a[1:5:2], a[:, 2], a[::-1] returns a view: it shares memory, so writing to the slice changes the original. This is NumPy's single most examined trap and differs from list slicing, which copies.
• Fancy (advanced) indexing with an integer array a[[0, 2, 4]] or a boolean mask a[a > 50] always returns a copy.
• Boolean masking combines with &, |, ~ (not and, or, not) and needs brackets: a[(a > 10) & (a < 20)].
• 2-D access is a[row, col]; a[1] gives row 1; a[:, 1] gives column 1 as 1-D. Negative indices count from the end.
• np.where(cond, x, y) is a vectorised if-else; np.where(cond) alone returns the indices. np.select handles multiple conditions; np.clip bounds values.
• Use .copy() whenever you intend to modify independently. a.base is not None tells you an array is a view.

Broadcasting and axis semantics

• Broadcasting rule: compare shapes right-to-left; dimensions are compatible if equal or if one of them is 1; a missing leading dimension is treated as 1. The result takes the larger length on each axis.
• (3,4) + (4,) → (3,4); (3,1) + (1,4) → (3,4); (3,4) + (3,) → error (4 vs 3). Fix with a + b[:, None].
• No data is actually copied — broadcasting is a stride trick, which is why centring a 1-crore-row matrix by its column means costs almost nothing extra.
• axis = 0 means "down the rows" (collapse rows, one result per column); axis = 1 means "across the columns" (one result per row). a.sum() with no axis reduces everything to a scalar.
• Aggregations: sum, mean, std, var, min, max, argmin, argmax, cumsum, cumprod, median, percentile, any, all. NaN-safe twins: np.nansum, np.nanmean, etc.
• NumPy's std/var default to population form (ddof=0); pandas defaults to sample form (ddof=1). This mismatch is a favourite interview question.
• Linear algebra: a @ b or np.dot (matrix product), np.linalg.inv, det, solve, eig, svd, matrix_rank, norm. * is always element-wise, never matrix multiplication.
• Universal functions (ufuncs) np.sqrt, np.exp, np.log, np.sin act element-wise and accept an out= buffer for in-place work.

pandas objects: Series, DataFrame and Index

• Series = 1-D values + an Index (labels) + a name and dtype. DataFrame = an ordered dict of Series sharing one row Index, with its own column Index; each column has its own dtype.
• Construct from dict of lists, list of dicts, NumPy array with columns=, or read in: read_csv, read_excel, read_json, read_sql, read_parquet. Useful read_csv arguments: usecols, dtype, parse_dates, na_values, thousands=',', nrows, chunksize, encoding (utf-8 or latin-1 for legacy Indian government CSVs), skiprows.
• Inspection: head/tail, shape, info() (dtypes, non-null counts, memory), describe() (count, mean, std, min, quartiles, max; include='all' adds object columns), dtypes, nunique, value_counts(normalize=True, dropna=False), memory_usage(deep=True).
• Selection: df['col'] → Series, df[['a','b']] → DataFrame. **.loc is label-based and its slice end is inclusive; .iloc is position-based and its end is exclusive**. df.at/df.iat are fast scalar accessors. df.query("salary > 50000 and city == 'Pune'") is a readable filter.
• Index work: set_index, reset_index, sort_index, reindex, rename, MultiIndex for hierarchical rows (xs, droplevel, swaplevel); df.index.is_unique matters before any join.
• Chained assignment df[df.a > 1]['b'] = 0 may silently fail (SettingWithCopyWarning) because the first step can return a copy. Always write df.loc[df.a > 1, 'b'] = 0. pandas 3.0 adopts Copy-on-Write, under which chained assignment never propagates — so write .loc assignments today.
• Missing data: NaN (float), None (object), pd.NaT (datetime), pd.NA (nullable dtypes). isna, notna, dropna(how='any'/'all', subset=, thresh=), fillna(value), ffill()/bfill() (the old fillna(method=...) was deprecated in 2.1 and removed in pandas 3.0), interpolate. Integer columns with gaps become float64 unless you use the nullable Int64 dtype.
• Dtype economy: astype('category') for repeated strings (state, city, product) can cut memory by 90%+; downcast integers; string[pyarrow] and Arrow-backed dtypes are the current direction of pandas.

Transform, group, join, reshape

• Column creation: direct assignment, or assign for chaining. apply(axis=1) is a row-wise Python loop — slow; prefer vectorised arithmetic, np.where, map on a Series, or .str/.dt accessors.
• Accessors: .str (lower, strip, contains, replace, split, extract, len, zfill — handles PIN codes and GSTINs), .dt (year, month, quarter, dayofweek, day_name, days_in_month), .cat (categories, codes).
• Sorting and ranking: sort_values(by=, ascending=, na_position=), nlargest/nsmallest, rank(method='dense'/'min'/'average').
• GroupBy = split, apply, combine. df.groupby('city')['salary'].mean(); multiple keys give a MultiIndex; agg takes a dict ({'salary':['mean','max'],'id':'count'}) or named aggregation (avg=('salary','mean')); transform returns a result broadcast back to the original shape (used for group-mean imputation and share-of-group); filter keeps whole groups by a predicate. Use as_index=False or reset_index() to get a flat table; dropna=False keeps NaN keys; observed=True for categoricals.
• Joins: pd.merge(left, right, on=, how='inner'/'left'/'right'/'outer'/'cross', suffixes=, indicator=True, validate='one_to_one'). df.join merges on index; pd.concat([...], axis=0/1) stacks. A many-to-many key silently multiplies rows — check duplicated() and use validate=.
• Reshaping: pivot (unique index-column pairs, no aggregation), pivot_table (aggregates, aggfunc, margins=True for totals), melt (wide → long: id_vars, value_vars), stack/unstack (move levels between index and columns), crosstab (frequency table, normalize=).
• Time series: pd.to_datetime(col, dayfirst=True, format='%d-%m-%Y') for Indian DD-MM-YYYY data; a DatetimeIndex enables resample('ME').sum(), asfreq, shift, diff, pct_change, rolling(7).mean(), expanding, and tz_convert('Asia/Kolkata') (IST = UTC+05:30). Indian financial year April–March: derive with (date - pd.DateOffset(months=3)).dt.year.
• Duplicates and categories: duplicated(subset=, keep='first'/'last'/False), drop_duplicates, cut (fixed bins), qcut (equal-count bins), clip.
• Output: to_csv(index=False), to_excel, to_parquet (typed, compressed, far smaller than CSV), to_sql.

Performance, interoperability and limits

• Order of preference for speed: NumPy/pandas vectorised expression → groupby/transform → map/.str → apply → iterrows (worst; itertuples is several times faster than iterrows).
• Build a list of frames and pd.concat once; appending inside a loop is O(n²) and DataFrame.append was removed in pandas 2.0.
• Memory rule of thumb: a loaded DataFrame often occupies 2–10× the CSV size; use usecols, explicit dtype, category, chunksize, or Parquet for datasets of a few crore rows. Beyond single-machine memory, move to Dask, Polars or PySpark — pandas is single-threaded and in-memory by design.
• df.to_numpy() gives a single array (upcast to a common dtype) for scikit-learn; df.values is the older form. pd.DataFrame(arr, columns=...) goes back.
• Limits to state in an exam: no built-in parallelism, high memory overhead, NaN forcing floats in classic dtypes, ambiguity of apply return shapes, and the object dtype making string columns slow.
• Floating-point equality: compare with np.isclose/np.allclose, never ==, and remember np.nan != np.nan.
• Compliance note for Indian work: personal data in these frames (Aadhaar, PAN, phone, salary) falls under the Digital Personal Data Protection Act, 2023; mask or hash identifiers before sharing notebooks, and keep only what the purpose needs. Rules under the Act have been notified in stages — check the current status before quoting timelines.

Key points

• ndarray - fixed-dtype, contiguous, n-dimensional array; fast because the loop runs in C, not Python.
• shape/size/ndim/dtype - shape is a tuple, size is the product of the shape, ndim is the length of the shape.
• Vectorisation - whole-array operations replace explicit loops; the first thing an interviewer checks.
• Basic slice = view, fancy/boolean index = copy - writing to a view edits the parent array.
• Broadcasting - align shapes right-to-left; compatible if equal or 1; no data copied.
• axis=0 down the rows (per column), axis=1 across the columns (per row) - drop(axis=1) removes columns.
• *** is element-wise, @ is matrix multiplication** - never mix them in linear algebra answers.
• ddof - NumPy std is population (ddof=0), pandas std is sample (ddof=1).
• Series vs DataFrame - 1-D labelled values vs 2-D table of Series sharing a row Index.
• **.loc label-based, end inclusive; .iloc position-based, end exclusive** - the classic output-prediction question.
• SettingWithCopyWarning - caused by chained assignment; fix with a single .loc assignment.
• GroupBy split-apply-combine - agg reduces, transform returns the original shape, filter drops whole groups.
• merge how= - inner keeps matches, left keeps all left rows, outer keeps all; indicator=True shows where each row came from.
• pivot vs pivot_table - pivot needs unique pairs, pivot_table aggregates duplicates.
• melt vs pivot - melt goes wide → long, pivot/unstack goes long → wide.
• NaN handling - NaN is a float, so an int column with gaps becomes float unless Int64 is used; NaN == NaN is False.
• category dtype - huge memory saving for repeated strings; an *ordered* category fixes group order, while observed=True in groupby drops unused categories instead of listing them as empty groups.
• apply/iterrows are the slow path - prefer vectorised code; itertuples beats iterrows when a loop is unavoidable.

Formulas and examples

Example 1 - shape, memory and dtype economy. A sales extract has 50,00,000 rows (50 lakh) and these columns: amount (float64), qty (int64), state (object, 36 distinct values averaging 10 characters).

• amount: 50,00,000 × 8 = 4,00,00,000 bytes = 40 MB (using 1 MB = 10⁶ bytes). Same for qty → 40 MB.
• state as object: each cell is a pointer (8 bytes) plus a Python string (≈ 49 + 10 ≈ 59 bytes) ≈ 67 bytes → ≈ 335 MB.
• state as category: 36 levels fit in int8 codes → 50,00,000 × 1 = 5 MB, plus 36 stored strings (≈ 2 KB) ≈ 5 MB.
• Total: 415 MB → 85 MB, a 79.5% reduction; downcasting qty to int32 saves another 20 MB → 65 MB.

Example 2 - broadcasting and normalisation. Marks array M of shape (4, 3) — 4 students, 3 subjects:

M = [[80, 70, 90],
     [60, 50, 70],
     [90, 80, 100],
     [70, 60, 80]]

• Column means (M.mean(axis=0)) = (80+60+90+70)/4 = 75, (70+50+80+60)/4 = 65, (90+70+100+80)/4 = 85 → [75, 65, 85], shape (3,).
• M - M.mean(axis=0) broadcasts (4,3) with (3,) → row 1 becomes [5, 5, 5]; row 2 [-15, -15, -15]; row 3 [15, 15, 15]; row 4 [-5, -5, -5].
• Row means (M.mean(axis=1)) = 80, 60, 90, 70 → shape (4,). M - M.mean(axis=1) fails (4 vs 3). Correct form: M - M.mean(axis=1)[:, None], which gives [[0,-10,10],[0,-10,10],[0,-10,10],[0,-10,10]].
• Column 1 (80, 60, 90, 70): mean 75, deviations +5, −15, +15, −5; Σd² = 25+225+225+25 = 500. NumPy std = √(500/4) = √125 ≈ 11.18; pandas std = √(500/3) = √166.67 ≈ 12.91. Check: 11.18² × 4/3 = 166.7 ✓.

Example 3 - groupby, transform and share of total.

emp = pd.DataFrame({
    'city':   ['Pune','Pune','Mumbai','Mumbai','Mumbai','Chennai'],
    'salary': [60000, 80000, 120000, 100000, 80000, 50000]})

• emp.groupby('city')['salary'].mean() → Chennai 50,000; Mumbai (120000+100000+80000)/3 = 1,00,000; Pune (60000+80000)/2 = 70,000.
• emp.groupby('city')['salary'].transform('mean') returns 6 values, one per original row: 70000, 70000, 100000, 100000, 100000, 50000 — shape preserved, which agg does not do.
• Share within city: emp['salary'] / emp.groupby('city')['salary'].transform('sum') → Pune 0.4286 and 0.5714; Mumbai 0.40, 0.3333, 0.2667; Chennai 1.00.
• Named aggregation: emp.groupby('city', as_index=False).agg(avg=('salary','mean'), n=('salary','size')) → three rows with avg and n as plain columns.
• emp.groupby('city')['salary'].filter(lambda s: s.mean() > 60000) keeps the Pune and Mumbai rows (5 rows), dropping Chennai entirely.

**Example 4 - merge cardinality and how=.** orders has 5 rows with cust_id = [1, 1, 2, 3, 5]; cust has 4 rows with cust_id = [1, 2, 2, 4].

• indicator=True on the outer merge labels rows both (4), left_only (2), right_only (1).
• validate='one_to_one' would raise MergeError here, because cust_id repeats on both sides — exactly the check that catches silently inflated revenue totals.

**Example 5 - .loc vs .iloc and view vs copy (output prediction).**

s = pd.Series([10, 20, 30, 40], index=['a','b','c','d'])
s.loc['a':'c']     # a 10, b 20, c 30   -> 3 items (end INCLUSIVE)
s.iloc[0:3]        # a 10, b 20, c 30   -> 3 items (end EXCLUSIVE)
s.iloc[0:2]        # a 10, b 20         -> 2 items

a = np.arange(6)         # [0 1 2 3 4 5]
v = a[1:4]; v[0] = 99    # v is a VIEW  -> a becomes [0 99 2 3 4 5]
c = a[[1,2,3]]; c[0] = 7 # c is a COPY  -> a unchanged: [0 99 2 3 4 5]
print(a.sum())           # 0+99+2+3+4+5 = 113

Where it is used

• IT services and analytics firms (TCS, Infosys, Wipro, Accenture, Cognizant, Mu Sigma, Fractal, ZS Associates): data preparation, reconciliation and reporting pipelines feeding dashboards and models.
• BFSI (HDFC, ICICI, Axis, SBI analytics, Paytm, PhonePe, Razorpay, Zerodha, CRED): transaction aggregation, fraud feature building, portfolio and NAV rollups, RBI/SEBI regulatory extracts.
• E-commerce, quick commerce and consumer tech (Flipkart, Myntra, Swiggy, Zomato, Zepto, Ola, Meesho): demand and pricing analysis, cohort and funnel work, groupby + resample on event logs.
• Health, pharma and manufacturing (Dr. Reddy's, Cipla, Tata Steel, L&T, Bosch): clinical trial tables, sensor/IoT time-series resampling and quality analytics.
• Public data and research: MoSPI, RBI DBIE, data.gov.in, Census, NFHS, GSTN and NSE/BSE bhavcopy files are nearly always read with read_csv/read_excel and cleaned in pandas.
• Job roles: Data Analyst, Data Engineer (as the staging layer before Spark), Data Scientist, BI Developer, Quantitative Analyst, ML Engineer, Research Assistant — pandas fluency is usually the screening filter.

Interview questions

• Q: Why is a NumPy array faster than a Python list for numeric work? A: It stores one dtype in contiguous memory with no per-element object overhead, so operations run as a single compiled C loop with good cache locality and SIMD use.
• Q: What is broadcasting and when does it fail? A: It is the automatic alignment of shapes right-to-left, where dimensions must be equal or one of them 1; it fails when two unequal lengths (neither 1) meet, such as (3,4) with (3,).
• Q: Difference between .loc and .iloc? A: .loc selects by labels and includes the end of a slice; .iloc selects by integer position and excludes the end.
• Q: When does indexing return a view and when a copy? A: Basic slicing of an ndarray returns a view sharing memory; fancy (integer-array) and boolean indexing return copies. In pandas, treat any result as possibly a copy and assign through a single .loc.
• Q: What causes SettingWithCopyWarning? A: Chained assignment such as df[mask]['col'] = x, where the first operation may return a copy so the write never reaches df; use df.loc[mask, 'col'] = x.
• Q: Difference between agg, transform and apply on a GroupBy? A: agg reduces each group to a scalar, transform returns a result aligned to the original rows, and apply passes the whole group and can return anything — the most flexible and the slowest.
• Q: merge versus join versus concat? A: merge joins on columns or indexes with SQL-style how; join is a convenience merge on the index; concat simply stacks frames along an axis without key matching.
• Q: Why did my left join produce more rows than the left frame? A: The right frame has duplicate join keys, so each left row matches several right rows; check with duplicated() and enforce validate='m:1'.
• Q: Why does an integer column become float after adding a missing value? A: NaN is a float64 value and classic NumPy integer dtypes cannot hold it; use the nullable Int64 dtype to keep integers with pd.NA.
• Q: pivot or pivot_table? A: pivot only reshapes and raises on duplicate index-column pairs; pivot_table aggregates duplicates with aggfunc and can add margins.
• Q: Why does df.std() differ from np.std(df.values)? A: pandas uses the sample estimator (ddof=1) and NumPy the population one (ddof=0); they differ by the factor √(n/(n−1)).
• Q: How would you process a 20 GB CSV with 8 GB of RAM? A: Read in chunksize batches with explicit dtype/usecols and aggregate incrementally, convert to Parquet, or move to Dask, Polars or PySpark — pandas itself is in-memory.
• Q: What does axis=1 mean in df.drop('col', axis=1) versus df.sum(axis=1)? A: In both it means "work along the columns": drop removes a column label, and sum adds across the columns to give one value per row.

Common mistakes

• Using and/or/not instead of &/|/~ on arrays and Series, or omitting brackets around each condition — gives "truth value of an array is ambiguous".
• Mixing up the axes: using axis=0 when a per-row result is needed, or forgetting axis=1 in drop.
• Modifying a NumPy slice and being surprised the parent changed (view), or editing a boolean-indexed result and finding the original untouched (copy).
• Chained assignment instead of a single .loc assignment, so the write is silently lost.
• Forgetting that most pandas methods return a new object: df.dropna() or df.sort_values() without reassigning (or inplace=, now discouraged) leaves df unchanged.
• Looping with iterrows or apply(axis=1) where a vectorised expression or transform would do, and appending inside a loop instead of one pd.concat.
• Merging without checking key uniqueness, so duplicated keys inflate aggregates; and comparing floats with == instead of np.isclose.
• Parsing Indian DD-MM-YYYY dates without dayfirst=True/format=, which silently swaps day and month for days ≤ 12.

Related topics

Python essentials · Data cleaning and feature engineering · SQL queries and joins · Advanced SQL: subqueries, CTEs and window functions · Data visualisation · Python ML libraries: scikit-learn, PyTorch and TensorFlow