Python ML libraries: scikit-learn, PyTorch and TensorFlow

Artificial Intelligence and Data Science (B.E./B.Tech) · Programming and data · free study notes for exams, placements and interviews

In short

scikit-learn, PyTorch and TensorFlow (with Keras) are the three libraries that carry almost all production machine learning in Python. scikit-learn is the classical-ML workhorse for tabular data — one uniform fit/predict API over regression, classification, clustering and model selection; PyTorch and TensorFlow are deep-learning frameworks built on GPU tensors and automatic differentiation, used for vision, speech, text and recommendation models. Final-year AI&DS labs, AICTE model-curriculum practicals and NPTEL courses are built around them, and the GATE DA (Data Science and AI) paper examines the mathematics they implement — loss functions, gradient descent, regularisation and evaluation metrics — even though it does not ask about API syntax. In Indian hiring (TCS, Infosys, Wipro, Accenture, Fractal, Mu Sigma, ZS, Flipkart, Swiggy, Razorpay, Jio, HDFC, Krutrim, Sarvam AI) a data-science interview almost always includes "walk me through your scikit-learn pipeline" and "write a PyTorch training loop". All three are free and open source (scikit-learn BSD-3, PyTorch BSD-style, TensorFlow Apache 2.0).

Explained

Which library for which job

scikit-learn: tabular/structured data up to a few million rows, CPU only, no GPU training. Linear and logistic regression, ridge/lasso, SVM, k-NN, naive Bayes, decision trees, random forest, gradient boosting (HistGradientBoostingClassifier), k-means, DBSCAN, PCA. Best accuracy-per-effort on most Indian business datasets (churn, credit risk, demand).
PyTorch: research and most new deep learning. Define-by-run (dynamic) graphs, plain Python control flow, easiest to debug; dominant in papers and in LLM/vision work (Hugging Face, torchvision).
TensorFlow + Keras: deep learning with a strong deployment story — TF Serving, TFLite/LiteRT for Android and microcontrollers, TFX pipelines, TensorBoard. Keras gives a high-level compile/fit API.
Rule of thumb: rows-and-columns problem → scikit-learn; images, audio, text, sequences, or a custom loss/architecture → PyTorch or TensorFlow. Boosted trees (XGBoost, LightGBM, CatBoost) still beat neural nets on most tabular data.
They interoperate: all three accept NumPy arrays; torch.from_numpy and tensor.numpy() share memory. Keras 3 (default from TF 2.16) runs on TensorFlow, PyTorch or JAX backends. Versions move fast — scikit-learn 1.x, PyTorch 2.x, TensorFlow 2.x are current families, but check the installed version with __version__ before relying on any API detail.

The scikit-learn estimator API

Every object is an estimator with a fit(X, y) method. X is 2-D of shape (n_samples, n_features); y is 1-D. A 1-D X must be reshaped — X.reshape(-1, 1).
Three mixins: *predictors* add predict (and predict_proba/decision_function), *transformers* add transform and fit_transform, *models* add score (R² for regressors, accuracy for classifiers).
Hyperparameters are set in the constructor (C, max_depth, n_estimators); learned parameters appear after fitting with a trailing underscore (coef_, intercept_, feature_importances_, classes_, n_iter_).
**train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)** — stratify keeps class proportions, essential for imbalanced data; random_state makes the split reproducible.
Pipeline: Pipeline([("sc", StandardScaler()), ("clf", LogisticRegression())]) chains transformers and ends in an estimator. The pipeline's fit calls fit_transform on each step; its predict only transforms. This is the single defence against data leakage, because the scaler is fitted on training folds only.
**ColumnTransformer** applies different transformers to different columns (numeric vs categorical) inside one estimator; make_pipeline/make_column_transformer auto-name steps.
Model selection: cross_val_score, cross_validate, KFold, StratifiedKFold, TimeSeriesSplit (never shuffle time series); GridSearchCV (exhaustive) and RandomizedSearchCV (sampled, better when the grid is large). After search, best_params_, best_score_ and best_estimator_ (refitted on all data by default).
Metrics: accuracy_score, precision_recall_fscore_support, confusion_matrix, classification_report, roc_auc_score, mean_squared_error, mean_absolute_error, r2_score. scoring="f1_macro" etc. in CV.
Persistence: joblib.dump(model, "model.joblib"). A pickle is tied to library versions and is unsafe to load from untrusted sources — record the version alongside it.
Limitations: single machine, no GPU, no autograd, no sequence models; MLPClassifier exists but is a toy next to PyTorch/Keras.

PyTorch: tensors, autograd and the training loop

Tensor = n-dimensional array with a dtype (float32 default for data, int64 for class labels) and a device (cpu, cuda). torch.zeros, ones, randn, arange, tensor(...). Shapes change with view/reshape, permute, squeeze, unsqueeze.
Autograd: any tensor with requires_grad=True records operations in a dynamic graph; loss.backward() walks it backwards (reverse-mode automatic differentiation, the chain rule) and accumulates ∂loss/∂θ into each θ.grad.
Gradients accumulate — you must call optimizer.zero_grad() (or set_to_none=True) each iteration, otherwise gradients from previous batches add up.
**nn.Module:** subclass it, create layers in __init__, define forward(x). Never call forward directly — call model(x) so hooks run. nn.Sequential for simple stacks. model.parameters() feeds the optimizer.
Common layers/losses: nn.Linear, Conv2d, MaxPool2d, BatchNorm2d, Dropout, LSTM, Embedding, MultiheadAttention; nn.MSELoss, nn.CrossEntropyLoss (takes raw logits and integer class indices — it applies log-softmax internally, so never add a softmax layer before it), nn.BCEWithLogitsLoss.
Optimisers: torch.optim.SGD(params, lr, momentum), Adam, AdamW; schedulers such as StepLR, CosineAnnealingLR, ReduceLROnPlateau.
Data: Dataset implements __len__ and __getitem__; DataLoader(ds, batch_size=64, shuffle=True, num_workers=4) batches, shuffles and prefetches.
The canonical loop: for each epoch → for each batch → zero_grad → forward → loss → backward → step.
**model.train() vs model.eval()** switches dropout off and makes batch-norm use running statistics; wrap validation in with torch.no_grad(): to save memory and time. Forgetting either is the classic bug.
GPU: device = "cuda" if torch.cuda.is_available() else "cpu"; move both model and batch. Mixed precision (torch.autocast) and torch.compile(model) (PyTorch 2.x) speed training up.
Saving: torch.save(model.state_dict(), path) then model.load_state_dict(torch.load(path)) — save the state dict, not the whole object. Export via TorchScript or ONNX for serving.

model = nn.Sequential(nn.Linear(20, 64), nn.ReLU(), nn.Linear(64, 3))
crit, opt = nn.CrossEntropyLoss(), torch.optim.Adam(model.parameters(), lr=1e-3)
for epoch in range(10):
    model.train()
    for xb, yb in train_loader:
        xb, yb = xb.to(device), yb.to(device)
        opt.zero_grad()
        loss = crit(model(xb), yb)   # yb = int64 class indices
        loss.backward()
        opt.step()

TensorFlow and Keras

TF2 is eager by default — operations execute immediately like NumPy. @tf.function traces a Python function into a static graph for speed and for export; tf.Variable is mutable trainable state, tf.Tensor is immutable.
**tf.GradientTape** is TensorFlow's autodiff: operations inside the with block are recorded, then tape.gradient(loss, model.trainable_variables) returns gradients, applied with optimizer.apply_gradients(zip(grads, vars)). Use persistent=True for more than one gradient call.
Three Keras model styles: *Sequential* (linear stack), *Functional* (Input → layers → Model(inputs, outputs); supports multiple inputs/outputs and skip connections), *subclassing* keras.Model (full control, like PyTorch).
**model.compile(optimizer="adam", loss="sparse_categorical_crossentropy", metrics=["accuracy"])** then model.fit(X, y, epochs, batch_size, validation_split=0.2, callbacks=[...]), model.evaluate, model.predict. model.summary() prints layers, output shapes and parameter counts.
Loss choice: sparse_categorical_crossentropy for integer labels, categorical_crossentropy for one-hot, binary_crossentropy for two classes. If the last layer has no softmax, pass from_logits=True.
Callbacks: EarlyStopping(monitor="val_loss", patience=5, restore_best_weights=True), ModelCheckpoint, ReduceLROnPlateau, TensorBoard, CSVLogger.
**tf.data:** Dataset.from_tensor_slices(...).shuffle(1000).batch(32).prefetch(tf.data.AUTOTUNE) — the standard input pipeline; cache() for small datasets.
Saving and deployment: model.save("m.keras") (Keras 3 format) or SavedModel directory; model.save_weights for weights only. Convert to TFLite/LiteRT for Android and embedded boards, serve with TF Serving, orchestrate with TFX.
Comparison with PyTorch: Keras needs far less boilerplate for standard training; PyTorch gives clearer stack traces and easier custom loops. Both compute the same mathematics.

Training mechanics common to both frameworks

Epoch = one pass over the training set; batch = samples per gradient update; step/iteration = one update. Larger batches are faster per epoch but generalise slightly worse and need a higher learning rate.
Learning rate is the most important hyperparameter: too high → loss diverges to NaN; too low → very slow. Typical starting points: 1e-3 for Adam, 1e-2 with SGD+momentum 0.9.
Weight initialisation: Glorot/Xavier for tanh and sigmoid, He/Kaiming for ReLU; zero initialisation fails to break symmetry — all units in a layer receive identical gradients and stay identical — so it must never be used for weights (biases may start at zero).
Activations: ReLU (default hidden), LeakyReLU/GELU, sigmoid for binary output, softmax for multi-class output, linear for regression output.
Vanishing/exploding gradients: mitigated by ReLU, batch/layer normalisation, residual connections, gradient clipping (clip_grad_norm_).
Overfitting control: more data and augmentation, dropout, L2 (weight decay), early stopping, smaller model. Diagnose from the gap between training and validation curves.
Reproducibility: seed Python, NumPy and the framework (torch.manual_seed, tf.random.set_seed); GPU kernels are still not bit-exact unless deterministic algorithms are forced.
Data protection: personal data used to train models in India falls under the Digital Personal Data Protection Act, 2023 (consent, purpose limitation, data-fiduciary duties); rules were being notified in stages, so confirm the current position. RBI-regulated lenders also face model-governance and explainability expectations — hence SHAP/LIME and scikit-learn's permutation_importance in BFSI projects.

Key points

Estimator API - every scikit-learn object follows fit, then predict or transform; this uniformity is why pipelines work.
**fit_transform vs transform** - fit only on training data; using fit_transform on test data leaks information and inflates scores.
Pipeline - bundles preprocessing plus model into one estimator so cross-validation refits preprocessing inside every fold.
**random_state** - fixes the pseudo-random seed; without it splits and models change between runs.
Trailing underscore - coef_, classes_ are learned attributes available only after fit.
GridSearchCV cost - number of fits = (parameter combinations) × k folds, plus one refit; n_jobs=-1 parallelises over CPU cores.
Tensor - GPU-capable n-D array; float32 features, int64 labels for classification.
Autograd - reverse-mode automatic differentiation; loss.backward() in PyTorch, tape.gradient() in TensorFlow.
**zero_grad()** - mandatory in PyTorch because .grad accumulates across backward passes.
**model.train() / model.eval()** - toggles dropout and batch-norm behaviour; torch.no_grad() disables graph building for inference.
**CrossEntropyLoss** - expects logits, not softmax probabilities; Keras equivalent needs from_logits=True if the output layer is linear.
Eager vs graph - TF2 runs eagerly; @tf.function compiles to a graph for speed and export.
**Keras fit** - handles the loop, metrics, validation and callbacks; EarlyStopping with restore_best_weights is the standard anti-overfitting callback.
Dense layer parameters - (inputs + 1) × units, the +1 being the bias.
Dynamic vs static - PyTorch builds the graph each forward pass (easy debugging); TF graph mode optimises a fixed graph (faster, deployable).
Persistence - joblib for scikit-learn, state_dict for PyTorch, .keras/SavedModel for Keras; ONNX is the cross-framework exchange format.
No GPU in scikit-learn - for large deep models or big-data training, move to PyTorch/TensorFlow or Spark MLlib.
Tabular reality check - gradient-boosted trees usually beat neural networks on structured data.

Formulas and examples

Mean squared error: MSE = (1/n) Σ(yᵢ − ŷᵢ)² — y actual, ŷ predicted, n samples; RMSE = √MSE (same unit as y, e.g. ₹)
Coefficient of determination: R² = 1 − SS_res/SS_tot = 1 − Σ(yᵢ−ŷᵢ)² / Σ(yᵢ−ȳ)² (dimensionless)
Binary cross-entropy: L = −(1/n) Σ [yᵢ ln pᵢ + (1−yᵢ) ln(1−pᵢ)] — p = predicted probability
Categorical cross-entropy: L = −(1/n) Σᵢ Σ_c y_{ic} ln p_{ic} (nats, natural log)
Sigmoid σ(z) = 1/(1+e^−z); Softmax p_c = e^{z_c} / Σ_j e^{z_j}
Gradient descent: θ ← θ − η ∂L/∂θ — η learning rate (dimensionless step scale)
SGD with momentum: v ← βv + ∂L/∂θ; θ ← θ − η v (β ≈ 0.9)
Adam: m ← β₁m + (1−β₁)g; v ← β₂v + (1−β₂)g²; m̂ = m/(1−β₁ᵗ); v̂ = v/(1−β₂ᵗ); θ ← θ − η·m̂/(√v̂ + ε) (β₁=0.9, β₂=0.999, ε≈10⁻⁸; 10⁻⁷ in Keras)
L2 (weight decay): L_total = L + λ Σθ²; scikit-learn uses C = 1/(2λ)-like inverse strength — larger C means weaker regularisation
Standardisation (StandardScaler): z = (x − μ)/σ, fitted on training data only
Steps per epoch = ⌈N/B⌉ — N training samples, B batch size; total updates = epochs × steps
Dense layer parameters = (n_in + 1) × n_out; Conv2D parameters = (k_h × k_w × C_in + 1) × C_out
Conv/pool output size = ⌊(W − K + 2P)/S⌋ + 1 — W input size, K kernel, P padding, S stride ("same" padding with S=1 keeps W)
Accuracy = (TP+TN)/(TP+TN+FP+FN); Precision = TP/(TP+FP); Recall = TP/(TP+FN); F1 = 2PR/(P+R) = 2TP/(2TP+FP+FN)
Fits in a grid search = (Π of parameter-value counts) × k + 1 refit

Example 1 — classification report from a confusion matrix. A fraud classifier on 1,000 test transactions gives TP = 120, FP = 30, FN = 20, TN = 830.

Accuracy = (120 + 830)/1,000 = 0.950 (95.0%)
Precision = 120/(120+30) = 120/150 = 0.800
Recall = 120/(120+20) = 120/140 = 0.857
F1 = 2×120/(2×120 + 30 + 20) = 240/290 = 0.828
Reading: of 150 flagged transactions, 30 are false alarms; of 140 real frauds, 20 are missed. With only 14% positives, accuracy is misleading — report F1 or ROC-AUC, and use stratify=y plus class_weight="balanced".

Example 2 — one gradient-descent step, hand versus autograd. Model ŷ = w·x, loss L = (ŷ − y)², with x = 2, y = 5, initial w = 1, η = 0.1.

Forward: ŷ = 2, error = −3, L = 9.
∂L/∂w = 2(wx − y)·x = 2(−3)(2) = −12.
Update: w ← 1 − 0.1(−12) = 2.2.
Check: ŷ = 4.4, error = −0.6, L = 0.36 — loss fell from 9 to 0.36.
In PyTorch: w = torch.tensor(1.0, requires_grad=True); L = (w*2 - 5)**2; L.backward() → w.grad is -12.0. If backward() is called twice without zero_grad(), w.grad becomes −24 — the accumulation trap.

Example 3 — softmax and cross-entropy from logits. A 3-class head outputs logits z = [2.0, 1.0, 0.1]; true class is 0.

e² = 7.389, e¹ = 2.718, e^0.1 = 1.105; sum = 11.212.
p = [7.389/11.212, 2.718/11.212, 1.105/11.212] = [0.659, 0.242, 0.099] (sums to 1).
Loss = −ln(0.659) = 0.417 nats. A perfect prediction gives 0; a uniform guess gives −ln(1/3) = 1.099, so the model is already better than chance.
Equivalent code: nn.CrossEntropyLoss()(torch.tensor([[2.0,1.0,0.1]]), torch.tensor([0])) → 0.417.

Example 4 — parameter counts (MNIST-style). Keras MLP: Input 784 → Dense 128 (ReLU) → Dense 64 (ReLU) → Dense 10 (softmax).

784×128 + 128 = 100,352 + 128 = 100,480; 128×64 + 64 = 8,192 + 64 = 8,256; 64×10 + 10 = 650.
Total = 100,480 + 8,256 + 650 = 109,386 trainable parameters (≈ 0.42 MB in float32).
Now a CNN on 32×32×3 images: Conv2D(16, 3×3, padding "same") → output 32×32×16, params = (3×3×3 + 1)×16 = 28×16 = 448. MaxPool 2×2 → 16×16×16. Flatten → 4,096. Dense 10 → 4,096×10 + 10 = 40,970. CNN total = 41,418 — far fewer than a dense layer on raw pixels, because kernels are shared.
With "valid" padding instead, output = (32 − 3)/1 + 1 = 30, i.e. 30×30×16.

Example 5 — training budget and search cost. N = 60,000 training images, batch size B = 128, 10 epochs.

Steps per epoch = ⌈60,000/128⌉ = ⌈468.75⌉ = 469; total updates = 4,690.
At 12 ms per step, training time ≈ 4,690 × 0.012 = 56.3 s per run.
Now tune a scikit-learn SVM with GridSearchCV over C ∈ {0.1, 1, 10} and gamma ∈ {0.001, 0.01, 0.1, 1} using 5-fold CV: 3 × 4 = 12 combinations → 12 × 5 = 60 fits, plus 1 refit on the full training set = 61 fits. Each fold trains on 4/5 of the data. RandomizedSearchCV(n_iter=5) would need only 5 × 5 + 1 = 26 fits.

Aspect	scikit-learn	PyTorch	TensorFlow / Keras
Main use	Classical ML on tabular data	Deep learning, research	Deep learning, production
Core API	`fit` / `predict` / `transform`	Tensors, `nn.Module`, autograd	Keras `compile` / `fit`; `GradientTape`
Graph	None	Dynamic (define-by-run)	Eager, graph via `@tf.function`
GPU / TPU	No	CUDA/ROCm GPUs	GPUs and TPUs
Training loop	Hidden inside `fit`	Written by you	`fit`, or custom with a tape
Save format	joblib pickle	`state_dict`, TorchScript, ONNX	`.keras`, SavedModel, TFLite
Deployment	Flask/FastAPI service	TorchServe, ONNX Runtime	TF Serving, TFLite, TFX, TF.js
Learning curve	Easiest	Pythonic, moderate	Easy with Keras, deep stack below

Where it is used

IT services and analytics (TCS, Infosys, Wipro, Cognizant, Accenture, Fractal, Mu Sigma, ZS, EXL): scikit-learn pipelines for churn, propensity, pricing and demand forecasting delivered to client dashboards.
BFSI and fintech (HDFC, ICICI, Bajaj Finserv, Paytm, PhonePe, Razorpay, NPCI): credit scoring, fraud and anti-money-laundering models — logistic regression and gradient boosting preferred for explainability, with deep models for sequence-based fraud.
Consumer internet (Flipkart, Meesho, Swiggy, Zomato, Ola, Myntra, Hotstar): PyTorch/TensorFlow recommenders, ETA and search-ranking models; TFLite for on-device features in the app.
Healthtech and manufacturing (Apollo, Qure.ai, Niramai, Tata Steel, L&T, Siemens India): CNNs for X-ray/retina screening and visual defect inspection; predictive maintenance on sensor data.
Language and GenAI (Krutrim, Sarvam AI, AI4Bharat at IIT Madras, Bhashini): PyTorch plus Hugging Face for Indic speech and language models; ISRO, DRDO and IMD labs use both frameworks for imagery and weather models.
Job roles: Data Scientist, ML Engineer, Data Analyst, AI/Computer-Vision Engineer, MLOps Engineer; interviews at these firms expect a scikit-learn pipeline and a hand-written PyTorch or Keras training loop on a whiteboard.

Interview questions

Q: Why must a scaler be fitted inside a Pipeline rather than on the full dataset? A: Fitting on all data lets test-set mean and standard deviation leak into training, giving optimistic validation scores; a Pipeline refits the scaler on each training fold only.
Q: Difference between fit, transform and fit_transform? A: fit learns parameters, transform applies them, fit_transform does both — use it only on training data.
Q: What does random_state do? A: It seeds the pseudo-random generator so splits, shuffles and stochastic algorithms are reproducible; it does not affect model quality.
Q: Why call optimizer.zero_grad() in PyTorch? A: backward() accumulates gradients into .grad, so without clearing them the update uses the sum of gradients from earlier batches — useful only when deliberately accumulating gradients over micro-batches.
Q: What breaks if you forget model.eval()? A: Dropout stays active and batch-norm keeps using batch statistics, so validation predictions become noisy and inconsistent between runs.
Q: Should a softmax layer precede nn.CrossEntropyLoss? A: No — it already applies log-softmax internally; adding another softmax flattens gradients and slows learning.
Q: Eager versus graph execution in TensorFlow? A: Eager runs ops immediately and is easy to debug; @tf.function traces the code into a static graph that is faster and exportable, but Python side effects then run only during tracing.
Q: How does autograd work? A: The framework records every operation on tracked tensors in a graph, then applies the chain rule backwards from the scalar loss to every parameter (reverse-mode automatic differentiation).
Q: When would you use the Keras Functional API over Sequential? A: When the model is not a single chain — multiple inputs or outputs, shared layers, or skip/residual connections.
Q: How many model fits does a 4×3 grid with 5-fold CV perform? A: 12 × 5 = 60, plus one refit on the full training set with the best parameters (61), so parallelise with n_jobs=-1 or switch to RandomizedSearchCV.
Q: Your model has 99% accuracy but misses most frauds — what do you report? A: The class imbalance makes accuracy useless; report precision, recall, F1 and ROC-AUC/PR-AUC, and use stratified splits with class weights or resampling.
Q: Why is scikit-learn not used for image classification? A: It has no GPU support, no convolution layers and no autograd, so training deep CNNs would be infeasibly slow and architecturally impossible.
Q: Loss is NaN after a few steps — first checks? A: Learning rate too high, unnormalised inputs, a log or division by zero in a custom loss, or exploding gradients; lower the learning rate, scale inputs, and clip gradient norms.

Common mistakes

Scaling or encoding the whole dataset before train_test_split — data leakage, the single most-penalised error in lab vivas and interviews.
Passing a 1-D array as X (ValueError: Expected 2D array) or passing one-hot labels to a loss that expects integer class indices (and vice versa in Keras).
Forgetting zero_grad(), or forgetting model.eval() and torch.no_grad() during validation and inference.
Reporting training accuracy, or tuning hyperparameters on the test set, instead of using cross-validation and keeping the test set untouched until the end.
Shuffling time-series data in KFold — this trains on the future to predict the past; use TimeSeriesSplit.
Leaving the model on CPU while the data is on GPU (or mixing float32 and float64) and then blaming the framework for the type/device error.
Quoting accuracy on an imbalanced dataset without the confusion matrix, and writing sklearn metric formulas with precision and recall interchanged.