API integration and middleware

In short

API integration is the work of making separate software systems — an ERP, a payment gateway, a CRM, a bank's core, a mobile app — exchange data reliably and securely; middleware is the layer of software that sits between them and does the connecting, routing, translating, queueing and securing. The subject matters because almost no Indian enterprise runs on one system: a retailer's order has to touch an e-commerce front end, a payment aggregator, a warehouse system, GST e-invoicing and a CRM within seconds. It is examined in university papers on enterprise computing, SOA and distributed systems, and in GATE-style questions on RPC, messaging and distributed transactions. In placements it is the core of every integration, backend and platform-engineering interview at TCS, Infosys, Wipro, Accenture, Cognizant, LTIMindtree, Razorpay, PhonePe, Zoho, Freshworks and every bank's technology arm.

Explained

What middleware is, and its families

• Middleware - software that provides common services (communication, data translation, transactions, security) to applications so each application need not implement them. It is "glue" plus "plumbing", not business logic.
• Why it exists: heterogeneity (different languages, data formats, protocols, release cycles), and the need for loose coupling - a change in one system should not force a change in the others.
• Types:
  • RPC / remote-invocation middleware - call a remote procedure as if local (gRPC, Java RMI, historically CORBA/ORB, DCOM).
  • Message-oriented middleware (MOM) - asynchronous messages via queues and topics (IBM MQ, RabbitMQ, ActiveMQ, Apache Kafka, Amazon SQS/SNS).
  • Object/component and application servers - Tomcat, JBoss, WebLogic, WebSphere; provide threading, pooling, transactions, security to hosted components.
  • Transaction-processing middleware - coordinates distributed transactions (XA, Tuxedo, CICS); guarantees atomicity across resources.
  • Database middleware - ODBC/JDBC drivers, connection pools, ORMs, change-data-capture tools.
  • Integration middleware - ESB, API gateway, iPaaS, ETL/ELT tools, service mesh.
• Terminology for exams: EAI (enterprise application integration) is the discipline; SOA (service-oriented architecture) is the architectural style; ESB is the classic middleware product that implements it; microservices + API gateway is the modern realisation.

Integration styles and architectures

• Four classical integration styles (Hohpe & Woolf, the standard syllabus list):
  1. File transfer - one system writes a file, the other reads it (SFTP batch, NEFT files, NACH mandates). Simple, high latency, no schema enforcement.
  2. Shared database - both read one schema. Immediate consistency, but the tightest possible coupling; a schema change breaks everyone.
  3. Remote procedure invocation - synchronous request/response (REST, gRPC, SOAP). Easy to reason about; caller is blocked and fails if callee is down.
  4. Messaging - asynchronous, decoupled, buffered. Most resilient; hardest to debug and reason about ordering.
• Topologies:
  • Point-to-point - every pair wired directly. For *n* systems the worst case is *n(n−1)/2* interfaces - the "spaghetti" problem.
  • Hub-and-spoke / ESB - all traffic through a broker; *n* adapters instead of *n²* links, central routing and transformation, but a single operational bottleneck.
  • API-led / microservices - each capability exposes a versioned API; a gateway handles cross-cutting concerns; a service mesh (Istio/Envoy, Linkerd) handles service-to-service mTLS, retries and telemetry as a sidecar.
  • Event-driven architecture - producers publish events to a log/topic; consumers subscribe. Enables fan-out, replay and CQRS/event sourcing.
• Core enterprise integration patterns (EIP) - implemented in Apache Camel, Spring Integration, MuleSoft: message channel, message router, content-based router, message translator, message filter, splitter, aggregator, scatter-gather, normaliser, canonical data model, wire tap, dead-letter channel, idempotent receiver, claim check, process manager/orchestrator.
• Orchestration vs choreography - orchestration has a central controller calling services in order (BPEL, Step Functions, Camunda); choreography has services reacting to each other's events with no conductor.
• ETL vs API - ETL/ELT moves bulk data on a schedule into a warehouse; APIs move small payloads on demand. Both are integration; the choice is latency versus volume.

API styles, contracts and data formats

• SOAP - XML envelope (optional Header + mandatory Body, with Fault carried inside the Body) over HTTP/JMS, contract in WSDL, discovery historically via UDDI, extensions as WS-\* (WS-Security, WS-ReliableMessaging, WS-AtomicTransaction). Verbose but strongly contracted; still the norm in banking, insurance and telecom back ends.
• REST - resource URIs, HTTP verbs, JSON; described by OpenAPI/Swagger. (Treated in detail in its own note; here it is simply one integration style among several.)
• GraphQL - one endpoint, client-specified query shape; avoids over-fetching and multiple round trips; harder to cache and rate-limit, and vulnerable to deeply nested expensive queries.
• gRPC - HTTP/2 + Protocol Buffers, strongly typed IDL, code generation, bidirectional streaming; 3-10× smaller payloads than JSON and the usual choice for internal service-to-service calls. Needs a proxy (grpc-web) for browsers.
• Webhooks - the server calls *you* when an event happens (payment captured, shipment delivered). Must be verified with an HMAC-SHA256 signature over the raw body, must be idempotent (duplicates are normal), must reply 2xx fast and process asynchronously.
• Streaming and real-time - WebSocket, Server-Sent Events, MQTT for IoT and low-bandwidth links.
• Data formats and mapping - XML (XSD validation, XSLT/XPath transformation), JSON (JSON Schema, JSONPath), CSV for batch, Avro/Protobuf/Thrift binary with schema registries for streams. A canonical data model plus a mapping document is the standard way to avoid *n²* format conversions.
• Versioning and compatibility - URI version (/v2/), header or media-type version; never remove or retype a field in place. Backward compatible = old clients still work; forward compatible = old code tolerates new fields (additive change only).
• Contract-first development - write OpenAPI/WSDL/.proto first, generate stubs, mock the API, test with contract tests (Pact) so provider and consumer can be built in parallel.

Messaging middleware, delivery guarantees and transactions

• Queue (point-to-point) - one message consumed by exactly one of the competing consumers; used for work distribution. Topic (publish/subscribe) - every subscriber gets a copy; used for event fan-out.
• Standards and products - JMS (Java API, not a wire protocol), AMQP (wire protocol - RabbitMQ), MQTT (lightweight pub/sub for devices), STOMP, Kafka (partitioned, replicated, retained commit log - a stream store rather than a classic broker).
• Delivery semantics: at-most-once (fire and forget; may lose), at-least-once (retry until acknowledged; may duplicate - the practical default), exactly-once (only within a transactional boundary, e.g. Kafka read-process-write with transactions; across heterogeneous systems it is achieved as *at-least-once delivery + idempotent processing*).
• Ordering - global ordering is expensive; Kafka guarantees order only within a partition, so a key (e.g. account number) must be chosen to co-locate related events. Parallel consumers destroy order unless partitioned by key.
• Reliability mechanisms - persistent messages, producer acknowledgements, consumer acknowledgement after processing, dead-letter queue (DLQ) for poison messages, redelivery limits, exponential backoff with jitter, message TTL, replay from offset.
• Backpressure and lag - consumer lag is the queue depth; alert on lag, not just on errors. Unbounded queues turn a throughput problem into a latency and memory problem.
• Distributed transactions:
  • Two-phase commit (2PC/XA) - prepare then commit via a coordinator; gives ACID across resources but blocks on coordinator failure and scales poorly.
  • Saga pattern - a sequence of local transactions with compensating transactions (refund, cancel) to undo; eventual consistency, no global lock. The standard microservices answer.
  • Transactional outbox + CDC - write the business row and the event row in one local DB transaction, then publish from the outbox (Debezium) so the DB and the broker never diverge. Avoids the classic "DB committed but message lost" dual-write bug.
  • Idempotency key - a client-supplied unique key stored server-side so a retried POST creates one payment, not two. Mandatory on money movement.

API gateways, management and security

• API gateway - single entry point in front of many services; does routing, protocol translation, request/response transformation, aggregation, authentication, authorisation, rate limiting, quotas, caching, logging, metrics, canary routing. Products: Apigee, Kong, AWS API Gateway, Azure API Management, WSO2, Spring Cloud Gateway, NGINX, Tyk.
• Gateway vs ESB vs service mesh - gateway governs north-south (outside-in) traffic and is API-centric; ESB does heavyweight transformation and orchestration between back ends; mesh governs east-west service-to-service traffic transparently.
• API management lifecycle - design → mock → publish to a developer portal with docs and sandbox keys → onboard consumers with plans/tiers → monitor and monetise → deprecate with a sunset header and notice period.
• Authentication and authorisation:
  • API key - identifies an application, not a user; low security, good for quotas.
  • Basic auth over TLS - simple, poor rotation story.
  • OAuth 2.0 - delegated authorisation; grants: authorisation code + PKCE (apps/SPAs), client credentials (server-to-server), device code; refresh_token to renew. OpenID Connect adds authentication and an id_token.
  • JWT - signed (RS256/HS256) token carrying claims (iss, sub, aud, exp, scope); stateless validation at the gateway. Must verify signature, issuer, audience and expiry, and keep lifetimes short since a JWT cannot be revoked easily.
  • mTLS - mutual certificate authentication, used between banks, NPCI members and Account Aggregator participants.
  • HMAC request signing - signature over method, path, body and timestamp; defeats tampering and replay (with a nonce/timestamp window).
• Other gateway policies - IP allow-listing, payload size limits, schema validation (reject malformed input at the edge), CORS, WAF, field-level encryption/tokenisation of card and Aadhaar data, PII masking in logs.
• OWASP API Security Top 10 essentials - broken object-level authorisation (always check the caller owns the id), broken authentication, excessive data exposure, lack of rate limiting, mass assignment, SSRF.

Resilience, performance and observability

• Timeouts - every outbound call needs one; a missing timeout propagates one slow dependency into a full outage. Caller timeout should exceed the downstream p99, and the budget must shrink down the call chain.
• Retries - only for idempotent or key-protected operations, with exponential backoff and jitter, and a bounded attempt count. Naive retries cause a retry storm that multiplies load on an already failing service.
• Circuit breaker - states Closed → Open → Half-open; after a failure threshold the breaker opens and fails fast for a cool-down, then probes. Resilience4j, Hystrix (now inactive), Envoy outlier detection.
• Bulkhead - separate thread pools/connection pools per dependency so one slow partner cannot exhaust the whole server. Rate limiting algorithms: fixed window, sliding window, token bucket (allows bursts), leaky bucket (smooths output).
• Caching - gateway response cache, ETag/If-None-Match, Redis for lookup data; cache only idempotent GETs and respect privacy of personal data.
• Observability - correlation ID / trace ID propagated through every hop (W3C traceparent), structured logs, metrics (RED: rate, errors, duration), distributed tracing (OpenTelemetry, Jaeger, Zipkin), synthetic monitoring. Agree SLI/SLO and an error budget, and define them on p95/p99 latency, never on the mean.
• Testing integrations - contract tests, mock/stub servers (WireMock), sandbox environments of the partner, replayed production payloads, chaos tests (kill the broker, add 2 s latency), load tests with JMeter/k6/Gatling.

Indian platforms, standards and compliance

• IndiaStack APIs - UPI (NPCI; collect/pay, UPI AutoPay mandates, callbacks over mTLS with member certificates), Aadhaar authentication and e-KYC (UIDAI, through licensed AUA/KUA entities; governed by the Aadhaar Act, 2016 as amended in 2019 - purpose limitation and no storage of biometrics), DigiLocker and API Setu for verified documents, e-Sign, Jan Parichay for government single sign-on.
• Tax and compliance APIs - GSTN APIs consumed through a GSP/ASP (GST Suvidha Provider / Application Service Provider) model, e-invoice (IRP) APIs returning an IRN and signed QR, e-Way bill APIs; TRACES/Income-tax reporting portals. GST on software, SaaS and API subscription services is at the 18% slab (slabs are 5%/18%/40% since 22 September 2025).
• Financial data sharing - Account Aggregator framework under RBI's NBFC-AA master direction, with FIP/FIU/AA roles and a machine-readable consent artefact; ecosystem standards maintained by Sahamati. OCEN for credit rails, Bharat Bill Payment for bills.
• Commerce - ONDC built on the Beckn protocol: an open, asynchronous, catalogue-and-callback API specification (search/on_search, select, init, confirm, status) that buyer and seller apps implement rather than integrating pairwise.
• Regulatory constraints on integrations - RBI's 2018 directive requires payment-system data to be stored in India; card data must be tokenised rather than stored by merchants and aggregators (card-on-file tokenisation in force since October 2022); RBI Payment Aggregator/Payment Gateway guidelines govern who may touch payment flows; digital-lending rules restrict data collection by lending service providers.
• Data protection - the Digital Personal Data Protection Act, 2023 governs personal data in every API payload: notice and consent, purpose limitation, data minimisation, breach notification to the Data Protection Board, and obligations on data fiduciaries and their processors (contracts with integration vendors). Its rules were notified with phased implementation; the exact compliance dates have been changing, so verify them before relying on a date. The Information Technology Act, 2000 (s.43A, s.72A and the CERT-In directions of 2022 on incident reporting and log retention) continues to apply. Security management is usually certified against IS/ISO/IEC 27001 and card flows against PCI DSS.

Key points

• Middleware - the layer that connects applications by handling communication, translation, transactions and security instead of business logic.
• Loose coupling - the real goal of integration; measured by whether one system can change without forcing changes in others.
• Four integration styles - file transfer, shared database, remote procedure invocation, messaging.
• Point-to-point cost - *n(n−1)/2* interfaces; a hub/ESB or gateway reduces it to *n* adapters.
• ESB vs API gateway vs service mesh - heavy transformation hub vs north-south API edge vs transparent east-west sidecar.
• Synchronous vs asynchronous - request/response gives simplicity and immediate answers; messaging gives buffering, fan-out and survival of downstream outages.
• Delivery semantics - at-most-once, at-least-once, exactly-once; practical systems use at-least-once plus idempotent receivers.
• Idempotency key - client-supplied unique key that makes a retried write safe; essential for payments.
• DLQ - dead-letter queue holding messages that failed repeatedly, so one poison message does not block the stream.
• Kafka ordering - guaranteed only within a partition, so partition by a business key.
• 2PC/XA - ACID across resources but blocking and non-scalable; saga with compensating transactions is the microservices alternative.
• Transactional outbox - single local transaction writes data and event, publisher ships it later; cures the dual-write problem.
• OAuth 2.0 grants - authorisation code + PKCE for apps, client credentials for machine-to-machine; OIDC adds identity.
• JWT validation - verify signature, iss, aud and exp; keep lifetimes short because revocation is hard.
• Circuit breaker states - Closed, Open, Half-open; fails fast to protect a struggling dependency.
• Token bucket - rate limiter that permits bursts up to bucket capacity at a fixed refill rate.
• Correlation ID - one identifier propagated through every hop; without it a multi-hop failure cannot be traced.
• Canonical data model + contract-first - one neutral schema and an OpenAPI/WSDL/.proto contract agreed before code, so teams build in parallel.

Formulas and examples

1. Spaghetti versus hub. A manufacturer has 9 systems (SAP ERP, CRM, WMS, MES, HRMS, e-commerce, bank, GST portal, BI).

• Point-to-point worst case: I = 9 × 8 / 2 = 36 interfaces, each needing its own mapping and monitoring.
• Through an integration layer: 9 adapters + a canonical model. At roughly 8 person-days per interface, 36 × 8 = 288 days against 9 × 12 = 108 days including the hub's extra effort - about a 62% saving, at the cost of one shared component to operate.

2. Chained availability and the case for redundancy. A customer request passes gateway (99.95%), auth service (99.9%), core service (99.9%) and database (99.95%).

• A_total = 0.9995 × 0.999 × 0.999 × 0.9995 = 0.99700 = 99.70%.
• Downtime = 0.00300 × 43,200 ≈ 129 minutes/month - far worse than any single component, which is why "four nines on each box" does not give four nines end to end.
• Run the auth service as 2 independent instances: A = 1 − (1 − 0.999)² = 1 − 0.000001 = 99.9999%. New total = 0.9995 × 0.999999 × 0.999 × 0.9995 ≈ 0.99800 → ≈ 86 min/month, a 33% cut from one change.

3. Little's Law sizing of an API gateway. Traffic 1,200 req/s, average response 150 ms.

• In-flight requests L = 1,200 × 0.150 = 180. With a 200-thread pool, utilisation = 180/200 = 90% - too close to saturation.
• If a downstream slows to 400 ms: L = 1,200 × 0.4 = 480 > 200 threads, so requests queue, latency climbs further and the pool exhausts. Fix with a timeout at ~1.5× p99, a circuit breaker, and a bulkhead that caps that dependency at, say, 60 threads.
• Bandwidth at 8 KB per response = 1,200 × 8,192 × 8 ÷ 10⁶ ≈ 78.6 Mbps; gzip at about 4:1 brings it to ≈ 20 Mbps.

4. Queue capacity and a consumer count. Orders arrive at λ = 500 msg/s; one consumer processes μ = 40 msg/s.

• Bare minimum = 500/40 = 12.5 → 13 consumers (ρ ≈ 0.96 - unusable).
• At a 70% utilisation target: 500 ÷ (40 × 0.7) = 17.86 → 18 consumers; use at least 18 partitions so each has one.
• Single-consumer M/M/1 check at λ = 40, μ = 50: ρ = 0.8, W = 1/(50 − 40) = 0.1 s = 100 ms, W_q = 0.8/10 = 80 ms, L_q = 0.64/0.2 = 3.2 messages waiting. Push λ to 48 and W = 1/2 = 500 ms - a 20% traffic rise multiplies latency 5×.
• Storage: 500 msg/s × 2 KB = 1 MB/s ≈ 86.4 GB/day; 7-day retention with replication factor 3 = 86.4 × 7 × 3 ≈ 1.81 TB.

5. Caching and rate limiting. Backend latency 120 ms, gateway cache 5 ms, hit ratio 70%.

• Effective latency = 0.7 × 5 + 0.3 × 120 = 3.5 + 36 = 39.5 ms (a 67% reduction), and backend load drops from 1,200 to 360 req/s.
• Token bucket with capacity 100 and refill 20/s: a burst of 200 simultaneous calls → 100 pass immediately, the remaining 100 are served at 20/s over 5 s, or rejected with HTTP 429 and a Retry-After header.
• Retry amplification: 2 retries on a fully failing dependency means 3× the traffic hits it - hence a circuit breaker rather than blind retries.

Where it is used

• Banking, payments and fintech - UPI, IMPS and NEFT integrations, card tokenisation, Account Aggregator consent flows, core-banking middleware at HDFC, ICICI, SBI, Razorpay, PhonePe, Paytm, Pine Labs; mTLS, HSM-signed requests and reconciliation jobs are everyday work.
• ERP and enterprise IT - SAP Integration Suite/PI-PO, Oracle Integration Cloud, MuleSoft, Boomi, WSO2, TIBCO and IBM ACE connecting SAP/Oracle ERP to CRM, warehouse, dealer portals and GST e-invoicing; the bread-and-butter of TCS, Infosys, Wipro, HCLTech and Accenture delivery.
• E-commerce, logistics and ONDC - catalogue, inventory, payment and shipment APIs across Flipkart/Amazon marketplaces, Delhivery/Shiprocket aggregators, and Beckn-based ONDC buyer/seller apps.
• Government and public digital services - API Setu, DigiLocker, UIDAI e-KYC, GSTN GSP platforms, IRCTC and state service portals; vendors building on IndiaStack rails.
• SaaS products and telecom - Zoho, Freshworks and Chargebee publishing webhooks and REST/GraphQL APIs; telcos integrating OSS/BSS, billing and recharge channels through messaging middleware.
• Job roles - integration developer/architect, API developer, backend engineer, middleware administrator (MQ/Kafka), platform/SRE, solution architect; certifications in MuleSoft, Apigee, Kafka and cloud integration are commonly asked for.

Interview questions

• Q: What is middleware and why is it needed? A: Software between applications that supplies communication, translation, transaction and security services, so heterogeneous systems can integrate without each one re-implementing the plumbing or depending on the other's internals.
• Q: When would you choose messaging over a REST call? A: When the caller must not block or fail if the receiver is down, when traffic is bursty and needs buffering, or when one event must fan out to many consumers; REST is better when the caller needs the answer immediately.
• Q: Difference between an ESB and an API gateway? A: An ESB is an internal hub doing heavy routing, transformation and orchestration between back-end systems; a gateway sits at the edge of the API estate and applies security, rate limiting, routing and observability to externally consumed APIs.
• Q: How do you achieve exactly-once processing? A: In practice with at-least-once delivery plus an idempotent consumer - store a deduplication/idempotency key or use the message id as a primary key - or by using a transactional producer-consumer within one system; true exactly-once across heterogeneous systems is not achievable.
• Q: What is the dual-write problem and how is it solved? A: Writing to the database and publishing to a broker as two separate operations can leave them inconsistent if one fails; the transactional outbox pattern writes both rows in one local transaction and a CDC publisher emits the event afterwards.
• Q: 2PC or saga? A: 2PC gives atomicity but blocks on coordinator failure and does not scale across services or clouds; a saga uses local transactions with compensating actions and accepts eventual consistency, which is the normal microservices choice.
• Q: Which OAuth 2.0 grant for a server-to-server integration, and why not password grant? A: Client credentials, because no end user is present; the resource-owner password grant is deprecated as it forces credential sharing and prevents MFA.
• Q: How do you secure an incoming webhook? A: Verify the HMAC signature over the raw body with the shared secret, check a timestamp/nonce against replay, allow-list source IPs where published, respond 2xx quickly and process asynchronously with idempotent handling of duplicates.
• Q: A downstream partner API becomes slow and your service starts failing. What do you do? A: Enforce a timeout, open a circuit breaker to fail fast, isolate the dependency in its own thread/connection pool (bulkhead), queue or degrade the feature, and alert on p99 latency rather than averages.
• Q: How do you version an API without breaking consumers? A: Make only additive, backward-compatible changes within a version; introduce breaking changes as a new major version (URI or media type), run both in parallel, and publish a deprecation and sunset timeline.
• Q: Why does Kafka not guarantee global ordering? A: Order is maintained only within a partition; messages are spread across partitions for parallelism, so related events must share a partition key such as the account or order id.
• Q: What compliance concerns shape an Indian payment integration? A: Payment data must be stored in India under RBI's 2018 directive, card details must be tokenised rather than stored, the flow must fit RBI's payment-aggregator rules, personal data handling must satisfy the DPDP Act, 2023, and incidents must be reported under the CERT-In directions.

Common mistakes

• Making an outbound call with no timeout, or with a timeout longer than the caller's own, so one slow partner cascades into a full outage.
• Retrying non-idempotent writes without an idempotency key - duplicate payments and duplicate orders, then a retry storm on the failing service.
• Assuming "exactly once" exists end to end, and so not writing a deduplication check in the consumer.
• Confusing authentication with authorisation: validating a JWT's signature but never checking that the caller owns the object id in the URL (broken object-level authorisation).
• Transforming data without a canonical model, producing *n²* bespoke mappings that no one can regression-test.
• Logging full request bodies containing Aadhaar numbers, card data, OTPs or tokens - a DPDP/PCI breach and an audit finding.
• Quoting or designing to average latency, ignoring p95/p99 and consumer lag, so the system looks healthy while a tenth of users time out.

Related topics

REST APIs · Server-side web development · Back-end development with Node.js · Enterprise systems: ERP and CRM · E-commerce systems · Distributed systems and microservices