Indian language computing: Unicode, scripts and input tools

In short

Indian language computing is the set of standards and tools by which Brahmi-derived and Perso-Arabic scripts are encoded, stored, sorted, rendered and typed on computers. Its backbone is Unicode, which assigns one code point per character and replaced the 8-bit ISCII standard (IS 13194:1991) and the glyph-based legacy font encodings such as Krutidev and ML-TTKarthika. It matters because an Indic string is not one byte per letter: a single visible syllable may be several code points, stored in logical order but displayed in a reordered visual order, so naive length, sorting and substring code breaks. It is examined in BA language-technology and computational-linguistics papers, in UGC-CBCS "computer applications in languages" courses, and in placement rounds for localisation, publishing, e-governance and NLP roles (C-DAC, NIC, Bhashini, AI4Bharat, Reverie, Process9, newspapers, Amazon/Google/Microsoft India language teams).

Explained

From ISCII to Unicode

• ISCII (Indian Script Code for Information Interchange), IS 13194:1991 of the Bureau of Indian Standards, is an 8-bit code: 0x00–0x7F is ASCII, 0xA0–0xFF carries one shared Indic chart for the nine major Brahmi-derived scripts, with an ATR (attribute) byte switching the script in which those codes are drawn.
• Its insight: these scripts share a phonetic order (vowels, then velars to labials, then semivowels, sibilants), so one chart serves many scripts. Its limits: 8 bits, one script at a time, no Perso-Arabic, no Ol Chiki or Meetei Mayek, and no multilingual document.
• A separate MeitY/C-DAC effort defined a Perso-Arabic Script Code for Information Interchange (PASCII) for Urdu, Kashmiri and Sindhi; it never displaced Unicode in practice.
• Unicode / ISO/IEC 10646 gives every character in every script one code point in a space of 1,114,112 positions (17 planes × 65,536), with over 150,000 characters and about 170 scripts assigned. Unicode is revised annually (17.0 came in 2025); quote the version your software was tested against.
• The Government of India adopted Unicode as the standard for Indian language data; GIGW 3.0 (Guidelines for Indian Government Websites and Apps) requires government content to be Unicode-compliant rather than font-dependent.
• Key design rule: Unicode encodes characters, not glyphs. क्ष is not encoded; it is क + virama + ष and the *font* draws the ligature.

The Indic blocks and the ISCII-parallel layout

• Each major Brahmi-derived script gets a 128-point block, laid out in the same ISCII-derived order, so corresponding letters sit at the same offset:

• Supplementary and extension blocks: Devanagari Extended (U+A8E0–U+A8FF), Vedic Extensions (U+1CD0–U+1CFF), Tamil Supplement (U+11FC0–U+11FFF), Meetei Mayek (U+ABC0–U+ABFF), and historic scripts beyond the BMP — Brahmi (U+11000–U+1107F), Grantha, Modi, Sharada, Kaithi, Tirhuta, Nandinagari, Dogra.
• Shared punctuation sits once, in the Devanagari block, and is used by all: danda U+0964 (।), double danda U+0965 (॥); the rupee sign ₹ is U+20B9 (added 2010, design by Udaya Kumar).
• Functional characters you must know: virama/halant (Devanagari U+094D, Tamil *pulli* U+0BCD, Malayalam *chandrakkala* U+0D4D) kills the inherent *a* and requests a conjunct; nukta U+093C; anusvara U+0902, chandrabindu U+0901, visarga U+0903, avagraha U+093D; ZWNJ U+200C forces the explicit-halant form, ZWJ U+200D forces a half-form/alternate conjunct.
• Script-specific points: Tamil has no aspirated consonants and few conjuncts, and borrows Grantha letters ஜ ஷ ஸ ஹ; Malayalam encodes atomic chillu letters ൺ ൻ ർ ൽ ൾ ൿ (U+0D7A–U+0D7F) alongside the consonant+chandrakkala spelling, which is why old-lipi and reformed-lipi texts can differ in code points for the same word.

Storage: encoding forms, logical order and grapheme clusters

• A code point is an abstract number; an encoding form turns it into bytes. UTF-8 uses 1–4 bytes (every Brahmi-derived Indic BMP character takes 3 bytes, but Urdu's Perso-Arabic letters at U+0600–U+06FF take 2); UTF-16 uses 2 bytes in the BMP and a surrogate pair (4 bytes) above it; UTF-32 is a fixed 4 bytes.
• UTF-8 is ASCII-compatible, endian-free and the web default; UTF-16 is the internal form of Java, C#, JavaScript and Windows APIs; UTF-32 wastes space but indexes directly.
• Text is stored in logical (phonetic) order, not visual order. कि is stored क (U+0915) + ि (U+093F) even though ि *prints to the left* of क. Reph (र् as the hook in कर्म) is likewise stored where it is pronounced and drawn over a later consonant.
• Three different "lengths" of the same string must never be confused:
  1. bytes (storage, buffers, network),
  2. code points (Python len, database character counts),
  3. extended grapheme clusters (what a user calls one letter — cursor movement, truncation, character limits).
• Unicode 15.1 (2023) added Indic conjunct-break rules so that consonant + virama + consonant stays in one grapheme cluster; older implementations split it, which is why cursor behaviour differs across old and new software.
• Normalization (UAX #15): NFD decomposes, NFC composes. Two-part vowel signs decompose canonically (Bengali ো U+09CB = U+09C7 + U+09BE; Tamil ொ U+0BCA = U+0BC6 + U+0BBE). Precomposed nukta forms such as क़ U+0958 and ড় U+09DC have canonical decompositions but are composition exclusions, so NFC leaves them as base + nukta — the single-code-point spelling must be normalised before any comparison.
• Collation (UTS #10, Unicode Collation Algorithm, with per-language tailorings in CLDR): code-point order is not dictionary order. Matras, anusvara, visarga and conjuncts need locale rules, and Hindi, Marathi, Tamil and Malayalam dictionary conventions differ.

Rendering: fonts, shaping engines and legacy encodings

• Rendering an Indic string is shaping: the engine reorders characters, substitutes conjunct glyphs and positions marks. Steps, in outline:
  1. split the run into syllable clusters (consonants + virama + vowel signs + marks);
  2. reorder pre-base matras and reph;
  3. apply OpenType substitution features — akhn (akhand ligatures like क्ष, ज्ञ), rphf (reph), half, blwf (below-base, e.g. the Devanagari rakaar in क्र, Kannada/Telugu subjoined forms), pstf, vatu, cjct, pres/abvs/blws/psts, haln;
  4. position marks with GPOS; 5. draw.
• Engines: HarfBuzz (Linux, Chrome, Android, LibreOffice) with the Indic and Universal Shaping Engine paths, DirectWrite/Uniscribe on Windows, CoreText on macOS/iOS. A font without the right GSUB tables shows broken halants even though the data is correct.
• Common Indic fonts: Noto Sans/Serif Devanagari, Tamil, Telugu etc.; Nirmala UI / Mangal / Latha / Vijaya on Windows; Lohit and Rachana/Meera (Malayalam) from free-software projects; C-DAC GIST fonts; Jameel Noori Nastaleeq for Urdu.
• Urdu adds right-to-left layout under the Unicode Bidirectional Algorithm (UAX #9), with LRM/RLM marks and directional isolates, plus Nastaliq calligraphic cascading that demands very heavy font logic.
• Legacy font encodings — Krutidev, DevLys, Chanakya, Shree-Lipi, ML-TTKarthika, APS, Kiran — are *not* encodings of Indian characters at all: the bytes are Latin letters and the Indic appearance comes from the font. Consequences: text is unsearchable, unsortable, unspellcheckable, breaks when the font is missing, and cannot be mixed with other scripts. Converters (Krutidev↔Unicode, ML-TTKarthika↔Unicode) exist and are standard migration work in Indian government offices, courts and newspapers.

Input: keyboards, transliteration, handwriting and speech

• InScript (Indian Script Code keyboard, MeitY/TDIL; the Enhanced InScript layout is the current notified version) is phonetic and uniform across all Brahmi-derived scripts — the same key gives क / ক / க / క — so one layout serves eleven scripts. Vowels on the left half, consonants on the right, matras on the keys corresponding to their vowels.
• Typewriter/Remington layouts (Remington GAIL, Remington CBI for Hindi) survive because trained typists and many state government typing tests still use them; they are glyph-ordered, not phonetic.
• Transliteration (phonetic) input: type namaste → नमस्ते. Google Input Tools, Microsoft Indic Language Input Tool (ILIT), Lipikaar, Quillpad, Varamozhi/Mozhi for Malayalam. Easiest for new users; ambiguous for retroflex/aspirate distinctions, so it needs a dictionary-backed candidate list.
• Scholarly/ASCII transliteration schemes: ISO 15919 (the international standard romanization, with diacritics), IAST, and ASCII-only schemes ITRANS, Harvard-Kyoto, Velthuis, SLP1, and WX notation (common in Indian NLP corpora). Round-trip losslessness differs: ISO 15919 and SLP1 are reversible, loose ITRANS is not.
• IME frameworks: Windows TSF/IME, ibus-m17n / SCIM with m17n tables on Linux, Android input-method services (Gboard Indic, Indic Keyboard by Swathanthra Malayalam Computing), browser-level virtual keyboards.
• Other modalities: handwriting recognition on touch devices, OCR for Indic print (Tesseract Indic models, IIIT/C-DAC and Bhashini OCR), and speech input — ASR and TTS for 22 scheduled languages under Digital India Bhashini / National Language Translation Mission, with ULCA as the data and model registry, plus AI4Bharat (IIT Madras) resources such as IndicTrans2 and IndicNLP corpora.

Locales, databases, the web and domain names

• A locale (hi-IN, ta-IN, bn-IN, ur-IN) fixes collation, date and number formats. Indian grouping is lakh–crore: CLDR pattern #,##,##0 gives 12,34,56,789 where en-US gives 123,456,789. Currency is ₹ with that grouping; the Saka national calendar is in CLDR as indian.
• Libraries: ICU4C/ICU4J (collation, normalization, transliteration, break iteration) is the reference implementation; Python 3 str is code-point based with unicodedata; Java String is UTF-16 and needs codePointAt; indic_nlp_library and indic-transliteration are common in Indian NLP work.
• Databases: MySQL's old utf8 is a 3-byte subset that silently rejects anything above the BMP — use **utf8mb4** with a utf8mb4_0900_ai_ci-style or language-specific collation; Oracle uses AL32UTF8; PostgreSQL UTF8; SQL Server needs NVARCHAR or a _UTF8 collation. Index key-length limits are measured in bytes, so an Indic column exhausts them three times faster.
• Web: <meta charset="utf-8">, serve Content-Type: text/html; charset=utf-8, set lang="hi"/dir="rtl" for Urdu, and subset webfonts per script to control page weight.
• Domains and email: IDN under IDNA2008 encodes Indic labels to ASCII via Punycode (xn--). NIXI operates .भारत and sibling .in IDNs across 22 languages/15 scripts; universal acceptance (applications accepting Indic email addresses) is still the weak link. Mixed-script labels invite homograph spoofing, so registries apply script-mixing rules.

Key points

• Unicode - one code point per character across all scripts; replaced ISCII and legacy font encodings as the national standard for Indian language data.
• ISCII (IS 13194:1991) - 8-bit BIS standard, one shared chart for nine Brahmi scripts plus an ATR script-switch byte.
• ISCII-parallel blocks - Indic Unicode blocks keep the same relative order, so a fixed offset maps क→క→க for letters that exist in both.
• Character vs glyph - क्ष has no code point; it is क + U+094D + ष, drawn as a ligature by the font.
• Virama/halant - U+094D (Tamil pulli U+0BCD, Malayalam chandrakkala U+0D4D) removes the inherent vowel and requests a conjunct.
• ZWNJ (U+200C) / ZWJ (U+200D) - force the explicit-halant form or the half/alternate conjunct form respectively.
• Logical order - storage follows pronunciation; the pre-base matra ि and reph are *displayed* out of storage order.
• Encoding forms - UTF-8 (3 bytes per Brahmi-derived Indic BMP character, 2 for Urdu's Perso-Arabic letters), UTF-16 (2 bytes, 4 for supplementary via surrogates), UTF-32 (4 fixed).
• Three lengths - bytes ≠ code points ≠ grapheme clusters; user-visible letters are grapheme clusters (UAX #29).
• Normalization - NFC/NFD; multi-part matras recompose, but U+0958–U+095F and ড়-type nukta forms are composition exclusions.
• Collation - UCA + CLDR tailoring; code-point sort is not Hindi or Tamil dictionary sort.
• Shaping - HarfBuzz/DirectWrite apply OpenType akhn, rphf, half, blwf, pstf, pres, haln to form syllables.
• InScript - phonetic keyboard uniform across Brahmi-derived scripts; Remington layouts persist for trained typists and state typing tests.
• Transliteration input - namaste → नमस्ते; schemes ISO 15919 (standard), IAST, ITRANS, Harvard-Kyoto, SLP1, WX.
• Legacy font encodings - Krutidev/ML-TTKarthika store Latin bytes; unsearchable and unsortable, and must be converted.
• Bidi (UAX #9) - Urdu's right-to-left layout and Nastaliq cascading need the bidirectional algorithm and heavy font logic.
• Indian locale - lakh/crore grouping #,##,##0, ₹ U+20B9, Saka calendar, danda U+0964 as full stop.
• Bhashini / TDIL / AI4Bharat - the current Indian public stack for ASR, TTS, MT and OCR across the 22 scheduled languages.

Formulas and examples

Example 1 — the three lengths of नमस्ते. Code points: न U+0928, म U+092E, स U+0938, ् U+094D, त U+0924, े U+0947 → 6 code points.

• UTF-8: all in U+0800–U+FFFF → 6 × 3 = 18 bytes.
• UTF-16: all BMP → 6 × 2 = 12 bytes (len in JavaScript = 6).
• UTF-32: 6 × 4 = 24 bytes.
• Grapheme clusters: न | म | स्ते → 3 under Unicode 15.1 conjunct rules; older libraries report 4 (न | म | स् | ते).

So VARCHAR(5) would reject this 3-letter word, and truncating at 10 bytes would cut a character in half.

Example 2 — offset transliteration and where it fails.

• क U+0915 → Telugu: 0x0915 + (0x0C00 − 0x0900) = 0x0915 + 0x300 = U+0C15 క ✔
• क U+0915 → Tamil: 0x0915 + (0x0B80 − 0x0900) = 0x0915 + 0x280 = U+0B95 க ✔
• ख U+0916 → Tamil: 0x0916 + 0x280 = U+0B96, which is unassigned — Tamil has no aspirated letters.

Conclusion: the offset trick is script-conversion, not transliteration; it is lossy wherever the target inventory is smaller (Tamil aspirates, Gurmukhi vowels, Malayalam chillu), so real work uses ICU transliterators or indic-transliteration.

Example 3 — a supplementary-plane character (Brahmi GA, U+11015; Brahmi KA is U+11013).

• CP − 0x10000 = 0x11015 − 0x10000 = 0x1015 = 4,117 = 4 × 1024 + 21.
• H = 0xD800 + 4 = 0xD804; L = 0xDC00 + 21 = 0xDC15 → UTF-16 bytes D8 04 DC 15 (4 bytes).
• UTF-8: 4 bytes = F0 91 80 95.
• Python len('\U00011015') = 1; Java/JavaScript length = 2; MySQL utf8 (3-byte) rejects or mangles it — utf8mb4 is mandatory for Brahmi, Grantha, Modi and Sharada manuscript data.

Example 4 — storage sizing for a Hindi database. A digitisation project stores 5,00,000 records, each with a 40-character Devanagari title.

• Per title: 40 × 3 = 120 bytes (UTF-8) against 40 bytes in old 8-bit ISCII — a 3× increase.
• Total title data: 5,00,000 × 120 = 60,000,000 bytes = 60 MB (≈57.2 MiB), against 20 MB in ISCII.
• In UTF-16 it would be 40 × 2 = 80 bytes per title = 40 MB: for pure Indic text UTF-16 is actually *more compact* than UTF-8, which is why some Indic-only stores choose it — but UTF-8 wins the moment ASCII markup, URLs and digits are mixed in.

Example 5 — Indian number and currency formatting. Value 12345678.5 with locale hi-IN, grouping #,##,##0.00, currency ₹:

• group the last 3 digits, then 2 at a time: 678 | 45 | 23 | 1 → ₹1,23,45,678.50 = 1 crore 23 lakh 45 thousand 678 rupees 50 paise.
• the same value in en-US prints $12,345,678.50 — so a hardcoded thousands-grouping routine produces wrong invoices for Indian users.

Where it is used

• E-governance and public records: NIC and state portals, land records, court orders and FIRs in Unicode Hindi/Marathi/Tamil; bulk Krutidev→Unicode migration of decades of office files; GIGW compliance audits.
• Publishing and media: Indian newspapers and DTP houses (InDesign/Shree-Lipi workflows), e-books and EPUB in Indic scripts, font engineering for Devanagari and Nastaliq.
• Language technology: Bhashini/ULCA, AI4Bharat, C-DAC GIST, translation and TTS/ASR pipelines, Indic OCR for archives and manuscripts (Grantha, Modi, Sharada).
• Product localisation and QA: UPI apps, IRCTC, banking and insurance apps offering 10–13 languages; roles such as localisation engineer, i18n QA, linguist-annotator; testing matra rendering, truncation and sorting on Android and iOS.
• Databases and search: utf8mb4 migrations, Elasticsearch/Solr Indic analysers, stemming and normalization for Hindi/Tamil search, deduplication of names across spelling variants.
• Academia and lexicography: Sanskrit digital corpora (GRETIL, DCS, SLP1/ITRANS encodings), Tamil Virtual Academy and Project Madurai texts, dictionary and concordance projects, computational-linguistics dissertations.

Interview questions

• Q: Why does one Devanagari syllable often take more than one code point? A: Unicode encodes characters, not glyphs, so क्षि is क + virama + ष + ि; the conjunct glyph is produced by the font's OpenType rules at display time.
• Q: What is the difference between logical order and visual order? A: Text is stored in pronunciation order (क then ि), but the pre-base matra and reph are drawn in a different position, so storage order and screen order differ.
• Q: How many bytes does a Tamil letter take in UTF-8, and why? A: Three, because Tamil lies in U+0B80–U+0BFF, inside the U+0800–U+FFFF range that UTF-8 encodes in three bytes.
• Q: What goes wrong with MySQL's utf8 for Indian text? A: It is a 3-byte subset, which handles BMP Indic scripts but silently fails on supplementary-plane scripts like Brahmi or Grantha and on emoji; utf8mb4 is the correct charset.
• Q: What is the difference between ZWJ and ZWNJ in Indic text? A: ZWNJ (U+200C) blocks the conjunct and forces the visible halant form; ZWJ (U+200D) requests the half-form or an alternate conjunct instead of the default ligature.
• Q: Why is InScript the same layout for Hindi and Tamil? A: Because the Indic Unicode blocks share an ISCII-derived phonetic order, the same key position maps to the corresponding sound in each script, so a typist learns once.
• Q: A client's 2003 Hindi documents show garbage. Diagnose. A: They are almost certainly in a legacy font encoding such as Krutidev — Latin bytes drawn with an Indic font — and need a mapping-table conversion to Unicode, not a charset change.
• Q: Why can len() of a Hindi word mislead a character-limit check? A: It counts code points, not user-perceived letters; matras, viramas and nuktas inflate the count, so limits and truncation must use extended grapheme clusters.
• Q: What is the role of normalization in an Indic search box? A: Forms such as क़ exist both precomposed and as क + nukta, and multi-part matras decompose, so inputs and index entries must be normalised (usually NFC) before comparison.
• Q: Why does sorting Hindi by code point give the wrong order? A: Code-point order ignores language rules for matras, anusvara and conjuncts; dictionary order needs the Unicode Collation Algorithm with CLDR tailoring for the locale.
• Q: What extra problem does Urdu add over Devanagari? A: Right-to-left layout governed by the bidirectional algorithm, plus Nastaliq's cascading calligraphy, which needs very extensive font substitution and positioning.
• Q: Is ISO 15919 the same as ITRANS? A: No. ISO 15919 is the international romanization standard using diacritics; ITRANS is an ASCII-only scheme, and loose ITRANS is not reliably reversible.

Common mistakes

• Treating one Indic character as one byte, or sizing VARCHAR/buffers in bytes while thinking in letters — the usual cause of truncated matras and "?" boxes.
• Reversing or truncating a string by code point, which detaches a matra or virama from its consonant and produces nonsense.
• Calling Krutidev or ML-TTKarthika "a Unicode font" — they are glyph encodings, and copying such text into a Unicode system yields Latin gibberish.
• Blaming the data when a conjunct renders wrongly; the fault is usually a font without the needed GSUB tables or an old shaping engine.
• Using the ISCII block-offset trick as general transliteration and losing letters that the target script does not have (Tamil aspirates, Malayalam chillu).
• Comparing or de-duplicating strings without normalising, so the precomposed and decomposed spellings of the same word are counted as different.
• Hardcoding three-digit thousands grouping, or dir="ltr" for Urdu, and then reporting the formatting bug as a data problem.

Related topics

Languages of India: language families, scripts and the Eighth Schedule · Linguistics basics · Classical languages of India · Grammar traditions: Panini, Tolkappiyam and regional grammars · Natural language processing for Indian languages · Translation and interpretation studies