Information retrieval and indexing

Library and Information Science (B.LISc) · Knowledge organisation · free study notes for exams, placements and interviews

In short

Information retrieval (IR) is the process of representing, storing, organising and searching recorded information so that a user's query brings back the documents that answer it. Indexing is the representation side of that process: deciding what a document is about and expressing it in terms a searcher can match. In B.LISc courses these two are examined together as the "retrieval" half of knowledge organisation, alongside classification and cataloguing, and they underpin OPACs, institutional repositories, library discovery layers and the search engines that every reader now uses. Indian recruiters who test this topic include university and IIT/IIM/IISc libraries, DRDO, ISRO, CSIR and ICAR laboratory information centres, CSIR-NIScPR (formerly NISCAIR) and similar documentation units, DELNET, INFLIBNET (Shodhganga, Shodh Shuddhi, e-ShodhSindhu), Parliament and High Court libraries, and the many publishers, legal databases and e-commerce firms that hire LIS graduates as taxonomists and metadata analysts. It is also the core of the UGC-NET Library and Information Science Paper II unit on information processing and retrieval.

Explained

What information retrieval is, and the classical model

Definition: IR is the identification and delivery of documents or data relevant to a stated information need from a collection, as distinct from data retrieval, which returns exact matching records.
Data retrieval vs information retrieval: data retrieval (SQL on a payroll table) is deterministic, exact-match and complete; IR is probabilistic, partial-match, ranked and tolerant of error, because "aboutness" is a matter of degree.
The two-sided model: the *document* is represented by an index entry; the *need* is represented by a query; the *matching function* compares the two. Retrieval fails when either representation is poor, not only when matching is poor.
Information need levels (Taylor, examined in most syllabi): Q1 visceral (unexpressed), Q2 conscious (vague), Q3 formalised (stated), Q4 compromised (stated in the system's language). Reference interview and search negotiation move the user from Q1 to Q4.
Components of an IR system: document collection → document analysis and indexing → index file (usually an inverted file) → query formulation → matching/ranking → output → relevance feedback.
Lancaster's view of an index as a device that locates subjects, and Ranganathan's "fifth law - the library is a growing organism" is why index languages need vocabulary control and updating.

Indexing: definition, process and types

Indexing is the analysis of a document's intellectual content and the expression of that content in a set of index terms (descriptors, subject headings or class numbers).
Steps in subject indexing (the sequence asked in exams): (1) familiarisation with the document - title, abstract, contents, introduction, conclusion; (2) subject analysis - identify the core subject and its facets; (3) translation of concepts into the index language; (4) construction of index entries and their sequence; (5) cross-references and syndetic structure; (6) verification/revision.
Derived (derivative) indexing: terms are *taken from* the document's own text - title, abstract, full text. Cheap, fast, machine-friendly; suffers from uninformative titles and author idiosyncrasy. KWIC, KWOC and full-text search are derived.
Assigned indexing: terms are *allotted* from a controlled list or constructed by rules regardless of the author's wording. Costly, consistent, allows synonym and hierarchy control. Subject headings, thesaurus descriptors, PRECIS and POPSI are assigned.
Depth of indexing: *exhaustivity* = how many of a document's themes are indexed (raises recall, lowers precision); *specificity* = how precisely a term matches the concept (raises precision, lowers recall). Both are policy decisions, not properties of the document.
Indexing consistency: inter-indexer consistency in real services is commonly only about 30-60% term overlap; it falls as exhaustivity rises. Quote it as a reported range, not an exact law.
Human vs automatic indexing: humans read for aboutness and handle irony, images and novelty; machines are consistent, exhaustive, instant and cheap. Modern practice is hybrid - machine candidate terms, human validation.

Pre-coordinate indexing systems

Pre-coordinate: the indexer combines (coordinates) the component concepts into one multi-word heading *before* the entry is filed; the user scans a fixed citation order. Suits printed indexes and shelf arrangement.
Cutter's rules and Kaiser's "Systematic Indexing" (concrete + process) are the historical starting points; Kaiser's concrete-process pairing is a direct ancestor of faceted citation order.
Chain indexing (Ranganathan, 1934): derives alphabetical subject entries from the *chain* of a class number by taking successive links from the most specific upward, reversing the order. Used for decades in the *Indian National Bibliography* and in BNB until 1970.
Chain indexing terms: *sought links* (likely to be searched - get entries), *unsought links* (omitted), *false links* (connecting symbols with no term), *missing links* (a concept with no class number of its own).
PRECIS (Preserved Context Index System, Derek Austin, 1971): uses role operators (0 location, 1 key system/whole, 2 action, 3 agent, p part, q member of quasi-generic group, s role definer) and shunting to produce several entries that each preserve full context, with lead, qualifier and display positions. Developed for BNB; also notable for its machine production and the RIN (Reference Indicator Number) thesaurus.
POPSI (Postulate-based Permuted Subject Indexing, G. Bhattacharyya, DRTC, 1979): built on Ranganathan's general theory; elementary categories are Discipline, Entity, Property, Action (plus modifiers), arranged in that sequence and then permuted to give the needed approach terms. An Indian system, strongly examined in Indian universities.
Relational indexing (Farradane): uses nine relational operators from two psychological criteria (discrimination and awareness) to express concept relations precisely.
Other examples: subject headings strings in *Sears List of Subject Headings* and *Library of Congress Subject Headings* (with subdivisions - topical, geographic, chronological, form); PRESERVED context in classified catalogues.

Post-coordinate indexing and search logic

Post-coordinate: single concepts are indexed separately and the *user* combines them at search time with Boolean operators. Flexible, order-free, the basis of all databases.
Uniterm (Mortimer Taube, 1953): one card per single word; document accession numbers written in terminal-digit columns; coordination by comparing numbers across cards. Suffers *false drops* and no role/link control.
Devices to cut false drops: links (tie terms belonging to the same theme in a multi-theme document) and roles (show the function a term plays, e.g. product vs raw material). Example false drop: "effect of fertiliser on rice" vs "effect of rice husk on fertiliser" index identically without roles.
Boolean operators: AND narrows (intersection), OR broadens (union), NOT excludes (difference); precedence NOT → AND → OR unless parentheses are used. Also proximity (NEAR/n, ADJ), truncation (libra* → library, librarian), wildcards, field/limit searching (au=, ti=, so=, year), phrase searching in quotes.
Search strategies: *briefsearch*, *building block* (concept blocks ANDed), *citation pearl growing* (start with one good record, harvest its terms), *successive fractions*, *most specific facet first*.
Index file structures: inverted file (term → posting list of document IDs, with positions and frequencies) is the standard; also signature files and suffix structures. The inverted index is why a web search is fast.

Vocabulary control tools and standards

Why control: natural language suffers synonymy (*cinema/film/movies*), homonymy and polysemy (*bank*), varying word form (singular/plural, Indian/British/American spelling) and varying specificity. Control maps all of these to one preferred term.
Subject heading lists: *Sears List of Subject Headings* (small and school libraries), *LCSH* (large and academic libraries, now linked data via id.loc.gov), *MeSH* (NLM, medicine, with tree numbers and automatic explosion), *Universal Decimal Classification* used as an index language in science documentation.
Thesaurus: a controlled, structured vocabulary of descriptors with standard relationships - USE/UF (equivalence: preferred/non-preferred), BT/NT (hierarchy: broader/narrower, generic, whole-part, instance), RT (associative), SN (scope note), TT (top term). International examples (none of these is Indian): *Thesaurofacet*, *ERIC Thesaurus*, *AGROVOC* (FAO, used by ICAR libraries), *INIS Thesaurus*, *Root Thesaurus*.
Standards: thesaurus construction follows ISO 25964-1:2011 (and 25964-2:2013 for interoperability), which superseded ISO 2788 and 5964; the US counterpart is ANSI/NISO Z39.19. IS 7262 historically covered Indian guidelines for thesaurus construction - check whether the Indian Standard is current before quoting it, as many older IS numbers have been withdrawn or revised.
Other relevant standards: ISO 999 for index content and presentation; ISO 214 for abstracts; ISO 2709 / ANSI-NISO Z39.2 record structure that MARC uses; Z39.50 / SRU for federated search; OAI-PMH for metadata harvesting (the backbone of Shodhganga harvesting); Dublin Core (ISO 15836) for 15 core metadata elements; ISO 9001:2026 when a documentation centre's processes are certified.
Taxonomies, ontologies, folksonomies: taxonomy = hierarchy only; ontology adds typed relations and axioms (OWL/RDF, SKOS for publishing thesauri as linked data); folksonomy = uncontrolled user tags - high recall, poor precision, no hierarchy.
Indian practice: OPAC and discovery in Indian libraries run on Koha and DSpace (both open source, promoted by INFLIBNET and NIC), SOUL 2.0 (INFLIBNET's own library management software), and commercial discovery layers; retrieval there uses Lucene/Solr or Elasticsearch inverted indexes with stemming and stop-word lists.

Automatic indexing, ranking models and evaluation

Automatic indexing pipeline: tokenisation → case folding → stop-word removal → stemming (Porter algorithm) or lemmatisation → term weighting → index construction. Indian-language retrieval adds script normalisation and morphological analysis (agglutination in Malayalam/Telugu makes stemming hard).
Zipf's law explains why a few words dominate text and why mid-frequency words are the best index terms (Luhn's resolving power).
Retrieval models: (1) Boolean - exact set logic, no ranking; (2) Vector space model (Salton) - documents and query are vectors of weighted terms, ranked by cosine similarity, weights usually TF-IDF; (3) Probabilistic - rank by probability of relevance (BM25 is the practical form); (4) Language models; (5) Fuzzy, extended Boolean and p-norm; (6) Semantic/neural - LSI, word embeddings, dense vector search now used in discovery services.
Relevance feedback (Rocchio): re-weight the query towards terms of documents the user marks relevant and away from non-relevant ones; *pseudo-relevance feedback* assumes the top n are relevant.
Evaluation measures: precision, recall, fallout, F-measure, E-measure, MAP, P@10, nDCG, and the precision-recall inverse relationship.
Test collections and projects: Cranfield I and II (1957-66) established the laboratory evaluation paradigm and showed single-term index languages performed as well as elaborate ones; MEDLARS, SMART, STAIRS; TREC (NIST, since 1992); FIRE (Forum for Information Retrieval Evaluation) is the South Asian equivalent, dealing with Indian-language and cross-lingual retrieval.
Limitations of evaluation: recall needs knowledge of the whole collection's relevant set, so it is estimated by pooling; relevance is user-, time- and task-dependent.

Key points

Information retrieval - matching representations of documents against representations of needs; partial, ranked, probabilistic.
Indexing - deciding aboutness and expressing it in an index language so a searcher can match it.
Derived vs assigned indexing - terms taken from the text versus allotted from a controlled list.
Exhaustivity - number of themes indexed; raises recall, lowers precision.
Specificity - closeness of term to concept; raises precision, lowers recall.
Pre-coordinate - indexer combines concepts before filing (chain, PRECIS, POPSI, subject headings).
Post-coordinate - user combines single concepts at search time (Uniterm, Boolean databases).
Chain indexing - Ranganathan, 1934; reverse rendering of the class-number chain; sought, unsought, false and missing links.
PRECIS - Austin, 1971; role operators plus shunting give context-preserving entries; used for BNB.
POPSI - Bhattacharyya, DRTC, 1979; Discipline-Entity-Property-Action, then permuted.
Links and roles - devices that prevent false drops in post-coordinate systems.
Thesaurus relations - USE/UF, BT/NT, RT, SN, TT; built to ISO 25964-1:2011.
Inverted file - term → posting list of document IDs; the standard index structure behind OPACs and web search.
TF-IDF and cosine similarity - the vector space model's weighting and ranking; BM25 is the probabilistic workhorse.
Precision and recall - accuracy versus coverage of the retrieved set; they move in opposite directions.
Cranfield projects - origin of controlled IR evaluation; TREC and India's FIRE continue it.
Relevance feedback - query modification using the user's judgements (Rocchio).
Indian systems in use - Koha, DSpace, SOUL 2.0, e-ShodhSindhu, Shodhganga (OAI-PMH harvested), AGROVOC in ICAR libraries.

Formulas and examples

Precision P = a / (a + b) = relevant retrieved / total retrieved (dimensionless, usually %)
Recall R = a / (a + c) = relevant retrieved / total relevant in collection
Fallout = b / (b + d) = non-relevant retrieved / total non-relevant
F-measure (F1) = 2PR / (P + R); general Fβ = (1 + β²)PR / (β²P + R)
where a = relevant retrieved, b = non-relevant retrieved, c = relevant not retrieved, d = non-relevant not retrieved
Term frequency tf(t,d) = count of term t in document d (often 1 + log tf)
Inverse document frequency idf(t) = log₁₀ (N / df(t)); N = documents in collection, df = documents containing t
Weight w(t,d) = tf × idf
Cosine similarity sim(q,d) = (Σ wq,i · wd,i) / (√Σ wq,i² · √Σ wd,i²); value 0 to 1
Chain index entries from a chain of n links = number of sought links (never more than n)

Example 1 - precision, recall and F1. A Koha OPAC search for "water harvesting" in a collection of 10,000 records retrieves 80 records. The searcher judges 60 of them relevant. A later exhaustive check shows the collection holds 100 relevant records.

a = 60, b = 80 − 60 = 20, c = 100 − 60 = 40, d = 10,000 − 100 − 20 = 9,880.
P = 60/80 = 0.75 (75%); R = 60/100 = 0.60 (60%).
F1 = 2 × 0.75 × 0.60 / (0.75 + 0.60) = 0.90/1.35 = 0.667 (66.7%).
Fallout = 20/9,900 = 0.00202 (0.2%) - small, which is why fallout is a weak discriminator in large collections.

Example 2 - the precision-recall trade-off. Broadening the same search with OR and truncation (water* OR rainwater OR watershed) now retrieves 200 records, of which 90 are relevant.

P = 90/200 = 0.45 (45%); R = 90/100 = 0.90 (90%); F1 = 2 × 0.45 × 0.90/1.35 = 0.81/1.35 = 0.60.
Recall rose by 30 percentage points, precision fell by 30; F1 fell from 0.667 to 0.60, so the broader search is worse on a balanced measure but right if the user needs completeness (a systematic review or a patent search).

Example 3 - TF-IDF weights. Collection N = 1,000 documents. Document D mentions *groundwater* 20 times and *hydrology* 5 times. *groundwater* appears in 100 documents, *hydrology* in 10.

idf(groundwater) = log₁₀(1000/100) = log₁₀ 10 = 1; w = 20 × 1 = 20.
idf(hydrology) = log₁₀(1000/10) = log₁₀ 100 = 2; w = 5 × 2 = 10.
A term appearing in all 1,000 documents gives idf = log₁₀ 1 = 0, weight 0 - which is the mathematical reason stop words carry no discriminating power.

Example 4 - chain index entries. Class number for "Cataloguing of maps in university libraries", chain: Library science → Library operations → Cataloguing → Maps → University libraries. Sought links: Cataloguing, Maps, University libraries; "Library operations" judged unsought.

Entries (specific-to-general reversal): *University libraries, Maps, Cataloguing: Library science*; *Maps, Cataloguing, Library science*; *Cataloguing, Library science*.
3 sought links → 3 entries. Had all 5 links been sought, 5 entries would result - showing how chain indexing controls index bulk by the sought/unsought judgement.

Example 5 - Boolean set sizes and a false drop. Postings: *solar* = 1,200 documents, *pump* = 800, both terms = 150.

solar AND pump → 150; solar OR pump → 1,200 + 800 − 150 = 1,850; solar NOT pump → 1,200 − 150 = 1,050.
Of the 150, say 20 are about *pumping water to cool solar panels*, not solar-powered pumps - false drops caused by post-coordination; a role indicator (solar as power source) or proximity (solar ADJ pump) removes most of them.

Feature	Pre-coordinate	Post-coordinate
Who combines concepts	Indexer, before filing	User, at search time
Citation order	Fixed, must be learnt	None needed
Medium	Printed indexes, card catalogues	Databases, OPACs, web
Flexibility	Low	High
False drops	Rare (context preserved)	Common without links/roles
Examples	Chain, PRECIS, POPSI, LCSH strings	Uniterm, Boolean databases, Solr/Lucene

Model	Representation	Output	Strength	Weakness
Boolean	Sets of terms	Unranked set	Precise, transparent, reproducible	No ranking, no partial match, needs skill
Vector space	Weighted term vectors	Ranked by cosine	Partial match, ranking, feedback easy	Assumes term independence
Probabilistic (BM25)	Relevance probability	Ranked list	Theoretically grounded, strong in practice	Needs parameter tuning/training data
Semantic / neural	Dense embeddings	Ranked list	Handles synonyms and paraphrase	Opaque, compute-heavy, needs large corpora

Where it is used

Academic and research libraries: OPAC and discovery search in Koha/SOUL 2.0, subject indexing of theses in DSpace repositories under Shodhganga, and database searching on e-ShodhSindhu for faculty literature reviews.
Science and government documentation centres: DRDO, ISRO, CSIR, ICAR, BARC and DAE libraries run controlled-vocabulary indexing with INIS/AGROVOC-type thesauri; Parliament and legislative assembly libraries index debates and bills.
Publishing, legal and media firms: back-of-book and database indexing, taxonomy and metadata work for SCC Online, Manupatra, Indian Kanoon-type legal databases, and newspaper archives.
Corporate knowledge management and e-commerce: enterprise search, SharePoint/Confluence taxonomies, product-catalogue taxonomy and search-relevance roles at Indian IT and retail firms - a common exit route for LIS graduates into "taxonomist", "metadata analyst", "search quality analyst" jobs.
Health and agriculture information services: MeSH-based searching of PubMed in medical college libraries; AGRIS/AGROVOC indexing in agricultural universities; hospital record indexing with ICD codes.
Competitive examinations and teaching: UGC-NET/SET LIS Paper II, state public service librarian posts, and the DRTC/documentation-research training line where POPSI and chain indexing originated.

Interview questions

Q: How does information retrieval differ from data retrieval? A: Data retrieval gives exact, complete answers to precisely stated queries on structured data; information retrieval gives a ranked, partial-match set of documents judged likely relevant to an imprecise need.
Q: What is the difference between derived and assigned indexing? A: Derived indexing takes terms from the document's own text (titles, abstracts, full text); assigned indexing allots terms from a controlled vocabulary or constructs them by rules, independent of the author's words.
Q: Define exhaustivity and specificity and state their effect. A: Exhaustivity is how many themes of a document are indexed and mainly raises recall; specificity is how closely a term matches the concept and mainly raises precision.
Q: Why do precision and recall move in opposite directions? A: Broadening a search to capture more relevant items inevitably admits more non-relevant ones, so coverage is bought at the cost of accuracy, and vice versa.
Q: What are false drops and how are they prevented? A: Irrelevant hits that satisfy the query terms but not the intended relationship; links tie terms of the same theme, roles state a term's function, and proximity or phrase searching restores the context.
Q: What are the standard thesaurus relationships? A: Equivalence (USE/UF), hierarchical (BT/NT) and associative (RT), supported by scope notes and top terms, as specified in ISO 25964-1.
Q: Explain chain indexing in one sentence. A: Ranganathan's method of deriving alphabetical subject entries by taking the links of a class number from the most specific upward and rendering each sought link in reverse order.
Q: How does PRECIS differ from chain indexing? A: PRECIS uses role operators and shunting to generate several entries that each preserve the full context and does not depend on a classification number, whereas chain indexing is derived entirely from the chain of a class number.
Q: What is POPSI and why is it significant in India? A: Postulate-based Permuted Subject Indexing, developed by G. Bhattacharyya at DRTC on Ranganathan's general theory, using Discipline-Entity-Property-Action categories - an indigenous Indian contribution still examined and taught here.
Q: What does TF-IDF achieve? A: It weights a term high when it is frequent in a document but rare across the collection, so discriminating terms rank documents and ubiquitous words contribute nothing.
Q: Why is the inverted file the standard index structure? A: It stores for each term the list of documents (and positions) containing it, so a query is answered by merging short posting lists instead of scanning the whole collection.
Q: What did the Cranfield projects establish? A: The laboratory evaluation paradigm - test collection, queries and relevance judgements measured by precision and recall - and the surprising finding that simple single-term index languages performed as well as elaborate ones.
Q: Is a folksonomy a substitute for a thesaurus? A: No; user tags add vocabulary and currency cheaply but lack synonym control and hierarchy, so they help recall and discovery while a thesaurus remains necessary for precision and browsing.

Common mistakes

Swapping the precision and recall formulas - recall's denominator is all relevant documents in the collection, not all retrieved.
Calling chain indexing post-coordinate, or PRECIS a classification scheme; both are pre-coordinate indexing systems.
Confusing exhaustivity with specificity, or claiming both raise retrieval effectiveness together.
Treating a thesaurus as a mere word list and omitting the USE/UF, BT/NT, RT structure that makes it a controlled vocabulary.
Writing Boolean queries without parentheses and assuming left-to-right evaluation, so A OR B AND C is read wrongly.
Quoting withdrawn standards - ISO 2788 and ISO 5964 were replaced by ISO 25964, and several older Indian Standards have been withdrawn or revised.
Claiming a search engine "retrieves all relevant documents" - recall can never be known exactly in a large or open collection, only estimated by pooling.