# Project export: IndiaMacro: AI-Native Data for India’s Economy This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: OpenAI Build Week - Tagline: An open-source platform that transforms official Indian economic data into AI-ready, reproducible datasets for developers, researchers, students, and intelligent applications. - Devpost: https://devpost.com/software/indiamacro-ai-native-data-for-india-s-economy - GitHub: https://github.com/fotogfreaksandeep-arch/IndiaMacro - Video: https://www.youtube.com/embed/L7gc_38pLtk?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 1 GitHub contributor(s) — fotogfreaksandeep-arch (9 commits) ## Devpost submission (written by the team) ### Inspiration I have been a quant researcher for the last 8 years and whenever I need macroeconomic data, my go-to has always been the FRED API. I wanted something similar for India as well. There are a lot of public data sources available in India. For example, the Reserve Bank of India, or RBI, publishes a large amount of statistical data on its website. But there is no common Python library that I can reference for extracting and working with this data in a clean, research-friendly way. That gap was the inspiration for IndiaMacro: making Indian macroeconomic data easier to access, while respecting the reality that economic data is revised, published over time, and not always presented in a clean machine-readable format. ### What it does IndiaMacro is a Python library for reproducible, programmatic access to official Indian macroeconomic data. The current v0.2.0 release supports RBI Sectoral Deployment of Bank Credit data from verified RBI Bulletin issues published between July 2025 and June 2026. It provides: Current RBI Sectoral Credit data Historical published-vintage observations Explicit latest_publication and as_of resolution Offline replay from a versioned local cache Source provenance and deterministic hashes Strict parser contracts for known RBI source-layout transitions An optional Matplotlib visualization for Non-food Credit The important point is that it does not silently overwrite older published values. That matters for research, because using a revised value that was not available at the time can introduce look-ahead bias. ## ### How we built it I built IndiaMacro with ChatGPT and, specifically, Codex and GPT-5.6 Sol using an extra-high reasoning level. I gave GPT-5.6 Sol a goal: build a parser that can: Check the source data file Download a copy Study the contents Iterate over the parser Ensure that the data is extracted properly before it completes Codex helped turn that work into a proper Python package. It helped with the parser implementation, test coverage, validation, documentation, offline replay, provenance handling, packaging, and the reproducible demo. The project treats each publication as a vintage. Instead of trying to make one superficially clean time series, it preserves what RBI published at a particular point in time and lets the user explicitly choose a current or as_of view. ### Challenges we ran into The biggest challenge is that RBI data is often published in Excel workbooks with schemas that change over time. Even if you build a parser, it is still cumbersome to join different vintages and publication dates correctly. There are changing row layouts, changing date conventions, different growth columns, methodology changes, and revised values. It is also important not to overwrite already published data. That would be detrimental to research because it can introduce look-ahead bias. The data has to be point-in-time and preserve the publication context so there is data integrity. I had initially tried using a lower-intelligence model, but it essentially surrendered and said that I needed to manually download the files and provide them. GPT-5.6 Sol was different. It was tenacious: it continued through the difficult source formats, helped derive strict parser contracts, and kept validating the outputs rather than stopping at the first obstacle. ### Accomplishments we're proud of I am proud that IndiaMacro is not just a one-off spreadsheet extraction script. It now has: A released, installable Python package Verified support for 12 consecutive RBI Bulletin issues 5,950 published-vintage observations 3,060 current observations across 255 measure-specific series Explicit handling of historical revisions Strict support boundaries instead of silently guessing unsupported layouts Offline tests and replay A reproducible Non-food Credit example and visualization Public source code and release artifacts I am especially proud that the project keeps provenance, publication dates, source hashes, and methodology boundaries alongside the data. For quantitative research, those details are as important as the number itself. ### What we learned The biggest lesson was that collecting data is not the same as building research-grade data infrastructure. A clean-looking time series can still be misleading if it overwrites history, ignores revisions, uses data that was not known at the time, or quietly joins incompatible source layouts. I also learned how useful Codex and GPT-5.6 can be when the task is not simply “write some code.” This involved investigation, iteration, validation, source analysis, testing, release preparation, and the judgment to keep parser contracts strict. ### What's next IndiaMacro is just a start. It is far from over, but I see it as a fruitful endeavour. My aim is to continue building it as an open-source repository for quantitative researchers and for anybody interested in economics, finance, or macroeconomics who wants easier access to Indian data. The longer-term vision is AI-native data infrastructure for India’s economy: trustworthy, well-documented, point-in-time data that can be accessed programmatically and used confidently in research and analysis. I plan to add carefully verified sources over time, while keeping the same principles: strict contracts, reproducibility, provenance, explicit methodology boundaries, and no silent guessing. I hope to continue using Codex on this journey. ## README (from the GitHub repository) # IndiaMacro IndiaMacro is a focused, auditable data-access library for official Indian macroeconomic data. Version 0.2.0 is available as a GitHub release and covers one dataset: RBI Sectoral Deployment of Bank Credit from RBI Bulletin Current Statistics Tables 15 and 16. Verified historical support currently covers the 12 Bulletin issues from July 2025 through June 2026. This is a tested IndiaMacro boundary, not the full availability of RBI data. ## Built during OpenAI Build Week IndiaMacro existed before Build Week as v0.1.0: a current-only RBI connector with the June 2026 parser, cache, provenance, and offline replay. During Build Week, v0.2.0 added the verified July 2025–June 2026 historical path, strict layout-transition parsers, published vintages, explicit resolution, optional plotting, portable CI, and a reproducible demonstration. Codex accelerated source investigation, parser and test implementation, compatibility analysis, caching, the historical API, packaging, and release validation. The user set the scope and methodology: infrastructure before a dashboard, one deeply verified dataset, strict contracts instead of silent adaptation, explicit vintage handling, and bounded 16 GB local execution. The primary model used for this work was GPT-5.6 Sol with extra-high reasoning. ### Judge quick start Install the exact released wheel rather than a package with the same name from another index: ```bash python -m venv .venv source .venv/bin/activate python -m pip install "https://github.com/fotogfreaksandeep-arch/IndiaMacro/releases/download/v0.2.0/indiamacro-0.2.0-py3-none-any.whl" ``` Then run: ```python from indiamacro import rbi history = rbi.sectoral_credit_history( start_issue="2025-07", end_issue="2026-06", ) points = history.select( series_id="RBI.SECTION42.NON_FOOD_CREDIT.OUTSTANDING", view="current", ) print(len(history.vintages), len(history.current_observations), len(points)) ``` The first live run requires access to public RBI pages and populates a validated cache. Subsequent runs can replay the same source material without a network session by passing `offline=True`. ## Install v0.2.0 Version 0.2.0 has not been published to PyPI. Install the exact wheel from the public GitHub release: IndiaMacro supports Python 3.11 and 3.12. Its portable CI suite runs on Ubuntu, and the release demonstration has also been validated on macOS Apple silicon. The released wheel is pure Python (`py3-none-any`) and has no platform-specific compiled extension. Create and activate an isolated environment on macOS or Linux: ```bash python -m venv .venv source .venv/bin/activate ``` On Windows PowerShell: ```powershell python -m venv .venv .\.venv\Scripts\Activate.ps1 ``` Then install the released wheel: ```bash python -m pip install "https://github.com/fotogfreaksandeep-arch/IndiaMacro/releases/download/v0.2.0/indiamacro-0.2.0-py3-none-any.whl" ``` Matplotlib remains optional: ```bash python -m pip install "indiamacro[plot] @ https://github.com/fotogfreaksandeep-arch/IndiaMacro/releases/download/v0.2.0/indiamacro-0.2.0-py3-none-any.whl" ``` Development installs from a source checkout remain available: ```bash python -m pip install . python -m pip install ".[plot]" ``` Do not use `pip install indiamacro` to judge v0.2.0: this version is distributed through the GitHub release and has not been published to PyPI. ## Test a source checkout Install the development dependencies and run the portable validation layer: ```bash python -m pip install -e ".[test,plot,build]" ruff check . pytest -m "not local_evidence and not live" -rs python -m build python -m twine check dist/* ``` The portable suite runs serially, makes no live RBI requests, and does not depend on ignored local evidence. Maintainers who have the preserved RBI acceptance evidence can additionally run: ```bash pytest -m local_evidence -rs ``` Live RBI acceptance remains explicitly opt-in and is excluded from ordinary tests and CI: ```bash pytest -m live -rs ``` See [Testing IndiaMacro](docs/testing.md) for the purpose and evidence requirements of each test layer. ## Retrieve current data ```python from indiamacro import rbi credit = rbi.sectoral_credit() print(credit.observations.head()) print(credit.metadata.latest_observation_date) print(credit.metadata.freshness_status) print(credit.metadata.semantic_observations_sha256) ``` The result exposes a long-form pandas DataFrame as `credit.observations`, typed release and provenance metadata as `credit.metadata`, and the two complete RBI methodology notes as `credit.notes`. The Bulletin can lag RBI's dedicated monthly Sectoral Deployment release. `latest_observation_date` is the latest date represented by the returned data, while `freshness_status` reports the comparison with the dedicated release index. IndiaMacro never requests the dedicated release's blocked XLSX file. ## Verified publication history The historical connector covers the 12 verified RBI Bulletin issues from July 2025 through June 2026, inclusively: ```python from indiamacro import rbi history = rbi.sectoral_credit_history( start_issue="2025-07", end_issue="2026-06", ) vintages = history.vintages current = history.current_observations non_food = history.select( series_id="RBI.SECTION42.NON_FOOD_CREDIT.OUTSTANDING", view="current", ) non_food_yoy = history.select( series_id="RBI.SECTION42.NON_FOOD_CREDIT.YOY_GROWTH_REPORTED", view="current", ) resolved = history.resolve(policy="latest_publication", as_of="2026-04-30") ``` Historical collections are immutable tuples of frozen records with `Decimal` values; selection and resolution do not return pandas objects. The complete range contains 5,950 published-vintage observations and 3,060 current observations. Each Non-food Credit selection above yields 12 chart-ready points. The January-to-February 2026 boundary changes the current-date basis from the last reporting Friday to calendar month-end, so a continuous publication sequence does not imply unchanged methodology. See the [historical API guide](docs/usage/rbi_sectoral_credit_history.md) for cache, offline replay, hash, resolution, and exception details. ![Two-panel RBI Non-food Credit chart showing outstanding and reported YoY growth with the January–February 2026 reporting-date boundary](docs/assets/rbi_non_food_credit_history.svg) Plotting is an optional downstream demonstration: ```bash python -m pip install ".[plot]" python scripts/plot_rbi_sectoral_credit_history.py --offline ``` The [end-to-end tutorial](docs/tutorials/rbi_sectoral_credit_end_to_end.md) walks through live retrieval, offline replay, selection, explicit resolution, CSV export, chart generation, and provenance inspection. IndiaMacro's core purpose remains auditable data access; the chart demonstrates what trustworthy downstream applications can build on that infrastructure. ## Refresh, offline use, and cache ```python # Prefer a verified cache; retrieve live only when no compatible bundle exists. credit = rbi.sectoral_credit() # Force exact-title live discovery and bounded retrieval. credit = rbi.sectoral_credit(refresh=True) # Guarantee no network session is created. credit = rbi.sectoral_credit(offline=True) # Replay verified historical issues from their separate history cache. history = rbi.sectoral_credit_history( "2025-07", "2026-06", offline=True, ) ``` An explicit `cache_dir` may be passed to any call. Otherwise `INDIAMACRO_CACHE_DIR` is used when set, followed by the platform-standard user cache directory. Raw HTML pages and a versioned manifest are committed only after both tables validate and parse. Every cache read recalculates raw and output hashes. Missing, incompatible, or corrupt caches raise specific errors; the API never returns an empty DataFrame as an error substitute. Pre-release manifest schema 1 bundles used the misleading field `normalized_output_sha256`. v0.1.0 classifies those bundles a [README truncated for size] ## Detected evidence (automated analysis) Indexed codebase: 44 recognized source files, 609 KB. - HTML (language) — detected in the code - Python (language) — detected in the code ## Codebase structure (from repository index) ### Files (50 of 50) ``` .github/workflows/ci.yml .gitignore CHANGELOG.md docs/contracts/rbi_sectoral_credit_2026_transition.md docs/contracts/rbi_sectoral_credit_bulletin_v1.md docs/contracts/rbi_sectoral_credit_bulletin_v2.md docs/contracts/rbi_sectoral_credit_history_v1.md docs/investigations/rbi_sectoral_credit_2026_transition.md docs/investigations/rbi_sectoral_credit_history_census.md docs/investigations/rbi_sectoral_credit_official_access.md docs/releases/v0.2.0.md docs/testing.md docs/tutorials/rbi_sectoral_credit_end_to_end.md docs/usage/rbi_sectoral_credit_history.md docs/usage/rbi_sectoral_credit.md LICENSE MANIFEST.in pyproject.toml README.md scripts/accept_rbi_sectoral_credit_history.py scripts/build_rbi_sectoral_credit_2026_transition_evidence.py scripts/build_rbi_sectoral_credit_parser_evidence.py scripts/build_rbi_sectoral_credit_v2_evidence.py scripts/census_rbi_sectoral_credit_history.py scripts/investigate_rbi_sectoral_credit_sources.py scripts/plot_rbi_sectoral_credit_history.py scripts/run_rbi_sectoral_credit_live_acceptance.py scripts/spike_rbi_sectoral_credit_access.py src/indiamacro/__init__.py src/indiamacro/rbi/__init__.py src/indiamacro/rbi/sectoral_credit_bulletin_v2.py src/indiamacro/rbi/sectoral_credit_bulletin.py src/indiamacro/rbi/sectoral_credit_history.py src/indiamacro/rbi/sectoral_credit_plot.py src/indiamacro/rbi/sectoral_credit_transition_2026.py src/indiamacro/rbi/sectoral_credit.py tests/fixtures/rbi_non_food_credit_chart_v1.json tests/fixtures/rbi_sectoral_credit_compatibility_v1.json tests/fixtures/rbi_sectoral_credit_industries_v1.html tests/fixtures/rbi_sectoral_credit_major_v1.html tests/test_investigate_rbi_sectoral_credit_sources.py tests/test_sectoral_credit_2026_transition.py tests/test_sectoral_credit_bulletin_parser.py tests/test_sectoral_credit_bulletin_v2.py tests/test_sectoral_credit_connector.py tests/test_sectoral_credit_hashes.py tests/test_sectoral_credit_history_census.py tests/test_sectoral_credit_history.py tests/test_sectoral_credit_plot.py tests/test_spike_rbi_sectoral_credit_access.py ``` ### Dependencies - pyproject.toml: build@>=1.2, matplotlib@>=3.8, pandas@>=2.0, platformdirs@>=4.0, pytest@>=8,<9, requests@>=2.31, ruff@>=0.12, twine@>=6 ### Recent commits (newest first) - docs: update v0.2.0 installation and testing - test: make CI independent of local evidence - release: prepare IndiaMacro 0.2.0 - feat: add sectoral credit visualization demo - feat: add RBI sectoral credit history API - feat: support RBI sectoral credit 2026 transition - feat: add RBI sectoral credit parser v2 - research: census RBI sectoral credit history - release: prepare IndiaMacro 0.1.0 ## Key source files (fetched from GitHub, selected and truncated for size) ### CHANGELOG.md ```markdown # Changelog All notable changes to IndiaMacro are documented here. ## 0.2.0 — local release candidate - Added verified RBI Bulletin sectoral-credit history for July 2025 through June 2026. This is a tested support boundary, not complete RBI history. - Added strict parser families for the verified 2025–2026 layout transitions. - Productionized official ASP.NET Bulletin archive discovery and a versioned, atomic historical cache with network-free replay. - Added the published-vintage model, current-observation view, stable series selection, and explicit `latest_publication` and `as_of` resolution. - Added methodology-boundary and source-provenance metadata. - Added an optional Matplotlib plotting layer and reproducible Non-food Credit CSV, manifest, SVG, and PNG example workflow. - Expanded deterministic, cache-integrity, parser, history, resolution, visualization, and regression coverage. ## 0.1.0 - Added the initial current RBI Sectoral Deployment of Bank Credit connector. - Added current Bulletin discovery and the strict June 2026 parser. - Added source provenance, deterministic hashing, atomic caching, and offline replay. ``` ### docs/testing.md ```markdown # Testing IndiaMacro IndiaMacro separates tests by the evidence they require. Files under `spike-artifacts/` may be generated outputs or optional local acceptance inputs, but ordinary tests and GitHub Actions never require them. ## Portable tests Portable unit and contract tests use committed compact fixtures, synthetic records, and mocked responses. They never make live RBI requests and run in every clean checkout: ```bash ruff check . pytest -m "not local_evidence and not live" -rs ``` This is the pytest layer run serially in GitHub Actions on Python 3.11 and 3.12. ## Preserved-evidence acceptance tests Tests marked `local_evidence` replay complete preserved RBI pages, the historical census, or the 12-release acceptance cache. These inputs remain ignored because they are operational evidence rather than package source or compact deterministic fixtures. When the evidence is available in its documented `spike-artifacts/` location, run: ```bash pytest -m local_evidence -rs ``` To run all available offline tests while still excluding network acceptance: ```bash pytest -m "not live" -rs ``` An unavailable local-evidence test reports the specific missing evidence instead of making a network request. ## Live acceptance Tests marked `live` are opt-in checks against current RBI source behavior. They are excluded from CI and from the normal offline suite. If live tests are present and network access has been explicitly authorized, run: ```bash pytest -m live -rs ``` The standalone current-source acceptance script remains explicitly opt-in: ```bash python scripts/run_rbi_sectoral_credit_live_acceptance.py ``` ``` ### pyproject.toml ``` [build-system] requires = ["setuptools>=77"] build-backend = "setuptools.build_meta" [project] name = "indiamacro" version = "0.2.0" description = "Auditable programmatic access to official Indian macroeconomic data" readme = "README.md" license = "MIT" license-files = ["LICENSE"] requires-python = ">=3.11,<3.13" dependencies = ["pandas>=2.0", "platformdirs>=4.0", "requests>=2.31"] classifiers = [ "Development Status :: 4 - Beta", "Intended Audience :: Developers", "Intended Audience :: Science/Research", "Operating System :: OS Independent", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3.11", "Programming Language :: Python :: 3.12", "Topic :: Scientific/Engineering :: Information Analysis", ] [project.optional-dependencies] plot = ["matplotlib>=3.8"] test = ["pytest>=8,<9", "ruff>=0.12"] build = ["build>=1.2", "twine>=6"] [project.urls] Homepage = "https://github.com/fotogfreaksandeep-arch/IndiaMacro" Repository = "https://github.com/fotogfreaksandeep-arch/IndiaMacro.git" Issues = "https://github.com/fotogfreaksandeep-arch/IndiaMacro/issues" [tool.setuptools.packages.find] where = ["src"] [tool.pytest.ini_options] pythonpath = ["src"] markers = [ "local_evidence: requires ignored preserved RBI evidence or acceptance caches", "live: opt-in acceptance test that may contact a live RBI source", ] [tool.ruff] line-length = 100 target-version = "py311" ``` ### tests/test_investigate_rbi_sectoral_credit_sources.py ```python import importlib.util import sys from pathlib import Path import pytest SCRIPT = Path(__file__).parents[1] / "scripts" / "investigate_rbi_sectoral_credit_sources.py" SPEC = importlib.util.spec_from_file_location("source_investigation", SCRIPT) investigation = importlib.util.module_from_spec(SPEC) assert SPEC.loader is not None sys.modules[SPEC.name] = investigation SPEC.loader.exec_module(investigation) def test_discovers_exact_bulletin_table_links(): html = """ 15. Deployment of Gross Bank Credit by Major Sectors 16. Industry-wise Deployment of Gross Bank Credit """ assert investigation.discover_bulletin_tables(html) == { "major_sectors": "https://rbi.org.in/Scripts/BS_ViewBulletin.aspx?Id=100", "industries": "https://rbi.org.in/Scripts/BS_ViewBulletin.aspx?Id=101", } def test_rejects_ambiguous_bulletin_table_link(): html = """ 15. Deployment of Gross Bank Credit by Major Sectors 15. Deployment of Gross Bank Credit by Major Sectors 16. Industry-wise Deployment of Gross Bank Credit """ with pytest.raises(investigation.InvestigationError, match="Expected one HTML link"): investigation.discover_bulletin_tables(html) @pytest.mark.parametrize( ("kind", "specific_markers"), [ ( "major_sectors", "Non-food Credit Agriculture & Allied Activities Industry Services Personal Loans Priority Sector " "sector-wise and industry-wise bank credit covers select banks accounting for about 95 per cent " "of total non-food credit extended by all scheduled commercial banks", ), ( "industries", "Food Processing Textiles Chemicals & Chemical Products Infrastructure", ), ], ) def test_validates_semantically_structured_html(kind, specific_markers): header = ( f"{investigation.TABLE_TITLES[kind]}₹ Crore" "Outstanding as onGrowth (%)Apr. 30, 2026Y-o-Y" ) rows = "".join(f"{i}1001.0x" for i in range(12)) html = f"{header}{rows}
{specific_markers}
" result = investigation.validate_bulletin_table(html, kind) assert result["valid_structured_html"] is True assert result["unit"] == "₹ Crore" assert result["has_yoy_growth"] is True assert result["header_cells"][0] == "Outstanding as on" ``` ### tests/test_spike_rbi_sectoral_credit_access.py ```python import importlib.util import sys import zipfile from pathlib import Path import pytest import requests SCRIPT = Path(__file__).parents[1] / "scripts" / "spike_rbi_sectoral_credit_access.py" SPEC = importlib.util.spec_from_file_location("rbi_spike", SCRIPT) spike = importlib.util.module_from_spec(SPEC) assert SPEC.loader is not None sys.modules[SPEC.name] = spike SPEC.loader.exec_module(spike) def test_discovers_latest_release_by_month_not_page_position(): html = """ Sectoral Deployment of Bank Credit – April 2026 Apr 30, 2026 Jun 30, 2026 Sectoral Deployment of Bank Credit – May 2026 Sectoral Deployment of Bank Credit – December 2025 """ release = spike.discover_latest_release(html) assert release.title.endswith("May 2026") assert release.url == "https://rbi.org.in/Scripts/BS_PressReleaseDisplay.aspx?prid=3" assert release.release_date.isoformat() == "2026-06-30" def test_workbook_requires_statements_label_and_resolves_relative_url(): html = """ unrelated spreadsheet

Data are set out in Statements I and II.

""" assert spike.discover_workbook_url(html, spike.INDEX_URL) == "https://rbidocs.rbi.org.in/rdocs/content.xlsx" def test_rejects_non_rbi_workbook_host(): html = "Statements I and II" with pytest.raises(spike.SpikeError, match="non-official"): spike.discover_workbook_url(html, spike.INDEX_URL) def test_html_with_successful_http_semantics_cannot_validate_as_xlsx(tmp_path): response_file = tmp_path / "response.part" response_file.write_bytes(b"200 OK but blocked") with pytest.raises(spike.SpikeError, match="is HTML"): spike.validate_xlsx_response(response_file) def test_http_200_tspd_challenge_is_source_blocked(): response = requests.Response() response.status_code = 200 response.url = "https://rbidocs.rbi.org.in/rdocs/content/docs/file.xlsx" with pytest.raises(spike.SpikeError) as caught: spike._check_response_access( response, b'', "download_workbook", ) assert caught.value.status == "SOURCE_BLOCKED" def test_minimal_structural_xlsx_validates(tmp_path): workbook = tmp_path / "valid.part" with zipfile.ZipFile(workbook, "w") as archive: archive.writestr("[Content_Types].xml", "") archive.writestr("xl/workbook.xml", "") spike.validate_xlsx_response(workbook) ``` ### tests/test_sectoral_credit_hashes.py ```python from __future__ import annotations import hashlib from decimal import Decimal from pathlib import Path import pytest from indiamacro.rbi.sectoral_credit_bulletin import ( SEMANTIC_OBSERVATION_COLUMNS, parse_sectoral_credit_bulletin, provenance_bound_output_sha256, semantic_observations_sha256, ) FIXTURES = Path(__file__).parent / "fixtures" MAJOR = (FIXTURES / "rbi_sectoral_credit_major_v1.html").read_bytes() INDUSTRIES = (FIXTURES / "rbi_sectoral_credit_industries_v1.html").read_bytes() def _parse(major: bytes = MAJOR, industries: bytes = INDUSTRIES): return parse_sectoral_credit_bulletin( major, industries, major_sectors_url="https://rbi.org.in/table-15", industries_url="https://rbi.org.in/table-16", ) def test_semantic_hash_uses_the_documented_economic_columns() -> None: assert SEMANTIC_OBSERVATION_COLUMNS == ( "dataset_id", "series_id", "source_table", "source_row_code", "source_label", "parent_row_code", "sector_level_1", "sector_level_2", "sector_level_3", "measure", "observation_date", "comparison_date", "value", "unit", "population_id", "publication_date", "bulletin_period", "is_provisional", "is_memorandum", "footnote_references", ) def test_identical_semantic_observations_have_identical_hashes() -> None: observations = _parse().observations assert semantic_observations_sha256(observations) == semantic_observations_sha256( observations.copy(deep=True) ) def test_row_order_does_not_change_semantic_hash() -> None: observations = _parse().observations reordered = observations.sample(frac=1, random_state=17) assert semantic_observations_sha256(reordered) == semantic_observations_sha256( observations ) def test_dataframe_index_does_not_change_semantic_hash() -> None: observations = _parse().observations changed_index = observations.copy() changed_index.index = range(10_000, 10_000 + len(changed_index)) assert semantic_observations_sha256(changed_index) == semantic_observations_sha256( observations ) @pytest.mark.parametrize("column", ["source_url", "source_sha256", "parser_version"]) def test_transport_or_implementation_provenance_does_not_change_semantic_hash( column: str, ) -> None: observations = _parse().observations changed = observations.copy() changed.loc[changed.index[0], column] = f"changed-{column}" assert semantic_observations_sha256(changed) == semantic_observations_sha256( observations ) @pytest.mark.parametrize( ("column", "replacement"), [ ("observation_date", "2099-01-01"), ("unit", "OTHER_UNIT"), ("measure", "OTHER_MEASURE"), ("population_id", "OTHER_POPULATION"), ("series_id", "RBI.OTHER.SERIES"), ("is_memorandum", True), ], ) def test_economic_classification_changes_semantic_hash(column: str, replacement) -> None: observations = _parse().observations changed = observations.copy() changed.loc[changed.index[0], column] = replacement assert semantic_observations_sha256(changed) != semantic_observations_sha256( observations ) def test_economic_value_change_changes_semantic_hash() -> None: observations = _parse().observations changed = observations.copy() changed.loc[changed.index[0], "value"] += Decimal("1") assert semantic_observations_sha256(changed) != semantic_observations_sha256( observations ) def test_page_chrome_mutation_separates_raw_semantic_and_provenance_identity() -> None: chrome_mutation = MAJOR.replace(b"", b"", 1) assert hashlib.sha256(chrome_mutation).hexdigest() != hashlib.sha256(MAJOR).hexdigest() original = _parse() changed = _parse(major=chrome_mutation) assert ( original.metadata.semantic_observations_sha256 == changed.metadata.semantic_observations_sha256 ) assert ( original.metadata.provenance_bound_output_sha256 != changed.metadata.provenance_bound_output_sha256 ) def test_identical_complete_observations_have_identical_provenance_hashes() -> None: observations = _parse().observations assert provenance_bound_output_sha256(observations) == provenance_bound_output_sha256( observations.copy(deep=True) ) @pytest.mark.parametrize("column", ["source_url", "source_sha256"]) def test_source_provenance_change_changes_provenance_bound_hash(column: str) -> None: observations = _parse().observations changed = observations.copy() changed.loc[changed.index[0], column] = f"changed-{column}" assert provenance_bound_output_sha256(changed) != provenance_bound_output_sha256( observations ) ``` ### scripts/run_rbi_sectoral_credit_live_acceptance.py ```python #!/usr/bin/env python3 """Run one clean-cache live connector acceptance and verified offline replay.""" from __future__ import annotations import argparse import importlib import json import sys from dataclasses import asdict from datetime import datetime, timezone from pathlib import Path from typing import Any, Iterable from indiamacro import rbi def _write_evidence(output_dir: Path, evidence: dict[str, Any]) -> None: output_dir.mkdir(parents=True, exist_ok=True) (output_dir / "live_acceptance_manifest.json").write_text( json.dumps(evidence, ensure_ascii=False, indent=2, sort_keys=True) + "\n", encoding="utf-8", ) live = evidence.get("live_result") or {} report = f"""# RBI sectoral-credit connector v1 validation report Status: **{evidence['status']}** ## Offline verification - Serial Ruff run before acceptance: PASS - Serial pytest run before acceptance: PASS (reported separately) - Mocked connector tests make no real network requests: PASS ## Live clean-cache acceptance - Started from an empty explicit cache: {evidence.get('empty_cache_start', False)} - Observation count: {live.get('observation_count')} - Semantic observations SHA-256: {live.get('semantic_observations_sha256')} - Provenance-bound output SHA-256: {live.get('provenance_bound_output_sha256')} - Bulletin publication date: {live.get('bulletin_publication_date')} - Latest observation date: {live.get('latest_observation_date')} - Major-sectors URL: {live.get('major_sectors_url')} - Industries URL: {live.get('industries_url')} - Dedicated-release period: {live.get('latest_dedicated_release_period')} - Freshness status: {live.get('freshness_status')} - Freshness gap months: {live.get('freshness_gap_months')} - Raw source pages committed: {evidence.get('raw_pages_cached', False)} ## Offline replay - Replay attempted with the same cache: {evidence.get('offline_replay_attempted', False)} - Network-session creation disabled during replay: {evidence.get('offline_network_disabled', False)} - Semantic SHA-256 matched live retrieval: {evidence.get('offline_semantic_hash_matched', False)} - Provenance-bound SHA-256 matched live retrieval: {evidence.get('offline_provenance_hash_matched', False)} ## Diagnostic {evidence.get('diagnostic') or 'None'} """ (output_dir / "connector_validation_report.md").write_text(report, encoding="utf-8") def run(cache_dir: Path, output_dir: Path) -> tuple[dict[str, Any], int]: if cache_dir.exists() and any(cache_dir.iterdir()): raise ValueError(f"Acceptance cache must start empty: {cache_dir}") cache_dir.mkdir(parents=True, exist_ok=True) evidence: dict[str, Any] = { "status": "FAIL", "acceptance_started_at_utc": datetime.now(timezone.utc).isoformat(), "empty_cache_start": True, "cache_dir": str(cache_dir.resolve()), "live_result": None, "raw_pages_cached": False, "offline_replay_attempted": False, "offline_network_disabled": False, "offline_semantic_hash_matched": False, "offline_provenance_hash_matched": False, "diagnostic": None, } try: live = rbi.sectoral_credit(refresh=True, cache_dir=cache_dir) metadata = asdict(live.metadata) evidence["live_result"] = { **metadata, "observation_count": len(live.observations), } bundle_dir = ( cache_dir / "rbi" / "sectoral_credit" / live.metadata.bulletin_publication_date / live.metadata.cache_bundle_id ) evidence["raw_pages_cached"] = all( (bundle_dir / name).is_file() for name in ("major_sectors.html", "industries.html", "manifest.json") ) connector = importlib.import_module("indiamacro.rbi.sectoral_credit") original_session = connector._new_session def reject_network(): raise AssertionError("Offline replay attempted to create a network session") evidence["offline_replay_attempted"] = True connector._new_session = reject_network evidence["offline_network_disabled"] = True try: offline = rbi.sectoral_credit(offline=True, cache_dir=cache_dir) finally: connector._new_session = original_session evidence["offline_semantic_hash_matched"] = ( offline.metadata.semantic_observations_sha256 == live.metadata.semantic_observations_sha256 ) evidence["offline_provenance_hash_matched"] = ( offline.metadata.provenance_bound_output_sha256 == live.metadata.provenance_bound_output_sha256 ) if ( not evidence["raw_pages_cached"] or not evidence["offline_semantic_hash_matched"] or not evidence["offline_provenance_hash_matched"] ): raise AssertionError("Cache preservation or offline replay validation failed") evidence["status"] = "PASS" exit_code = 0 except rbi.SourceAccessBlockedError as exc: evidence["status"] = "LIVE_SOURCE_BLOCKED" evidence["diagnostic"] = f"{type(exc).__name__}: {exc}" exit_code = 2 except rbi.SourceUnavailableError as exc: evidence["status"] = "LIVE_ENVIRONMENT_BLOCKED" evidence["diagnostic"] = f"{type(exc).__name__}: {exc}" exit_code = 2 except Exception as exc: evidence["status"] = "FAIL" evidence["diagnostic"] = f"{type(exc).__name__}: {exc}" exit_code = 1 evidence["acceptance_finished_at_utc"] = datetime.now(timezone.utc).isoformat() _write_evidence(output_dir, evidence) return evidence, exit_code def main(argv: Iterable[str] | None = None) -> int: parser = argparse.ArgumentParser(description=__doc__) parser.add_argument("--cache-dir", required=True, type=Path) parser.add_argument( "--output-dir", type=Path, default=Path("spike-artifacts/connector-v1 [truncated — 256 more characters] ``` ### scripts/build_rbi_sectoral_credit_parser_evidence.py ```python """Build bounded, offline evidence for the Bulletin sectoral-credit parser.""" from __future__ import annotations import argparse import hashlib import json from decimal import Decimal from pathlib import Path from indiamacro.rbi.sectoral_credit_bulletin import ( INDUSTRY_TITLE, MAJOR_TITLE, MEASURE_YOY_GROWTH, observations_to_csv_bytes, parse_sectoral_credit_bulletin, ) EXPECTED_HASHES = { MAJOR_TITLE: "a7143fcd39cf1d3538de036893d0798f0cef8c1a7afe298249d0ba5cebb2d17c", INDUSTRY_TITLE: "115b59651d1bc7f8b3f80480c4e7becc18e946172d4570f848a8881ea9360e8b", } MAJOR_URL = "https://rbi.org.in/Scripts/BS_ViewBulletin.aspx?Id=24257" INDUSTRY_URL = "https://rbi.org.in/Scripts/BS_ViewBulletin.aspx?Id=24258" GOLDEN_VALUES = { "III. Non-food Credit": Decimal("15.8"), "1. Agriculture & Allied Activities": Decimal("13.7"), "2. Industry (Micro and Small, Medium and Large)": Decimal("15.1"), "3. Services": Decimal("18.6"), "4. Personal Loans": Decimal("16.0"), "2.4 Textiles": Decimal("8.3"), "2.2.1 Sugar": Decimal("-0.4"), "2.18.3 Roads": Decimal("0.4"), } def _sha256(content: bytes) -> str: return hashlib.sha256(content).hexdigest() def _pairs(values: tuple[tuple[str, int], ...]) -> dict[str, int]: return dict(values) def _check_goldens(observations) -> list[dict[str, str]]: checks: list[dict[str, str]] = [] for label, expected in GOLDEN_VALUES.items(): matches = observations.loc[ (observations["source_label"] == label) & (observations["measure"] == MEASURE_YOY_GROWTH) ] if len(matches) != 1: raise AssertionError(f"Golden row {label!r} matched {len(matches)} observations") row = matches.iloc[0] if row["value"] != expected: raise AssertionError( f"Golden row {label!r}: expected {expected}, got {row['value']}" ) checks.append( { "source_label": label, "measure": MEASURE_YOY_GROWTH, "observation_date": row["observation_date"], "comparison_date": row["comparison_date"], "expected_value": format(expected, "f"), "result": "PASS", } ) return checks def build_evidence(major_path: Path, industry_path: Path, output_dir: Path) -> None: major = major_path.read_bytes() industry = industry_path.read_bytes() actual_hashes = {MAJOR_TITLE: _sha256(major), INDUSTRY_TITLE: _sha256(industry)} if actual_hashes != EXPECTED_HASHES: raise RuntimeError( "Preserved source hashes differ from the parsing contract: " f"expected={EXPECTED_HASHES!r}, actual={actual_hashes!r}" ) result = parse_sectoral_credit_bulletin( major, industry, major_sectors_url=MAJOR_URL, industries_url=INDUSTRY_URL, ) del major, industry output_csv = observations_to_csv_bytes(result.observations) output_hash = _sha256(output_csv) if output_hash != result.metadata.provenance_bound_output_sha256: raise AssertionError("Serialized output hash differs from parser metadata") golden_checks = _check_goldens(result.observations) metadata = result.metadata manifest = { "status": "PASS", "dataset_id": metadata.dataset_id, "layout_id": metadata.layout_id, "layout_signature_sha256": metadata.layout_signature_sha256, "parser_version": metadata.parser_version, "input_hashes": { "major_sectors": actual_hashes[MAJOR_TITLE], "industries": actual_hashes[INDUSTRY_TITLE], }, "publication_date": metadata.publication_date, "bulletin_period": metadata.bulletin_period, "current_observation_date": metadata.current_observation_date, "source_rows_by_table": _pairs(metadata.data_row_counts), "mapped_rows_by_table": _pairs(metadata.mapped_row_counts), "emitted_rows_by_table": _pairs(metadata.emitted_row_counts), "canonical_observation_count": len(result.observations), "observation_counts_by_measure": _pairs(metadata.observation_counts_by_measure), "observation_counts_by_population": _pairs(metadata.observation_counts_by_population), "unknown_row_count": metadata.unknown_row_count, "duplicate_source_column_count": metadata.duplicate_source_column_count, "duplicate_source_row_count": metadata.duplicate_source_row_count, "canonical_duplicate_key_count": metadata.canonical_duplicate_key_count, "growth_reconciliation": { "checks": metadata.growth_reconciliation_checks, "skipped": metadata.growth_reconciliation_skipped, "failures": metadata.growth_reconciliation_failures, "tolerance_percentage_points": ( metadata.growth_reconciliation_tolerance_percentage_points ), }, "golden_value_checks": golden_checks, "semantic_observations_sha256": metadata.semantic_observations_sha256, "provenance_bound_output_sha256": output_hash, } report = f"""# RBI sectoral-credit parser v1 validation report Status: **PASS** The two preserved June 2026 RBI Bulletin HTML pages were parsed offline. No network request, live discovery, browser, or historical source was used. ## Source and layout validation - Exact preserved SHA-256 values: PASS - Exact titles and unique semantic table matches: PASS - Unit `(₹ Crore)`: PASS - Publication date extraction (`{metadata.publication_date}`): PASS - Bulletin period derivation (`{metadata.bulletin_period}`): PASS - Merged header, source-column roles, and dates: PASS - Required population and methodology notes: PASS - Section-42 column-(2) override (`2025-05-02`): PASS - Complete explicit mappings: PASS ({dict(metadata.mapped_row_counts)}) - Unknown rows: PASS ({metadata.unknown_row_count}) - Missing req [truncated — 2721 more characters] ``` ### tests/test_sectoral_credit_history_census.py ```python from __future__ import annotations import hashlib import importlib.util import json import sys from copy import deepcopy from pathlib import Path import pytest SCRIPT = Path(__file__).parents[1] / "scripts" / "census_rbi_sectoral_credit_history.py" SPEC = importlib.util.spec_from_file_location("historical_census", SCRIPT) census = importlib.util.module_from_spec(SPEC) assert SPEC.loader is not None sys.modules[SPEC.name] = census SPEC.loader.exec_module(census) ARCHIVE_URL = census.ARCHIVE_URL CandidateAmbiguous = census.CandidateAmbiguous CensusClient = census.CensusClient RequestBudgetExceeded = census.RequestBudgetExceeded SourceBlocked = census.SourceBlocked SourceValidationError = census.SourceValidationError _extract_form_payload = census._extract_form_payload _v1_reference = census._v1_reference build_sample_schedule = census.build_sample_schedule boundary_expansion_periods = census.boundary_expansion_periods classify_v1_result = census.classify_v1_result discover_table_candidates = census.discover_table_candidates extract_table_inventory = census.extract_table_inventory taxonomy_delta = census.taxonomy_delta FORM_HTML = """
June """ ISSUE_HTML = """
Date : Jun 23, 2025
Reserve Bank of India Bulletin - June 2025
15. Deployment of Gross Bank Credit by Major Sectors 16. Industry-wise Deployment of Gross Bank Credit """ class FakeResponse: def __init__(self, body: bytes, *, status: int = 200) -> None: self.body = body self.status_code = status self.url = ARCHIVE_URL self.history: list[FakeResponse] = [] self.headers = {"Content-Type": "text/html; charset=utf-8"} def iter_content(self, _size: int): yield self.body def close(self) -> None: pass def test_archive_form_selection_preserves_state() -> None: payload = _extract_form_payload(FORM_HTML, year=2025, month=6) assert payload["__VIEWSTATE"] == "state" assert payload["hdnYear"] == "2025" assert payload["hdnMonth"] == "6" assert payload["UsrFontCntr$btn"] == "" assert "btnGo" not in payload def test_exact_and_variant_title_detection() -> None: found = discover_table_candidates(ISSUE_HTML) assert found["major"].number == "15" assert found["industry"].number == "16" variant = ISSUE_HTML.replace("Gross Bank Credit", "Bank Credit") found = discover_table_candidates(variant) assert found["major"].title.endswith("Bank Credit by Major Sectors") assert found["industry"].title.endswith("Deployment of Bank Credit") def test_candidate_official_domain_enforcement() -> None: evil = ISSUE_HTML.replace( 'href="BS_ViewBulletin.aspx?Id=101"', 'href="https://example.com/Scripts/BS_ViewBulletin.aspx?Id=101"', ) with pytest.raises(SourceValidationError, match="outside RBI"): discover_table_candidates(evil) def test_ambiguous_and_missing_candidates() -> None: duplicate = ISSUE_HTML.replace( "", '' "15. Deployment of Gross Bank Credit by Major Sectors", ) with pytest.raises(CandidateAmbiguous, match="Multiple major"): discover_table_candidates(duplicate) missing = discover_table_candidates("No tables") assert missing == {"major": None, "industry": None} def test_sample_schedule_is_bounded_and_deterministic() -> None: first = build_sample_schedule("2026-06") second = build_sample_schedule("2026-06") periods = [item["period"] for item in first] assert first == second assert len(first) == 37 assert len(periods) == len(set(periods)) assert periods[0] == "2010-03" assert periods[-1] == "2026-06" assert {"2026-01", "2025-06", "2024-06", "2023-06", "2022-06", "2020-06"} <= set(periods) def test_boundary_expansion_narrows_first_availability() -> None: issues = [ { "requested_bulletin_period": "2012-03", "v1_result": "MISSING_TABLE_PAIR", "major_table": None, "industry_table": None, }, { "requested_bulletin_period": "2013-03", "v1_result": "NEW_LAYOUT", "major_table": {}, "industry_table": {}, }, ] assert boundary_expansion_periods(issues, {"2012-03", "2013-03"}) == [ "2012-12", "2013-01", "2013-02", ] def test_request_budget_is_enforced_without_network(tmp_path: Path) -> None: client = CensusClient(tmp_path, max_requests=0) try: with pytest.raises(RequestBudgetExceeded): client.fetch(ARCHIVE_URL, context="offline budget test") finally: client.close() def test_semantic_signature_is_deterministic() -> None: path = Path("tests/fixtures/rbi_sectoral_credit_major_v1.html") html = path.read_text(encoding="utf-8") one = extract_table_inventory( html, published_title="15. Deployment of Gross Bank Credit by Major Sectors" ) two = extract_table_inventory( html, published_title="15. Deployment of Gross Bank Credit by Major Sectors" ) assert one["semantic_layout_signature"] == two["semantic_layout_signature"] assert one["structural_signature"] == two["structural_signature [truncated — 3353 more characters] ``` ### src/indiamacro/__init__.py ```python """IndiaMacro public package.""" from importlib.metadata import PackageNotFoundError, version try: __version__ = version("indiamacro") except PackageNotFoundError: __version__ = "0+unknown" from indiamacro import rbi as rbi from indiamacro.rbi.sectoral_credit_bulletin import ( AmbiguousTableError, DataValidationError, ParseMetadata, ParsedSectoralCredit, SectoralCreditParseError, SourceNote, UnmappedSeriesError, UnsupportedLayoutError, observations_to_csv_bytes, parse_sectoral_credit_bulletin, ) __all__ = [ "AmbiguousTableError", "DataValidationError", "ParseMetadata", "ParsedSectoralCredit", "SectoralCreditParseError", "SourceNote", "UnmappedSeriesError", "UnsupportedLayoutError", "__version__", "observations_to_csv_bytes", "parse_sectoral_credit_bulletin", "rbi", ] ``` [33 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]