SIPA Data Explorer — Methodology

Data Source & Scraping

All data originates from SIPA — Sistema de Informação para o Património Arquitectónico, the Portuguese national heritage database maintained by DGPC. SIPA has no public API — records are only accessible via individual ASP.NET detail pages.

Scraping architecture

The scraper uses Python's asyncio + httpx with configurable concurrency and delays between batches to avoid overloading the SIPA server. Each monument page is parsed with BeautifulSoup against the ASP.NET span element IDs that structure each record.

Database schema

The scraped data is stored in a unified SQLite database with a 47-column schema covering identification, location, typology, chronology, materials, use history, ownership, protection, bibliography, and multimedia fields. Two scraper passes (IDs 1–20,000 and 20,001–36,416) are merged with cross-population of column variants.

Field	Coverage	Records
name	100%	26,380
tipologia1	98.9%	26,103
chronology	75.8%	20,004
construction_period	71.7%	18,903
bibliography	69.0%	18,208
description	57.4%	15,155
technical_data	57.2%	—
materials	51.6%	13,611
architect_builder	52.6%	13,863
coordinates	16.0%	3,447

Temporal Processing

The construction_period field contains free-text chronological information ranging from precise dates ("1742") to century ranges ("Séc. 16 / Séc. 18") to prehistorical references ("300000 a.C."). The temporal parser handles: Arabic centuries ("Séc. 14"), Roman centuries ("Séc. XIV"), specific years, year ranges, BCE dates, decade references ("anos 40 do Séc. 20"), and mixed formats. Century references expand to full ranges (e.g. "Séc. 16" → 1500–1599).

Of 26,380 monuments, 18,852 produce valid temporal ranges and power the mega-timeline streamgraph and strip chart.

Persons Parser

The architect_builder field uses a semi-structured notation where UPPERCASE role labels precede person names, separated by inconsistent delimiters. The parser evolved through 5 versions to handle this complexity.

Core challenge

SIPA's notation is heterogeneous: ARQUITECTO: Name1. PINTOR: Name2, ESCULTOR: Name3 — dots, commas, slashes, and spaces all serve as block separators, with no consistent delimiter. The key insight: UPPERCASE role labels are the structural markers, not punctuation.

Parser architecture (v5)

Six separator-fix regexes detect role label boundaries within unsplit text, each targeting a specific punctuation pattern: .ROLE:, / ROLE:, name ROLE:, , ROLE:, - ROLE:, : ROLE:. All are case-sensitive against a curated vocabulary of 170+ role terms sorted longest-first. A 60% uppercase heuristic gate prevents false matches on names containing colons.

Version 5 added dictionary-validated lowercase role detection (construtor: Name) using a case-insensitive lookup with strict validation — only known role terms pass. This reduced residual colon-in-name errors from 277 to 131, all of which are structural features of the source data (project breakdowns, work descriptions), not parser errors.

Materials & Elements Parser

The materials field follows Portuguese notation describing what elements are made of which materials. The parser operates in three modes depending on segment structure:

Case A — Multiple "em" per segment: Split at each "em" marker to extract multiple element→material pairs from one segment. Case B — Single "em" with sub-clauses: Comma-separated elements before "em", with "de material" patterns in each. Case C — No "em": Fallback to standalone vocabulary scanning for the 38% of records that are material lists without explicit links.

Controlled vocabularies

51 base materials, 35 compound materials (ferro forjado, telha marselha), 104 architectural elements. Compound terms matched first (longest match wins). Plural forms normalized to singular. Material normalization resolves compounds: "cantaria de granito" → granito, "alvenaria de pedra" → pedra.

Validation

100-monument stratified sample (by field length, region, typology). Ground truth from second-pass parser with manual review.

Metric	Pre-patch	Post-patch	Target	Status
Elements F1	0.959	0.989	≥ 0.90	PASS
Materials F1	0.756	0.964	≥ 0.90	PASS
Pairs F1	0.703	0.963	≥ 0.85	PASS

Network construction

Three edge types: bipartite (element→material), element co-occurrence, material co-occurrence. All edges include both raw count and NPMI (Normalized Pointwise Mutual Information) to surface non-obvious associations — e.g. grade→ferro (NPMI=0.400) and estátua→bronze (NPMI=0.310) emerge as genuinely strong, while estrutura→granito becomes unremarkable despite high count.

Getty AAT Alignment

The Getty Art & Architecture Thesaurus provides international controlled vocabulary for heritage materials, elements, and building types. The SIPA mapping evolved through three versions:

Version	Terms	Coverage	Source
v1	142	—	Materials export parser vocabulary
v2	273	53.5% token mentions	+ description pipeline expansion
v3	387	97.9% segment resolution	+ normalization pipeline

Normalization pipeline (v3)

Five-phase pipeline:

Phase 1 — Inflection generation: Portuguese morphological rules (11 patterns: plurals, gender swap, participials) generate 1,407 variant forms from 387 canonical terms. Exclusions for fixed-gender suffixes (-ista, -ura, -eza).

Phase 2 — Compound decomposition: Each of 40,912 material-field segments is decomposed: first scan for 71 compound terms (longest-first), then split on 16 Portuguese prepositions, then match single tokens against the variant map, classifying leftovers as qualifiers (165 surface treatment/color terms), noise (65 context words), or residuals.

Phase 3 — New term identification: 114 new terms added: wood species (10), stone types (8), surface treatments (6), tile subtypes (5), architectural elements (40+), compound terms (22), modern materials (16).

Phase 4 — AAT verification: All terms carry Getty AAT IDs verified against a local XML index (58,997 concepts). 59 of 114 new terms verified; 55 awaiting full SPARQL confirmation.

Description AAT pipeline

A separate parser scans the description, technical_data, and description_complement fields using the mapping dictionary + morphological variants. Two-pass matching: compounds first, then single terms via inflection lookup. Validated on 15-monument stratified sample: P=1.000, R=0.963, F1=0.981 at AAT-concept level (zero false positives).

The unified AAT profile combines materials + description annotations: 15,793 monuments with AAT data, 209 unique concepts, 563,738 total mentions.

Bibliography Parser

The bibliography field contains free-text reference lists (avg. 3-4 per monument, up to 100+). The parser extracts structured fields: authors, title, place, year, type.

Key challenges

Compound surnames: Portuguese surnames with connectors ("e", "da", "de", "dos") — e.g. "Rocha e Sousa" must not split on "e". Solution: particle-aware splitting with a priority ordering (dos > do > da > de > e) and compound surname preservation.

IDEM entries: ~374 entries use "IDEM" (same author as above) — a bibliographic convention, not a person name. These are filtered from the network but the underlying references are preserved.

Deduplication: Title normalization + fuzzy matching achieves 60.1% deduplication ratio (26,149 unique from ~65,000 raw entries). Title contamination (author text bleeding into titles) reduced by 90.6% from v1 to v4.

CIDOC-CRM Semantic Mapping

All SIPA data is mapped to CIDOC-CRM 7.1.3 + CRMba 1.4 (Built Architecture extension) through a formal register of 22 field mappings (18 active, 4 planned). Each monument becomes a crmba:B1_Built_Work node connected to production events, temporal spans, places, persons, and materials via CRM properties.

The mapping uses 20 CRM classes and links to Getty AAT for material/element/typology classification. A 100-monument sample export produces 19,149 lines of Turtle RDF with ~12,045 triples. The full export covers all 26,380 monuments (2.26M triples, 215 MB Turtle). See the Ontology page for the interactive mapping browser.

External vocabulary linking: GeoNames owl:sameAs is active for 18 mainland districts (94.5% coverage). Getty ULAN (persons) and Wikidata (monuments) linking is planned.

Authors Network

The authors network extracts personal (non-corporate) authors from the bibliography and builds co-occurrence edges between authors who write about the same monuments. This is distinct from the persons network (who built monuments) — only 13 surname fragments overlap between the two networks.

IDEM/IBIDEM pseudo-entries are filtered. Co-authorship edges (published together) are tracked separately from co-occurrence edges (wrote about the same buildings). NPMI normalization surfaces genuine research partnerships beyond frequency effects.

Visualizations

Tesauro SIPA & Vocabulary Crosswalk

The Tesauro SIPA is a formal SKOS thesaurus published by DGPC on the ROSSIO infrastructure (FCSH/NOVA University), controlling the terminology used in SIPA record production. It contains 7,733 concepts organized in 15 micro-thesauri covering: Activities/Events/Processes, Urban Agglomerations, Attributes/Properties, Compartments/Spaces, Buildings/Structures, Architectural Elements, Entities, Materials/Products, Objects/Equipment/Furniture, Living Organisms, Landscape, Parks/Gardens, Records/Documents, Special Systems/Installations, and Territorial Units.

The Tesauro is multilingual: 100% Portuguese, 100% English (7,719 bilingual pairs), 81% French, 80% Italian. It was downloaded as SKOS Turtle (3.0 MB) and parsed into a structured JSON index for crosswalk analysis.

Three-layer vocabulary alignment: Our pipeline produces a crosswalk between three independent vocabularies. The SIPA Tesauro provides authoritative Portuguese terminology with English translations. Our hand-curated dictionary normalizes the raw text from SIPA records into 284 canonical terms. The Getty AAT provides international URI identifiers for each concept.

Layer	Source	Terms	Role
Tesauro SIPA	DGPC / ROSSIO	7,733	Authoritative PT vocabulary + EN translation
Our mapping	Hand-curated	284	Text normalization + deduplication
Getty AAT	Getty Research	58,997	International URI identifiers

Of our 277 mapped terms, 169 match exactly in the Tesauro SIPA and a further 61 match via plural or prefix variants (e.g. "granito" to "Granito (Pedra segundo a natureza)"), giving an 83% alignment rate. The remaining 47 are either compound terms, spelling variants, or highly specific materials that the Tesauro classifies at a different granularity level. The crosswalk data is available for download.

Adaptive Reuse Analysis

SIPA records separate original function (original_use, 26,317 filled) from current function (current_use, 22,849 filled), with a derived use_change flag. 6,157 monuments (23.3%) changed function. The dominant transition is toward cultural and recreational use: 2,794 monuments (45% of changed) became museums, historic landmarks, or exhibition spaces. The abandonment rate is significant: 1,150 monuments are classified as devoluto (abandoned), concentrated in residential, religious, and industrial typologies. An additional 79 are demolished and 39 have disappeared entirely.

These transitions are visualized as a D3 Sankey diagram in the Adaptive Reuse page, with 157 flows of 3+ monuments each.

Existence State Inference

SIPA's internal Grau field (encoding existence, heritage quality, record completeness, and accessibility) is not part of the public 28-field schema. To approximate the existence dimension, we derive a estado classification from the current_use and observations text fields. The inference produces: standing (21,327 / 80.8%), unknown (3,529 / 13.4%), abandoned (1,344 / 5.1%), demolished (79 / 0.3%), ruins (43 / 0.2%), disappeared (39 / 0.1%), and under construction (19 / 0.1%). This derived field is exposed as a filter in the Explorer and used in the CRM export.

Tools & Stack

Data pipeline: Python 3, SQLite, BeautifulSoup, asyncio/httpx (scraping), regex-based NLP (field parsing), local Getty AAT XML index (58,997 concepts). Visualization: D3.js v7 (streamgraph, force graphs), HTML5 Canvas (strip chart, persons network), static HTML with no build step. Ontology: CIDOC-CRM 7.1.3 TTL, CRMba 1.4 TTL, RDFLib for Turtle/JSON-LD export. Design system: Fraunces (headings), Helvetica (body), JetBrains Mono (data), dark theme with brick accent (#E6644E).