Pulse: Municipal Data Ingestion and Correlation for City Oversight
A system for pulling together municipal records, donor relationships, and project evidence so patterns can be reviewed without losing the source trail. This entry was rewritten against the running system, which had grown well past what the original post described.
What Pulse Is For
Pulse ingests municipal data, preserves the underlying sources, and correlates records that should be considered together during oversight work. It follows the chain from public decision to implementation, spending, and observable outcome.
The goal is not to jump to accusations. The goal is to surface evidence-backed patterns that deserve review — delays, cost drift, repeated exceptions, incomplete records, unusual donor proximity — with the trail back to the original record intact. That constraint shapes every design decision below.
The pipeline
The ordering matters: nothing can be correlated before it is ingested, and nothing should be reviewed before the trail back to the original claim is intact.
Collect, and keep the source
Around twenty collectors run on their own schedules: council decisions and votes from CivicWeb, provincial campaign-contribution and lobbyist-registry bulk exports, official award notices and tender records, federal corporate ownership, property and permit data, municipal statements of financial information, RSS feeds, and a local metasearch instance.
Each writes either RawItem-shaped rows or source documents and claims, deduplicated on source identity. Every record keeps a path back to the original.
Enrich
A processing pipeline runs on everything unprocessed: geocoding to neighbourhood, sentiment, zero-shot topic classification, named-entity recognition, an urgency score, then embedding. Every model loads lazily and runs locally.
Join on exact identifiers
Project numbers, bylaw references, parcel IDs, official page keys, vendor names, sourced entity matches. Broad similarity is useful for triage, but exact joins are what let the system build a defensible issue trail. Where the evidence is incomplete, the relationship stays unresolved rather than being filled in with an assumption.
Detect
A connector walks decisions and votes and raises signals for text-matched categories — competition exceptions, deferrals, closed-session use, development discretion — alongside outcome and amount driven ones. Nine concrete rules then run over the same evidence graph. Decision-anchored patterns become signals; cross-domain and temporal patterns become leads carrying a claim-backed chain.
Review
Findings surface in a council workbench — findings, issue trails, commitments and delivery, council activity — over a REST API, alongside a neighbourhood choropleth map and a relationship graph. Semantic search sits on top of the embeddings as an index back to source documents, not as a replacement for reading them.
The nine detection rules
The original version of this post described this layer only as “patterns that deserve review.” It is more specific than that. Nine named rules run against the evidence graph, each one a structural relationship between records rather than a judgement about a person.
| Rule | The structural relationship it looks for |
|---|---|
| vote_on_own_company | A decision-maker participating in a matter touching an entity they hold an interest in |
| vote_on_donor_company | A decision touching an entity connected to a disclosed contribution |
| donation_before_award | A disclosed contribution preceding an award, within a temporal window |
| lobbyist_contract_link | A registered lobbying relationship overlapping a contract record |
| undisclosed_family_interest | An ownership or relationship link not present in the disclosure record |
| contract_amendment_escalation | Amendments that move a contract materially away from its awarded value |
| contract_splitting | Awards divided so each falls below a procurement threshold |
| sole_source_repeat_vendor | Repeated non-competitive awards concentrating on one vendor |
| property_rezone_insider | A rezoning decision overlapping a traceable property interest |
Every one of these is a question, not an answer. A rule firing means two records line up in a way a human should look at. It does not mean anything improper occurred, and the system is built so that it cannot say otherwise.
The evidence firewall
Pulse uses a local language model, and the single most important design decision in the system is what that model is not allowed to do.
The rule, stated in the codebase.
Model output may score or summarise a finding. It never gates whether a signal is created or shown. A separate validation step checks generated text against the facts it was given and rejects invented numbers, dates, and identifiers outright. That is anti-hallucination validation, not a suppression layer — nothing is hidden because a model disliked it, and nothing is asserted because a model invented it.
All inference goes through a single local queue service rather than scattered calls, so the model path is one auditable chokepoint instead of a dozen. Nothing in the corpus is sent to a third-party API.
What it refuses to do
A system like this is defined as much by its declared blind spots as by its coverage. These are documented in the codebase rather than discovered later:
Registry coverage is partial, and says so
The corporate ownership collector reaches the federal registry only, so most small local vendors — provincially incorporated — will not match. Property title search requires an operator-mediated account, so the system queues human work rather than pretending to automate it. The lobbyist registry is provincial and does not cover municipal lobbying at all.
Unresolved beats convenient
Donor analysis treats only sourced relationships as valid graph edges, which keeps it out of inference-by-vibe. Undated search results are excluded from feeds entirely rather than being given a guessed date. Where a join cannot be made on an exact identifier, the relationship stays open.
What is being built now
The most recent work is an assistant layer: a chat interface over the corpus with tiered routing, a tool registry, and a two-level login. The routing tier matters for the same reason the firewall does — a cheap local model handles ordinary retrieval, and heavier reasoning is escalated deliberately rather than by default. Content embeddings were also generalised across every corpus in the system rather than just the base item table, which is what makes assistant retrieval work across council, procurement and disclosure material at once.
Why It Stays Conservative
Pulse is intentionally conservative about wording. The system can surface patterns, evidence gaps, and review candidates. It should not turn those into legal or disciplinary conclusions on its own.
That is what keeps the output useful. It gives a reviewer enough structure to follow the trail, and it leaves the final judgment to the record, the evidence, and the appropriate authority. A tool that cried fraud on a coincidence would be worse than no tool at all — it would burn the credibility that makes the genuine findings worth acting on.
Back to blog