Governed Data Platform

BigQuery made loading data easy, and the organization responded accordingly. There are now several thousand tables. A meaningful number contain personal data. Nobody can say which ones without opening them, and the access model is a small number of groups with broad read on entire datasets, because per-table grants stopped being maintainable around table four hundred.

The governance response so far has been a spreadsheet. It lists data owners, it is nine months out of date, and it has no relationship to what IAM actually permits.

Then a customer contract arrives with a data residency clause, or a privacy review asks which analysts can read the column containing national identifiers, and the honest answer is that nobody knows. Answering it properly means classifying data that has already landed, which is a much larger job than classifying it on the way in would have been.

Classification at the point of landing

Sensitive Data Protection — the service still widely known by its Cloud DLP API — is configured to inspect data as it arrives rather than as an annual audit. Info types are tuned to what your business actually holds, because the default detector set produces a large volume of false positives on any real dataset and a scanner nobody trusts is a scanner nobody reads.

De-identification and redaction are applied where raw values do not need to persist. Tokenization where the value must remain joinable, masking where it does not, and full redaction where the field should never have been collected in the first place.

Scheduled inspection jobs run against existing data on a cadence, with findings routed into Security Command Center so that data classification lands in the same place as the rest of your security posture rather than in a separate tool that only the data team looks at.

Dataplex, and making the catalogue load-bearing

A catalogue that only describes is documentation. Dataplex is configured so that the taxonomy actually enforces: policy tags applied to columns, attached to the classification the inspection produced, and wired into BigQuery column-level security so that the tag is what grants or denies the read.

That closes the gap between the governance model and the access model. When a column is tagged as containing personal data, the people without that tag’s reader role cannot select it — not by convention, by IAM.

Lakes, zones, and assets are structured so that raw, curated, and consumption data are separated by boundary rather than by naming convention, and so that a new dataset arrives into a zone with governance already attached rather than into an unmanaged project.

Column-level security, masking, and conditional access

• +BigQuery column-level security driven by policy tags, so access decisions live with the taxonomy rather than in a per-table grant nobody maintains.
• +Dynamic data masking, so that an analyst who needs to group by a column but not read it gets a masked value instead of an access denial — which is what stops people from requesting broader access than they need.
• +IAM Conditions for access that depends on context: time-bounded grants for an investigation, resource-scoped grants that expire, rather than a permanent group membership added during an incident and never removed.
• +VPC Service Controls perimeters around the projects holding regulated data, so that a set of valid credentials used from outside the perimeter still cannot exfiltrate.

Self-serve ingestion and quality gates

The ingestion framework exists so that a team with a new data source does not need to file a ticket with the platform team and wait a fortnight. A new source is onboarded through configuration: connector, schema, classification policy, target zone, quality expectations.

Quality gates run before data reaches consumption. Row counts against expected ranges, null rates on fields that should not be null, referential checks where they matter, freshness thresholds. Failures route to the owning team and stop promotion rather than quietly writing bad data into a table that a dashboard reads on Monday.

Pipelines are built on Dataflow and Pub/Sub for streaming and batch, orchestrated with Cloud Composer. The orchestration layer is where the quality gates and classification steps are sequenced, so governance is a step in the pipeline rather than a process beside it.

Boundaries

Source system integration beyond the agreed connectors is out of scope. Connectors in scope are named during Discovery. A source that needs bespoke extraction work against a legacy system is its own project.

BI and dashboard development is out of scope. The platform makes governed data available to your BI tool; building the reports on top belongs to the analysts who know what the business asks.

Migration from a legacy warehouse is out of scope. Moving history out of an existing warehouse is a large programme with its own reconciliation requirements, and folding it into a platform build hides risk in both.