Master Data — Bronze Layer

End-to-End Flow

Bronze is built on database mirroring plus a OneLake schema shortcut. Data is physically replicated once, by the mirror. Everything downstream is a pointer. No copy pipelines, no scheduled ingestion code.

Azure SQL (ESB Config DB) │ Fabric Mirroring — continuous, near real time, no code ▼ MasterData_ESBConfig_RawData (Mirrored Database) → Tables/dbo (Delta) │ OneLake schema shortcut — pointer only, no data copied ▼ MasterData_Bronze_DWH (Lakehouse) → schema "raw"

Querying raw.{table} in the Bronze lakehouse reads the mirror's live copy directly. Every table the mirror lands appears in raw automatically.

Step 1 — Source Mirroring

Item: MasterData_ESBConfig_RawData (Mirrored Database)
Source: Azure SQL Database (ESB Config), via Fabric connection 3e2f21f8-b56e-46f0-82e4-f73bf0bb2b5a
Behaviour: Fabric Mirroring replicates the source tables into OneLake as Delta, continuously and automatically. No table filter is configured, so the whole database (as permitted by the connection) is mirrored into Tables/dbo.
This is the only point in Bronze where data is physically copied.

Step 2 — Bronze Lakehouse & the raw Shortcut

Item: MasterData_Bronze_DWH (schema-enabled Lakehouse, default schema dbo)
raw is a single schema-level shortcut sitting directly under Tables, targeting the mirror's entire Tables/dbo folder. One pointer surfaces every mirrored table.
Cross-workspace schema shortcuts can be added through the create_schema_shortcut() notebook function call (e.g. CRL Bronze landing_crm). Note that OneLake shortcuts pass the calling user's identity through to the target, so consumers need read access on the source lakehouse as well.

Step 3 — Deployment Routine Current

Because a shortcut's target GUIDs are environment-specific, each environment recreates its shortcuts after deployment using a config-driven routine:

Item	Role
`Variable_Library`	Holds workspace and mirror GUIDs per environment. Dev values are the defaults; `QA` / `UAT` / `PRD` value sets override them. The active value set must be selected once in each workspace.
`00_functions` (Notebook)	Defines `create_schema_shortcut()` — calls the Fabric shortcuts API with `shortcutConflictPolicy=CreateOrOverwrite` to create a schema-level shortcut. Destination defaults to the attached lakehouse.
`01_create_shortcuts` (Notebook)	Loads `00_functions` via `%run`, reads the source GUIDs from the Variable Library, and creates `raw` ← mirror `dbo`.
`Deployment` (Data Pipeline)	Runs `01_create_shortcuts` post-deployment. The pipeline points at the notebook in its own workspace, so in each environment it automatically runs that environment's copy of the notebook. No per-environment IDs to maintain.

Environments

Every environment works the same way once deployed: the deployment routine creates the raw schema shortcut using that environment's GUIDs from the Variable Library value sets.

Env	Workspace GUID	Mirror GUID
DEV	`dab5b6e7-a539-47c1-9720-dc4c651aa986`	`116b9ee4-65d8-45b9-9964-c8d86da55299`
QA	`d7811402-fca3-45c0-8e13-ec3a48d787a7`	`ea571931-cb57-4c15-9022-111dbd4309fa`
UAT	`6403e0ae-35e1-458a-b5be-ecd9b6425d07`	`141e14fe-3ce2-483f-928d-70d943038e1d`
PRD	`96efc332-9895-42f3-8420-5b79bd76c83a`	`61267d7b-68b0-4a7a-83ca-5a48b708c484`

Rule of thumb: a workspace GUID and an item GUID must always be an environment-matched pair. Mixing rows produces EntityNotFound.

Open Items Before promoting

Destination is pinned to dev. 01_create_shortcuts relies on its attached default lakehouse, and git integration does not remap notebook default-lakehouse GUIDs. Promoted as-is, QA/UAT/PRD runs would create the shortcut in the dev lakehouse. Fix: add the Bronze lakehouse GUID to the Variable Library per value set and pass the destination explicitly.
PRD collision and governance decision. PRD's raw is a physical schema of curated per-table shortcuts. The schema shortcut cannot overwrite it, so the first PRD run will fail until the old shortcuts are removed. Removing them also switches PRD from a curated subset to the full mirror (test/scratch tables included). Confirm intent before promoting.
Active value set per workspace. Value sets deploy via git, but the active set is chosen manually in each workspace's Variable Library. One-time step per environment.
Minor: 00_functions uses notebookutils.runtime.context without importing notebookutils (works in a Fabric session, inconsistent elsewhere); the markdown heading in 01_create_shortcuts says "Landing_CRM" but the code creates raw.
Verify QA/UAT current raw mechanism before first promotion.