| 1 |
Infrastructure |
Create Azure DevOps repo: D365-Fabric-Gold |
Create a new Azure DevOps Git repository named D365-Fabric-Gold to hold all artefacts for the new shared Gold Fabric workspace. |
- Repo exists in turntownd365 / D365 BI - Kerv DevOps project
- Default branch is main
|
- Create repo in turntownd365 / D365 BI - Kerv DevOps project
- Set default branch to main
|
| 2 |
Infrastructure |
Provision Gold Fabric workspace — Dev, UAT, and Prod |
Create three Fabric workspaces for the Gold layer: Dev, UAT, and Prod. These sit between Silver and the Mart workspaces in the data flow. |
- Three workspaces created: D365-Fabric-Dev-Gold, D365-Fabric-UAT-Gold, D365-Fabric-Prod-Gold
- Appropriate capacity assigned to each
- Workspace Identity has PBI Access
- Workspaces visible in Fabric portal
|
- Create three workspaces: D365-Fabric-Dev-Gold, D365-Fabric-UAT-Gold, D365-Fabric-Prod-Gold
- Assign appropriate capacity to each
- Grant Workspace Identity PBI Access
- Verify workspaces are visible in Fabric portal
|
| 3 |
Infrastructure |
Create Gold Lakehouse in Dev workspace |
Create a Fabric Lakehouse named Gold in the Dev workspace. UAT and Prod are provisioned via CI/CD pipeline deployment from the repo definition. |
- Gold Lakehouse exists in D365-Fabric-Dev-Gold workspace
- SQL analytics endpoint accessible
- Lakehouse definition committed to D365-Fabric-Gold repo
|
- Create Gold Lakehouse in D365-Fabric-Dev-Gold workspace
- Verify SQL analytics endpoint is accessible
- Commit Lakehouse definition to D365-Fabric-Gold repo
|
| 4 |
Infrastructure |
Set up VariableLibrary in D365-Fabric-Gold repo |
Add a VariableLibrary item to D365-Fabric-Gold with value sets for Dev, UAT, and Prod, following the same pattern as existing repos. |
- VariableLibrary committed to repo
- Value sets exist for Dev, UAT, and Prod
- Variables include Gold Lakehouse workspace ID and item ID per environment
- Variables include Management Accounts workspace ID per environment (required by nb_0003_refreshArtifacts)
- Naming conventions match existing repos
|
- Commit VariableLibrary to repo
- Add value sets for Dev, UAT, and Prod
- Include Gold Lakehouse workspace ID and item ID per environment
- Include Management Accounts workspace ID per environment (required by nb_0003_refreshArtifacts)
- Follow naming conventions from existing repos
|
| 5 |
Infrastructure |
Set up azure-pipelines.yaml and CI/CD scripts in D365-Fabric-Gold |
Configure the Azure DevOps CI/CD pipeline for D365-Fabric-Gold, using the deployment script pattern from Bronze and Gold-ManagementAccounts repos. |
- azure-pipelines.yaml committed and pipeline registered in DevOps
- Pipeline deploys successfully to Dev on merge to main
- Auth scripts (SPN + user) in place
- Environment gates match existing repos
- Repo connected (Git sync) to D365-Fabric-Dev-Gold, D365-Fabric-UAT-Gold, and D365-Fabric-Prod-Gold workspaces
- Gold Lakehouse deploys successfully to UAT and Prod via pipeline
- Pipeline runs successfully end-to-end in UAT and Prod after initial deployment
|
- Commit azure-pipelines.yaml and register pipeline in DevOps
- Confirm pipeline deploys successfully to Dev on merge to main
- Add auth scripts (SPN + user)
- Match environment gates to existing repos
- Connect repo (Git sync) to D365-Fabric-Dev-Gold, D365-Fabric-UAT-Gold, and D365-Fabric-Prod-Gold workspaces
- Deploy Gold Lakehouse to UAT and Prod via pipeline
- Run pipeline end-to-end in UAT and Prod after initial deployment
|
| 6 |
Migration |
Migrate gold_config notebook to D365-Fabric-Gold |
Move the gold_config notebook from D365-Fabric-Gold-ManagementAccounts to D365-Fabric-Gold. gold_config is a config-provider notebook: it contains only table schema definitions (column names, data types, quality checks, source mappings) stored as a Python list, and exits with the config as JSON for other notebooks to consume. It has no Lakehouse or workspace references — no content changes are required on migration. |
- Notebook committed to D365-Fabric-Gold under Notebooks/Config/
- No content changes required — notebook contains only schema definitions with no Lakehouse or workspace references
- Notebook runs without error in Dev Gold workspace (verify mssparkutils.notebook.exit completes successfully)
- Old notebook removed from D365-Fabric-Gold-ManagementAccounts (PBI 17)
|
- Copy notebook to D365-Fabric-Gold under Notebooks/Config/ — no content changes needed
- Run notebook in Dev Gold workspace and confirm mssparkutils.notebook.exit completes without error
- Remove old notebook from D365-Fabric-Gold-ManagementAccounts (PBI 17)
|
| 7 |
Migration |
Migrate utility notebooks to D365-Fabric-Gold |
Move nb_0001_functions_gold and nb_0002_government from D365-Fabric-Gold-ManagementAccounts to D365-Fabric-Gold. nb_0003_refreshArtifacts is handled separately in PBI 8. |
- Both notebooks in D365-Fabric-Gold under Notebooks/Utility/
- Notebooks deploy and run successfully in Dev Gold workspace
- No references to old workspace remain in notebook content
- getSource() fix: The current check
if "Gold-" in current_layer in nb_0001_functions_gold will not match the new workspace name (D365-Fabric-Dev-Gold strips to "Gold" via regex, not "Gold-Something"). Update the condition to also handle the plain "Gold" case — e.g. if current_layer == "Gold" or "Gold-" in current_layer
|
- Move both notebooks to D365-Fabric-Gold under Notebooks/Utility/
- Deploy and run both notebooks in Dev Gold workspace
- Remove any references to the old workspace from notebook content
- getSource() fix: Update the condition in nb_0001_functions_gold from
if "Gold-" in current_layer to also handle the plain "Gold" case — e.g. if current_layer == "Gold" or "Gold-" in current_layer
|
| 8 |
Migration |
Move nb_0003_refreshArtifacts to D365-Fabric-Gold and update to target Management Accounts workspace |
Move nb_0003_refreshArtifacts to D365-Fabric-Gold so it sits alongside nb_0001_functions_gold (resolving the %run dependency). Update it to explicitly target the Management Accounts workspace when listing datasets and triggering refreshes, as it will now run from Gold not Management Accounts. Called from the Gold Orchestration pipeline as the final step after all notebooks complete. |
- Notebook moved to D365-Fabric-Gold under Notebooks/Utility/
- %run nb_0001_functions_gold continues to work — both notebooks are now in Gold
- list_datasets() updated to pass Management Accounts workspace ID explicitly
- refreshModel updated to accept and use the Management Accounts workspace ID rather than the current notebook workspace
- Management Accounts workspace ID sourced from VariableLibrary (Dev, UAT, Prod values)
- Notebook added as final step in Gold Orchestration pipeline (PBI 12)
- Notebook runs successfully in Dev and refreshes all semantic models in the Management Accounts workspace
- Notebook removed from D365-Fabric-Gold-ManagementAccounts
|
- Move notebook to D365-Fabric-Gold under Notebooks/Utility/
- Verify %run nb_0001_functions_gold continues to work with both notebooks in Gold
- Update list_datasets() to pass Management Accounts workspace ID explicitly
- Update refreshModel to accept and use the Management Accounts workspace ID rather than the current notebook workspace
- Source Management Accounts workspace ID from VariableLibrary (Dev, UAT, Prod values)
- Add notebook as final step in Gold Orchestration pipeline (PBI 12)
- Run notebook in Dev and confirm it refreshes all semantic models in the Management Accounts workspace
- Remove notebook from D365-Fabric-Gold-ManagementAccounts
|
| 9 |
Migration |
Migrate and rename all 16 dim notebooks to D365-Fabric-Gold |
Move all 16 Management Accounts dimension notebooks to D365-Fabric-Gold, renaming them to generic gold names (no _managementaccounts suffix) and updating write targets to the gold.* schema. |
- All 16 notebooks renamed and committed to D365-Fabric-Gold under Notebooks/Dimensions/
- Silver Lakehouse remains read source; Gold Lakehouse is write target
- All 16 run successfully in Dev and write to the correct gold.* tables:
-
| Notebook | Output Table |
| nb_dim_date | gold.dim_accountingdate |
| nb_dim_account | gold.dim_chartofaccount |
| nb_dim_company | gold.dim_company |
| nb_dim_cost_center_hierarchy | gold.dim_costcenter |
| nb_dim_financial_counterpart | gold.dim_counterpart |
| nb_dim_financial_department | gold.dim_department |
| nb_dim_financial_lead_project | gold.dim_leadproject |
| nb_dim_financial_office | gold.dim_office |
| nb_dim_financial_project | gold.dim_project |
| nb_dim_customer | gold.dim_customer |
| nb_dim_general_journal | gold.dim_generaljournal |
| nb_dim_security | gold.dim_security |
| nb_dim_supplier | gold.dim_supplier |
| nb_dim_ultimate_customer | gold.dim_ultimatecustomer |
| nb_dim_worker | gold.dim_worker |
| nb_dim_posting_type | gold.dim_postingtype |
- Row counts match pre-migration baseline for all 16 dim tables
|
- Rename all 16 notebooks and commit to D365-Fabric-Gold under Notebooks/Dimensions/
- Confirm Silver Lakehouse as read source and Gold Lakehouse as write target
- Run all 16 in Dev and verify each writes to the correct gold.* table (see AC for full list)
- Validate row counts against pre-migration baseline for all 16 dim tables
|
| 10 |
Migration |
Migrate and rename 3 fact notebooks to D365-Fabric-Gold |
Move nb_factgeneralledger_managementaccounts, nb_factgeneralledgerbudget_managementaccounts, and nb_factheadcount_managementaccounts from D365-Fabric-Gold-ManagementAccounts to D365-Fabric-Gold, renaming to generic gold names. |
- All 3 notebooks renamed and committed to D365-Fabric-Gold under Notebooks/Facts/
- Gold Lakehouse is write target; Silver remains read source
- All 3 run successfully in Dev and write to the correct gold.* tables:
-
| Notebook | Output Table |
| nb_fact_gl_transaction | gold.fct_generalledger |
| nb_fact_budget | gold.fct_generalledgerbudget |
| nb_fact_headcount | gold.fct_headcount |
- Row counts match pre-migration baseline
|
- Rename all 3 notebooks and commit to D365-Fabric-Gold under Notebooks/Facts/
- Confirm Gold Lakehouse as write target and Silver as read source
- Run all 3 in Dev and verify each writes to the correct gold.* table (see AC for full list)
- Validate row counts against pre-migration baseline
|
| 11 |
Migration |
Update all migrated notebooks to write to new Gold Lakehouse |
Review every migrated notebook and confirm all write operations target the new Gold Lakehouse. Remove any hardcoded references to the old ManagementAccounts Lakehouse. |
- Known hardcoded
management_accounts.* schema references fixed in active notebook code:
-
nb_dim_ultimate_customer (was nb_dimultimatecustomer_managementaccounts) — reads
management_accounts.dim_customer (line 31) and writes management_accounts.dim_ultimatecustomer (line 41)
nb_fact_gl_transaction (was nb_factgeneralledger_managementaccounts) — self-references own output table management_accounts.fct_generalledger (line 888), likely for incremental merge pattern
- All other notebooks scanned and confirmed free of active hardcoded schema references (commented-out references acceptable)
- Notebook metadata GUIDs are handled by CI/CD deployment — no manual action required
- PR review completed before merge
|
- Fix hardcoded
management_accounts.* schema references in active notebook code:
-
nb_dim_ultimate_customer — fix read of
management_accounts.dim_customer (line 31) and write of management_accounts.dim_ultimatecustomer (line 41)
nb_fact_gl_transaction — fix self-reference to management_accounts.fct_generalledger (line 888)
- Scan all other notebooks and confirm no active hardcoded schema references remain (commented-out references are acceptable)
- Note: notebook metadata GUIDs are handled by CI/CD deployment — no manual action required
- Complete PR review before merge
|
| 12 |
Orchestration |
Create Gold Orchestration pipeline in new Gold workspace |
Create a Data Pipeline in the Gold workspace that runs all dim and fact notebooks in the correct dependency order. |
- Pipeline exists in Gold workspace (Dev, UAT, Prod)
- Pipeline committed to D365-Fabric-Gold repo
- Dims run before facts
- nb_0003_refreshArtifacts runs as the final step after all fact notebooks complete
- Pipeline completes successfully in Dev
- Failures halt the pipeline correctly
|
- Create pipeline in Gold workspace (Dev, UAT, Prod)
- Commit pipeline to D365-Fabric-Gold repo
- Order dims to run before facts
- Add nb_0003_refreshArtifacts as the final step after all fact notebooks complete
- Run pipeline in Dev and confirm it completes successfully
- Verify failures halt the pipeline correctly
|
| 13 |
Orchestration |
Update Full Orchestration pipeline: Bronze → Silver → Gold → Mart |
Update the Full Orchestration pipeline to call the Gold workspace pipeline after Silver and the Management Accounts Mart pipeline after Gold. |
- Sequence is Bronze → Silver → Gold → Management Accounts Mart
- Pipeline tested end-to-end in Dev
- Failure in any stage halts downstream stages
|
- Update Full Orchestration pipeline sequence to Bronze → Silver → Gold → Management Accounts Mart
- Test pipeline end-to-end in Dev
- Verify failure in any stage halts downstream stages
|
| 14 |
Mart Cleanup |
Create shortcuts_config.json with Gold-to-MA reverse mapping |
Create a JSON config file that defines the source workspace and Lakehouse (by VariableLibrary variable name) and maps each Gold table to its target shortcut name. Using variable names rather than hardcoded IDs makes the config reusable for any source workspace. |
- shortcuts_config.json committed to D365-Fabric-Gold-ManagementAccounts repo under Config/
- File includes source_workspace_variable and source_lakehouse_variable fields referencing the VariableLibrary variable names (e.g.
Gold_Workspace_ID, Gold_Lakehouse_ID)
- File contains one mapping entry per table with source_table and shortcut_name fields:
-
| source_table (Gold) | shortcut_name (current MA table name) |
| gold.dim_accountingdate | management_accounts.dim_accountingdate |
| gold.dim_chartofaccount | management_accounts.dim_chartofaccount |
| gold.dim_company | management_accounts.dim_company |
| gold.dim_costcenter | management_accounts.dim_costcenter |
| gold.dim_counterpart | management_accounts.dim_counterpart |
| gold.dim_customer | management_accounts.dim_customer |
| gold.dim_department | management_accounts.dim_department |
| gold.dim_generaljournal | management_accounts.dim_generaljournal |
| gold.dim_leadproject | management_accounts.dim_leadproject |
| gold.dim_office | management_accounts.dim_office |
| gold.dim_postingtype | management_accounts.dim_postingtype |
| gold.dim_project | management_accounts.dim_project |
| gold.dim_security | management_accounts.dim_security |
| gold.dim_supplier | management_accounts.dim_supplier |
| gold.dim_ultimatecustomer | management_accounts.dim_ultimatecustomer |
| gold.dim_worker | management_accounts.dim_worker |
| gold.fct_generalledger | management_accounts.fct_generalledger |
| gold.fct_generalledgerbudget | management_accounts.fct_generalledgerbudget |
| gold.fct_headcount | management_accounts.fct_headcount |
|
- Create and commit shortcuts_config.json to D365-Fabric-Gold-ManagementAccounts repo under Config/
- Add source_workspace_variable and source_lakehouse_variable fields referencing the VariableLibrary variable names (e.g.
Gold_Workspace_ID, Gold_Lakehouse_ID)
- Add one mapping entry per table with source_table and shortcut_name fields (see AC for full mapping)
|
| 15 |
Mart Cleanup |
Create Lakehouse shortcuts in Management Accounts Lakehouse pointing to Gold Lakehouse tables |
Run nb_create_shortcuts in the Management Accounts workspace to create all 19 shortcuts using the reverse mapping in shortcuts_config.json. Shortcuts are named with the original Management Accounts table names so the semantic model requires no changes. |
- All shortcuts created in Management Accounts Lakehouse using current table names (e.g. management_accounts.dim_accountingdate, management_accounts.fct_generalledger)
- Each shortcut points to the correct Gold Lakehouse table
- Data visible via SQL endpoint for each shortcut
- Shortcuts committed to repo via shortcuts.metadata.json
- A Fabric pipeline created to run nb_create_shortcuts, committed to D365-Fabric-Gold-ManagementAccounts repo — triggered manually after deployment to UAT and Prod to create or refresh shortcuts
|
- Run nb_create_shortcuts to create all shortcuts in Management Accounts Lakehouse using current table names (e.g. management_accounts.dim_accountingdate, management_accounts.fct_generalledger)
- Verify each shortcut points to the correct Gold Lakehouse table
- Confirm data is visible via SQL endpoint for each shortcut
- Commit shortcuts to repo via shortcuts.metadata.json
- Create a Fabric pipeline to run nb_create_shortcuts, commit to D365-Fabric-Gold-ManagementAccounts repo, and trigger manually after deployment to UAT and Prod
|
| 16 |
Mart Cleanup |
Validate Management Accounts semantic model against shortcuts |
Confirm the Management Accounts semantic model resolves correctly against the shortcut tables. Because shortcuts use the original MA names, no model changes should be required. |
- All tables resolve without error
- Model refreshes successfully in Dev
- No broken relationships or missing columns
- Report visuals load correctly on main report pages
- Confirm management_accounts.dim_etlproperties is not referenced in the model — if it is, remove it before PBI 19 drops the table
|
- Verify all tables resolve without error
- Refresh model in Dev and confirm success
- Check for broken relationships or missing columns
- Load main report pages and verify visuals render correctly
- Confirm management_accounts.dim_etlproperties is not referenced in the model — if it is, remove it before PBI 19 drops the table
|
| 17 |
Mart Cleanup |
Remove Gold notebooks and Gold Lakehouse from D365-Fabric-Gold-ManagementAccounts |
Delete the dim notebooks, fact notebooks, gold_config, utility notebooks, and Gold Lakehouse write artefacts from D365-Fabric-Gold-ManagementAccounts now that they live in D365-Fabric-Gold. |
- Notebooks folder contains only Mart-specific items
- No Gold write notebooks deploy into Management Accounts workspace
- Change merged via PR
- No orphaned artefacts remain in Management Accounts workspace
|
- Delete dim notebooks, fact notebooks, gold_config, utility notebooks, and Gold Lakehouse write artefacts from D365-Fabric-Gold-ManagementAccounts
- Confirm Notebooks folder contains only Mart-specific items
- Merge change via PR
- Verify no orphaned artefacts remain in Management Accounts workspace
|
| 18 |
Mart Cleanup |
Update D365-Fabric-Gold-ManagementAccounts CI/CD to remove Gold deployments |
Update azure-pipelines.yaml in D365-Fabric-Gold-ManagementAccounts to remove Gold notebooks and config from the deployment. As Fabric deployment pipelines are additive, notebooks already deployed to Dev, UAT, and Prod must also be manually deleted from each workspace. |
- Gold notebook and config paths removed from azure-pipelines.yaml deployment steps
- Pipeline still deploys semantic model, reports, Mart pipelines, and shortcuts metadata
- Pipeline runs cleanly after change in Dev, UAT, and Prod
- Gold notebooks and config manually deleted from Dev, UAT, and Prod Management Accounts workspaces in Fabric portal
- Gold notebook and config folders removed from D365-Fabric-Gold-ManagementAccounts repo via PR (or confirmed already removed by PBI 18)
- Gold notebooks confirmed absent from all three environments and the DevOps repo before closing
- PR reviewed and merged
|
- Remove Gold notebook and config paths from azure-pipelines.yaml deployment steps
- Verify pipeline still deploys semantic model, reports, Mart pipelines, and shortcuts metadata
- Run pipeline in Dev, UAT, and Prod and confirm it runs cleanly
- Manually delete Gold notebooks and config from Dev, UAT, and Prod Management Accounts workspaces in Fabric portal
- Remove Gold notebook and config folders from D365-Fabric-Gold-ManagementAccounts repo via PR (or confirm already removed by PBI 18)
- Confirm Gold notebooks are absent from all three environments and the DevOps repo before closing
- Review and merge PR
|
| 19 |
Mart Cleanup |
Move dim_etlproperties from management_accounts schema to audit schema |
dim_etlproperties is an internal refresh-tracking table, not a business dim. Move it to the audit schema to sit alongside audit.qualitychecks and remove it from the management_accounts schema. |
- Table recreated as audit.dim_etlproperties with the same structure and data
- management_accounts.dim_etlproperties dropped
- Any notebook references to management_accounts.dim_etlproperties updated to audit.dim_etlproperties
- Semantic model confirmed unaffected (table should not be in the model)
|
- Recreate table as audit.dim_etlproperties with the same structure and data
- Drop management_accounts.dim_etlproperties
- Update any notebook references from management_accounts.dim_etlproperties to audit.dim_etlproperties
- Confirm semantic model is unaffected (table should not be in the model)
|
| 20 |
Operations |
Set up Fabric Deployment Pipeline for Gold workspace |
Create the Fabric Deployment Pipeline connecting the Gold Dev, UAT, and Prod workspaces. This promotes all workspace items — Gold Lakehouse, notebooks, and Gold Orchestration Data Pipeline — through environments. Must be in place before DevOps can trigger Gold promotions. |
- Fabric Deployment Pipeline created with three stages: D365-Fabric-Dev-Gold → D365-Fabric-UAT-Gold → D365-Fabric-Prod-Gold
- All workspace items visible in the pipeline: Gold Lakehouse, dim and fact notebooks, and Gold Orchestration Data Pipeline
- Deployment rules set for each workspace item to handle environment-specific configuration (e.g. Lakehouse connections)
- Successful deployment from Dev to UAT
- Successful deployment from UAT to Prod
|
- Create Fabric Deployment Pipeline with three stages: D365-Fabric-Dev-Gold → D365-Fabric-UAT-Gold → D365-Fabric-Prod-Gold
- Verify all workspace items are visible: Gold Lakehouse, dim and fact notebooks, and Gold Orchestration Data Pipeline
- Set deployment rules for each workspace item to handle environment-specific configuration (e.g. Lakehouse connections)
- Deploy from Dev to UAT and confirm success
- Deploy from UAT to Prod and confirm success
|
| 21 |
Validation |
End-to-end validation — Management Accounts reports match pre-migration figures |
Run a full refresh via the new pipeline sequence and validate that all Management Accounts report figures match the pre-migration baseline for at least two closed fiscal periods. |
- Full pipeline run completes without errors (Bronze → Silver → Gold → Mart)
- Key measures (GL amounts, budget amounts, headcount) match pre-migration values
- Any variances investigated and signed off
- Stakeholder sign-off obtained before Prod deployment
|
- Run full pipeline (Bronze → Silver → Gold → Mart) and confirm it completes without errors
- Validate key measures (GL amounts, budget amounts, headcount) match pre-migration values
- Investigate and sign off any variances
- Obtain stakeholder sign-off before Prod deployment
|
