NotebookUtils run makes the runned notebook inherit default lakehouse from caller

[mikkeldema]

Hey

I have a problem with using NotebookUtils. My design is that I have two workspaces wilt following utilities:

-BRONZE WORKSPACE-

Notebook: Ochrestrator - default lakehouse is Bronze

Notebook: Process Bronze - default lakehouse is Bronze

Datalake: Bronze

-SILVER WORKSPACE-

Notebook: Process Silver - default lakehouse is Silver

Datalake: Silver

From Orchestrator i make these two notebook runs, but when executing Silver it seems like the default lakehouse is inherited from the calling notebook, because when I print the tables in the default lakehouse from silver it shows the bronze tables.

            # Run Bronze Event Processor

            processEventBronze = notebookutils.notebook.run(

                "Bronze Event Processor",

                300,

                {

                    "useRootDefaultLakehouse": True,

                    "entityType": event['eventType'],

                    "schema": entitySourceTransformed,

                    "entityName": entityName,

                    "entityTableName": bronzeTableName,

                    "entityKey": entityKey,

                    "correlationId": correlationId,

                    "payload": payload

                },

                bronzeTargetWorkspace

            )

            print(processEventBronze)

            # Run Silver Event Processor

            processEventSilver = notebookutils.notebook.run(

                "Silver Event Processor",

                300,

                {

                    "useRootDefaultLakehouse": True,

                    "bronzeTableName": bronzeTableName,

                    "silverTableName": silverTableName,

                    "entityKey": entityKey,

                    "entityId": entityId,

                    "entityType": event['eventType']

                },

                silverTargetWorkspace

            )

Hope someone has an idea of why this is happening when I try to access the silver lakehouse data.

# Load the bronze and silver DataFrames

        bronze_df = spark.read.format("delta").load(f"abfss://{sourceWorkspace}@onelake.dfs.fabric.microsoft.com/{sourceLakehouse}/Tables/{bronzeTableName}").filter(col(entityKey) == entityId)

        silver_df = spark.read.format("delta").load(f"Tables/{silverTableName}").filter(col("source_id") == entityId)

[spencer_sa · Accepted solution]

I didn't know that .run keeps the same default lakehouse, but I can see it makes sense.  The default lakehouse is set at Spark Session start (you can parameterise it though)
.run doesn't create a new spark session, but reuses the old one ("The notebook being referenced runs on the Spark pool of the notebook that calls this function.") from here;
https://learn.microsoft.com/en-us/fabric/data-engineering/notebook-utilities

What we do is explicitly use the ABFSS path rather than default lakehouses.  (we also seperate the Notebooks/Pipelines into a separate workspace completely so have to use ABFSS paths to specify lakehouses.)

So df.read.format('delta').load('abfss://<silverworkspace>@onelake.dfs.fabric.microsoft.com/<silverlakehouse>/Tables/...')