Skip to main content
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Calling all Data Engineers! Fabric Data Engineer (Exam DP-700) live sessions are back! Starting October 16th. Sign up.

Reply
yongshao
Contributor

How to access another lakehouse delta table from pyspark when enabling lakehouse schemas feature

When creating a new lakehouse, check "lakehouse schemas" option

create a new pyspark notebook with the lakehouse 

from the notebook, how to access a delta table in another lakehouse (both lakehouses in same workspace)?

 

the following code used to work fine, but when the default lakehouse has "schemas" preview feature, it no longer works

df = spark.table("another_lakehouse_name.table_name")

 

1 ACCEPTED SOLUTION
yongshao
Contributor

4 REPLIES 4
nilendraFabric
Honored Contributor

 

hello @yongshao 

 

To access Delta tables across lakehouses with the schemas feature enabled in Fabric notebooks, you must use fully qualified namespace references.

 

df = spark.table("your_workspace_name.another_lakehouse_name.schema_name.table_name")

 

If this helps please accept the answer 

 

yongshao
Contributor

Hello @nilendraFabric 

df = spark.table("your_workspace_name.another_lakehouse_name.schema_name.table_name")

got the error AnalysisException: [REQUIRES_SINGLE_PART_NAMESPACE] spark_catalog requires a single-part namespace

yongshao
Contributor

Found out the cause - another lakehouse schemas feature should also be turned on

https://community.fabric.microsoft.com/t5/Data-Engineering/Cannot-Cross-workspace-Spark-SQL-queries-...

mathewthewise
New Contributor II

I did some testing.

My final simplified conclusion is: when in Notebook connected to multiple lakehouses (some with some without schema), make the DEFAULT LH one with schema.

Then I can refer to all others using WSNAME.LHNAME.SCHEMA.TABLE.
Note that LHs without schema use dbo as kind of a placeholder for their schema but still seem to need it atm.

Also setting DEFAULT LH to one without schema causes all the troubles.

Helpful resources

Announcements
Users online (27)