Skip to main content
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Calling all Data Engineers! Fabric Data Engineer (Exam DP-700) live sessions are back! Starting October 16th. Sign up.

Reply
dave_fackler
New Contributor

Error Reading Data from Mirrored Database Using Lakehouse Shortcut

I have a mirrored database set up in a trial Fabric workspace. The data is coming from an Azure SQL database. Works great. I have a lakehouse in the same workspace and I created a shortcut to the mirrored database in the lakehouse. So all of the mirrored tables now show up under the Files list in the lakehouse. I'm trying to do a very simple read of the data using a notebook and this code:

 

# Read Delta table using Spark
spark_df = spark.read.format("delta").load("/[lakehousename]/default/Files/[shortcutname]/[tablename]")
 
When I run this, I get a Spark_System_ABFS_OperationFailed error with this error message:
 
An operation with ADLS Gen2 has failed. This is typically due to a permissions issue. 1. Please ensure that for all ADLS Gen2 resources referenced in the Spark job, that the user running the code has RBAC roles "Storage Blob Data Contributor" on storage accounts the job is expected to read and write from. 2. Check the logs for this Spark application. Inspect the logs for the ADLS Gen2 storage account name that is experiencing this issue.
 
I am an admin in the workspace and have all rights to the mirrored database and the lakehouse (from what I can tell). Is there a permission somewhere that I'm missing? Or is this just not a supported method for reading the mirrored database data from a notebook?
1 ACCEPTED SOLUTION
tayloramy
Contributor

Hi @dave_fackler

This is supported, but you need the correct path and the right Fabric/OneLake permissions.

 

The error text mentions โ€œStorage Blob Data Contributor,โ€ but with Fabric OneLake, access is governed by Fabric item/OneLake permissions rather than Azure Storage RBAC. For mirrored databases, make sure your account has Read all OneLake data on the mirrored database (Workspace ->the mirrored database -> Share). That permission explicitly allows accessing the mirrored data files from Spark/OneLake Explorer. Docs: Share your mirrored database & permissions

Itโ€™s supported to read mirrored data from notebooks

Mirroring lands your source tables as Delta files in OneLake. You can explore/query them with Spark in notebooks by creating a Lakehouse shortcut to the mirrored database/tables and reading from it. Docs: Explore mirrored data with notebooks - Mirroring overview

Use the correct path (copy the ABFS path from the UI)

  1. In your Lakehouse, right-click the shortcut (or a table/file in it) -> Properties -> copy the ABFS path (or โ€œCopy relative path for Sparkโ€ if itโ€™s your default Lakehouse). Docs: Copy ABFS/relative paths
  2. Use that ABFS path in your notebook.
# Example: read a mirrored Delta table through a Lakehouse shortcut (ABFS path)
path = "abfss://<workspaceName>@onelake.dfs.fabric.microsoft.com/<lakehouseName>.lakehouse/Files/<shortcutName>/<schema>/<tableName>"
df = spark.read.format("delta").load(path)
display(df)

ABFS path format example (from Microsoft docs): abfss://myWorkspace@onelake.dfs.fabric.microsoft.com/myLakehouse.lakehouse/Files/. Docs: OneLake ABFSS path example

Tip: ensure the path targets the actual table folder that contains a _delta_log (thatโ€™s the Delta table root). Higher-level folders (schema/database only) wonโ€™t load as Delta tables. Delta read-by-path points to folder with _delta_log Delta logs live under _delta_log

Even easier: add the shortcut under Tables

If the target is Delta, create the shortcut in the Lakehouse Tables section (not Files). Fabric auto-discovers it as a table so you can read it like any managed table via Spark/SQL/semantic model.

# Read a table-shortcut registered under Tables
df = spark.read.table("dbo.MyShortcutTable")            # Spark
# or
spark.sql("SELECT * FROM dbo.MyShortcutTable").show()   # SQL

Docs: OneLake shortcuts (Tables vs Files, auto-recognition) - Lakehouse & Delta tables (auto-discovery incl. shortcuts)

Quick checklist

  • Grant yourself Read all OneLake data on the mirrored database (Share -> permissions). Docs
  • Copy the shortcutโ€™s ABFS path (or relative path) and use it in spark.read.format("delta").load(...). Docs
  • Point to the actual table folder (must contain _delta_log). Docs
  • For Delta tables, prefer the shortcut in Tables so itโ€™s addressable as a table. Docs

If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, please mark this as the solution.

View solution in original post

4 REPLIES 4
tayloramy
Contributor

Hi @dave_fackler

This is supported, but you need the correct path and the right Fabric/OneLake permissions.

 

The error text mentions โ€œStorage Blob Data Contributor,โ€ but with Fabric OneLake, access is governed by Fabric item/OneLake permissions rather than Azure Storage RBAC. For mirrored databases, make sure your account has Read all OneLake data on the mirrored database (Workspace ->the mirrored database -> Share). That permission explicitly allows accessing the mirrored data files from Spark/OneLake Explorer. Docs: Share your mirrored database & permissions

Itโ€™s supported to read mirrored data from notebooks

Mirroring lands your source tables as Delta files in OneLake. You can explore/query them with Spark in notebooks by creating a Lakehouse shortcut to the mirrored database/tables and reading from it. Docs: Explore mirrored data with notebooks - Mirroring overview

Use the correct path (copy the ABFS path from the UI)

  1. In your Lakehouse, right-click the shortcut (or a table/file in it) -> Properties -> copy the ABFS path (or โ€œCopy relative path for Sparkโ€ if itโ€™s your default Lakehouse). Docs: Copy ABFS/relative paths
  2. Use that ABFS path in your notebook.
# Example: read a mirrored Delta table through a Lakehouse shortcut (ABFS path)
path = "abfss://<workspaceName>@onelake.dfs.fabric.microsoft.com/<lakehouseName>.lakehouse/Files/<shortcutName>/<schema>/<tableName>"
df = spark.read.format("delta").load(path)
display(df)

ABFS path format example (from Microsoft docs): abfss://myWorkspace@onelake.dfs.fabric.microsoft.com/myLakehouse.lakehouse/Files/. Docs: OneLake ABFSS path example

Tip: ensure the path targets the actual table folder that contains a _delta_log (thatโ€™s the Delta table root). Higher-level folders (schema/database only) wonโ€™t load as Delta tables. Delta read-by-path points to folder with _delta_log Delta logs live under _delta_log

Even easier: add the shortcut under Tables

If the target is Delta, create the shortcut in the Lakehouse Tables section (not Files). Fabric auto-discovers it as a table so you can read it like any managed table via Spark/SQL/semantic model.

# Read a table-shortcut registered under Tables
df = spark.read.table("dbo.MyShortcutTable")            # Spark
# or
spark.sql("SELECT * FROM dbo.MyShortcutTable").show()   # SQL

Docs: OneLake shortcuts (Tables vs Files, auto-recognition) - Lakehouse & Delta tables (auto-discovery incl. shortcuts)

Quick checklist

  • Grant yourself Read all OneLake data on the mirrored database (Share -> permissions). Docs
  • Copy the shortcutโ€™s ABFS path (or relative path) and use it in spark.read.format("delta").load(...). Docs
  • Point to the actual table folder (must contain _delta_log). Docs
  • For Delta tables, prefer the shortcut in Tables so itโ€™s addressable as a table. Docs

If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, please mark this as the solution.

v-venuppu
Honored Contributor

Hi @dave_fackler ,

Thank you for reaching out to Microsoft Fabric Community.

Thank you @tayloramy for the prompt response.

I wanted to check if you had the opportunity to review the information provided and resolve the issue..?Please let us know if you need any further assistance.We are happy to help.

Thank you.

dave_fackler
New Contributor

Using the abfss path worked. Thank you for the assistance!

Hi @dave_fackler

Glad I could help. If the problem is resolved, kindly mark the post as the solution so that other forum users can easily find it in the future. 

 

If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, please mark this as the solution. 

Helpful resources

Announcements
Users online (9,584)