Spark reading writing to SQL Database

[P_work]

The following blog post illustrates reading/writing to Fabric Warehouse from Spark notebook.

@https://community.fabric.microsoft.com/t5/Data-Engineering-Community-Blog/Spark-Connector-for-Fabric-Warehouse-Unified-Analytics/ba-p/4611309

Are there examples to read and write to SQL Database(preview) using Spark within Fabric notebook? Say even this (currently failing): 

 df.write.jdbc(...)

[v-lgarikapat · Accepted solution]

Hi @P_work ,

Thanks for reaching out to the Microsoft fabric community forum.

@nilendraFabric  

Thanks for your prompt response

@P_work , I have attached the official document and relevant solved threads related to your use case. I hope you find them helpful

Spark connector for Microsoft Fabric Data Warehouse - Microsoft Fabric | Microsoft Learn

Solved: Re: Fabric Notebook Outputs to Azure SQL Database ... - Microsoft Fabric Community

Solved: Re: Write df to DataWarehouse tables - Microsoft Fabric Community

If this post helped resolve your issue, please consider giving it Kudos and marking it as the Accepted Solution. This not only acknowledges the support provided but also helps other community members find relevant solutions more easily.

We appreciate your engagement and thank you for being an active part of the community.

Best regards,
LakshmiNarayana.

[nilendraFabric]

Hi @P_work 

try this

df.write.mode("append").option("mergeSchema", "true").synapsesql("DW1.Core.CustomerData")

[P_work]

I believe the main issue was this:

notebookutils.credentials.getToken("pbi")

The examples provided were of little use.

Also, ensuring using runtime 1.3 appeared to help.

[v-lgarikapat · Accepted solution]

Hi @P_work ,

Thanks for reaching out to the Microsoft fabric community forum.

@nilendraFabric  

Thanks for your prompt response

@P_work , I have attached the official document and relevant solved threads related to your use case. I hope you find them helpful

Spark connector for Microsoft Fabric Data Warehouse - Microsoft Fabric | Microsoft Learn

Solved: Re: Fabric Notebook Outputs to Azure SQL Database ... - Microsoft Fabric Community

Solved: Re: Write df to DataWarehouse tables - Microsoft Fabric Community

If this post helped resolve your issue, please consider giving it Kudos and marking it as the Accepted Solution. This not only acknowledges the support provided but also helps other community members find relevant solutions more easily.

We appreciate your engagement and thank you for being an active part of the community.

Best regards,
LakshmiNarayana.

[v-lgarikapat]

Hi @P_work ,

If your issue has been resolved, please consider marking the most helpful reply as the accepted solution. This helps other community members who may encounter the same issue to find answers more efficiently.

If you're still facing challenges, feel free to let us know we’ll be glad to assist you further.

Looking forward to your response.

Best regards,
LakshmiNarayana.

[v-lgarikapat]

Hi @P_work ,

If your issue has been resolved, please consider marking the most helpful reply as the accepted solution. This helps other community members who may encounter the same issue to find answers more efficiently.

If you're still facing challenges, feel free to let us know we’ll be glad to assist you further.

Looking forward to your response.

Best regards,
LakshmiNarayana.

[v-lgarikapat]

Hi @P_work ,

As we haven't heard back from you, we are closing this thread. If you are still experiencing the issue, please feel free to create a new thread we’ll be happy to assist you further.

Thank you for your patience and support.

If you found our response helpful, please mark it as Accepted Solution so others with similar queries can find it easily.

Best Regards,

Lakshmi Narayana

[g3kuser · Accepted solution]

Here is how you connect to SQL DB from Fabric Notebook using SPN credentials

host = ''

database = ''

sql_query = ''

jdbc_url = f'jdbc:sqlserver://{host};database={database};authentication=ActiveDirectoryServicePrincipal;encrypt=true;trustServerCertificate=true;'

db_properties = {

"user": f"{client_id}",

"password": f"{client_secret}",

"driver": "com.microsoft.sqlserver.jdbc.SQLServerDriver"

}

#For Reading

spark.read.jdbc(url=jdbc_url, table=f"({sql_query}) AS query_alias", properties=db_properties)
#For Writing data

df.write.jdbc(url=jdbc_url, table=table_name, mode=mode, properties=db_properties)