Accessing On-Prem API via Python/PySpark in Notebooks

[YevgenyM]

Hi everyone,

We have an on-prem Aruba API endpoint that's accessible through the Power BI gateway.

I'm looking to access this data using Python or PySpark within Microsoft Fabric notebooks.

Has anyone successfully done this or can share guidance on how to set up the connection?

Thanks in advance!
Yevgeny

[v-ssriganesh]

Hello @YevgenyM,
Thank you for reaching out to the Microsoft fabric community forum.

Microsoft Fabric notebooks (Python/PySpark) don’t currently support direct connections to on-premises APIs via the Power BI gateway. A recommended workaround is to use an on-premises script to call the Aruba API and push the data to OneLake or Azure Data Lake Storage, which can then be accessed in Fabric notebooks. Configure the gateway to ensure network access to the API, store credentials securely and use PySpark to process the data in Fabric.

Best regards,
Ganesh Singamshetty.

[YevgenyM]

@v-ssriganesh

Do you know if it is planned in the future to be supported? 

[v-ssriganesh]

Hello @YevgenyM,

Currently, direct support for on-premises APIs through the gateway in Fabric notebooks is not available, and Microsoft has not announced a roadmap or ETA for this specific capability.

I’d recommend keeping an eye on the official Microsoft Fabric release notes: https://learn.microsoft.com/en-us/fabric/fundamentals/whats-new and also submitting this idea in the Fabric Ideas forum: https://community.fabric.microsoft.com/t5/Fabric-Ideas/idb-p/fbc_ideas. The more community votes an idea receives, the more likely it is to be considered by the product team for future updates.

Best regards,
Ganesh Singamshetty

[spaceman127]

Hi @YevgenyM,

another option, if available, would be to use an Azure Function that retrieves the data and then stores it.
Of course, this requires a connection to on-premises.

Best regards

[YevgenyM]

Nope, we don't have full control over azure services in the organization and prefer to stick to fabric services we do have full control over them.

Currently, it is most likely we write the data to on-prem MSSQL

The next step will be incremental copyjob to bronze lakehouse. 

[spaceman127 · Accepted solution]

All right.
This is actually very often the case.

However, we can think of two additional ideas.
Mirroring would be another idea if you already have the data in the SQL server or, as you have already written, incremental copy with CDC to a fabric SQL database.
Both are currently still in preview, but according to my tests they work very well.
Unless it is critical, these would be alternatives.

Best regards 

[YevgenyM]

I forgot about Mirrored MSSQL. 

I will check both options and reply here which one provies better stability and performance. 

@spaceman127, do we need MSSQL enterprise edition to enable mirroring or standard will be enough? 

[spaceman127]

I have not yet been able to test it with the Standard Edition. But that's more because I usually work with the Enterprise Edition.
I will add it to my list tomorrow when I have time to test it.

But without having tested it, I would say yes, it works.

[spaceman127]

I tested it, and it works.
Here is the Microsoft documentation on this topic.

https://learn.microsoft.com/en-us/fabric/mirroring/sql-server

Best regards

[Shahid12523]

You can't directly use the Power BI gateway from Fabric notebooks. Instead:
- Option A: Pull data into a Power BI dataset via the gateway, then query that dataset from Python/PySpark.
- Option B: Set up a secure cloud-accessible API (via Azure Function or reverse proxy) that talks to the on-prem Aruba API.
- Option C: Use Power Automate or a script to push Aruba API data into OneLake or a Lakehouse, then read it from notebooks.

[v-ssriganesh]

Hello @YevgenyM, 

Could you please confirm if your query has been resolved by the provided solutions? This would be helpful for other members who may encounter similar issues.

Thank you for being part of the Microsoft Fabric Community.