Consuming Kafka messages from VNET to Fabric LakeHouse - Connection Options with Private Endpoint

[BillGeo]

Hello Community,

I have the following setup and need guidance on the best approach to consume Kafka messages:

Current Infrastructure:

• Kafka server running in a VNET
• Microsoft Fabric LakeHouse
• Requirement to use private endpoints for secure connectivity

I'm exploring the following options and need clarification on their feasibility:

1. Can I use Fabric's Event Stream with private endpoints to consume messages from Kafka?

2. Is it possible to consume Kafka messages using Fabric Spark notebooks with private endpoint connectivity?

3. I know Azure Functions (Premium plan) supports private endpoints - is this the most reliable approach?

Additional Information:

• Need to maintain secure communication through private endpoints
• Looking for the most efficient and reliable method
• Interested in any best practices or architecture recommendations

Has anyone implemented similar scenarios? What would be the recommended approach?

Thank you in advance for your help!

[nilendraFabric · Accepted solution]

Hello @BillGeo 

Apache Kafka can be integrated as a source or destination in Microsoft Fabric’s Eventstream feature. However, this integration currently requires the Kafka cluster to be publicly accessible and does not support private endpoints or connections behind firewalls or virtual networks.
• The default security protocol for Kafka in Fabric is `SASL_PLAINTEXT`, which does not encrypt data transmission. Support for encrypted protocols like SSL is planned but not yet available

https://learn.microsoft.com/en-us/fabric/real-time-hub/add-source-apache-kafka

as discussed above  it is currently not possible to consume Kafka messages using Microsoft Fabric Spark notebooks with private endpoint connectivity. 

Kafka is hosted within a vnet, using Azure Functions (Premium Plan) with private endpoints is an reliable and secure approach to facilitate communication

https://learn.microsoft.com/en-us/azure/azure-functions/functions-create-vnet

https://learn.microsoft.com/en-us/samples/azure/azure-quickstart-templates/function-premium-vnet-int...

Hope this helps

Please accept the answer if you found this helpful and give kudos

[v-prasare]

Hi @BillGeo,

Thanks for reaching out to MS fabric community support. 

@nilendraFabric  Thanks for your prompt response here.

@BillGeo  , as we haven’t heard back from you, we wanted to check in to see if the resolution provided by our community member helps?

If you’re still facing any issues or have additional questions, please don’t hesitate to let us know.

 Looking forward to your feedback!

Thanks,

Prashanth Are

MS Fabric community support.

Did we answer your question? Mark post as a solution, this will help others identify similar issues easily!

If our response(s) assisted you in any way, don't forget to drop me a "Kudos"