Connecting data source directly to event stream

[Nalapriya]

I've an opensource RESTful API to stream real-time data. Now I'm connecting it using custom endpoint as datasource where I'm using Eventhub endpoint creedentials & connecting it. Is Event Hub a mandatory component for connecting to streaming sources or if direct connection to Event Stream is possible. The preferred method is to use all fabric services and not use any additional components.

[Nalapriya]

Thanks Neeko Tang for your suggestions. I'm already using custom endpoint to connect to my event stream but is there any possibility to connect my notebook/pipeline directly to event streams without using Event hub as custom endpoint type.

[xujx · Accepted solution]

Actually you already directly connect to eventstream. The endpoint offered in Eventstream with Eventhub protocol is backboned by Event hub, i.e., Event hub is the one of the key component behind Eventstream. For the protocol, if you don't want to use the Eventhub procotol, you may try the Kafkak protocol as well: Microsoft Fabric event streams overview - Microsoft Fabric | Microsoft Learn

[Anonymous]

Hi @Nalapriya ,

As far as I know, there is currently no support for notebooks/pipelines to connect directly to eventstream. The eventstream destination only contains the following:

But when deploying content from one pipeline stage to another, the copied content can contain eventstream.

Eventstream CI/CD - Git Integration and Deployment Pipeline - Microsoft Fabric | Microsoft Learn

Best Regards,

Neeko Tang

If this post  helps, then please consider Accept it as the solution  to help the other members find it more quickly. 

[Anonymous]

Hi @Nalapriya ,

Using Azure Event Hubs is not strictly mandatory for connecting to streaming sources in Microsoft Fabric, but it is highly recommended and often preferred due to its seamless integration and enhanced capabilities. Event Hubs acts as a robust event ingestion layer, providing low-latency, high-throughput data streaming, and compatibility with protocols like Apache Kafka.

However, if you prefer not to use additional components like Event Hubs, you can explore direct connections to event streams using other supported connectors in Microsoft Fabric. For instance, Fabric supports various source connectors such as Azure IoT Hub, Azure SQL Database Change Data Capture (CDC), PostgreSQL Database CDC, and more. These connectors allow you to bring real-time events into Fabric without necessarily relying on Event Hubs.

Add and manage eventstream sources - Microsoft Fabric | Microsoft Learn

Add a custom endpoint or custom app source to an eventstream - Microsoft Fabric | Microsoft Learn

Best Regards,

Neeko Tang

If this post  helps, then please consider Accept it as the solution  to help the other members find it more quickly.