Fabric notebook when run via ADF pipeline trigger run issue

[ati_puri]

We are invoking our fabric notebook from ADF pipeline(web activity), when it is run manually the notebook writes data to the bronze table, however, when it is triggered/scheduled it runs successfully but does not write a single line to the table. Anyone faced this issue? Parameters etc are all correct.

[apturlov]

Hi @ati_puri, in your scenario you invoke a Fabric Notebook using Fabric's REST API through external ADF pipeline using a Web activity. Key difference here is that the Notebook is invoked via Fabric's REST API compared to a manual Notebook run via Fabric's UI. I don't have hands-on experienc like that but from your description it looks like the Notebook does not have a Lakehouse destination set when invoked via API. I assume you know that a Notebook in Fabric must have a default Lakehouse to run successfully. Can you check what Lakehouse the Notebook is connected to when invoked via REST API? Also I would want to eliminate potential impact of the ADF's Web activity and try and trigger a Notebook using a REST API directly without ADF from a REST Client, for example.

[BhaveshPatel]

Hi @ati_puri 

You should check why it is running on the first instance in Delta Lake ( Bronze tables ). You need to share some screenshots. 

[v-priyankata]

Hi @ati_puri 

Thank you for reaching out to the Microsoft Fabric Forum Community.

@apturlov @BhaveshPatel Thanks you so much for your inputs.

The users' suggestions are quite helpful. Have you been able to try the super user's recommendations? If possible, could you share the points they mentioned so we can better understand and address the issue?

If I misunderstand your needs or you still have problems on it, please feel free to let us know.  

Thanks.

[v-priyankata]

Hi @ati_puri 
Have you had a chance to try the super user’s guidance? If you could share the points they mentioned, it would help us understand the issue better.

If I misunderstand your needs or you still have problems on it, please feel free to let us know.  

Thanks.

[v-priyankata]

Hi @ati_puri 

Hope everything’s going smoothly on your end. I wanted to check if the issue got sorted. if you have any other issues please reach community.

[ati_puri · Accepted solution]

Thanks @apturlov and @BhaveshPatel  for the inputs. The underlying issue was related to the DDL creation issue of one of the staging tables. However, as we used webactivity to invoke fabric notebooks, we could see some timeout issues, due to presence of parent and multiple child notebooks. We did increase the timeout in webactivity in ADF as well as in the child notebook call. This helped us to run the child notebooks completely and successfully. The ideal way will always be utilizing fabric pipelines and its notebooks so that the compute and connection strings remain in one environment to avoid the integreations issues. As webactivity coming out of ADF can have fabric notebook API call limitations which is be avoided in fabric.

Hope this helps other users who are working on the similar objectives.

[v-priyankata]

Hi @ati_puri 
It's great to hear that, and thank you for sharing the solution it will definitely be helpful for others facing the same issue. Please reach out to the community if you have any other questions; we're happy to assist you.