Running multiple instances of the same Notebook concurrently

[jisaac]

I have a pipeline that needs to runs two Notebook activities (A and B) before running the third Notebook activity (C).

Activities A and B both run the same Notebook but with different parameters: they're doing the same operation on different tables. After both of those tables have been operated on by the Notebooks, it is supposed to run Activity C which runs an operation on both of those tables.

The problem is that, despite being able to give it parameters, the Notebooks error when they are run at the same time. The second instance to run causes the first run to Fail with a strange error. Say the first run is A and the second is B by a few second delay. A fails after B starts. Clicking on A to view the Notebook snapshot pulls up two identical snapshots for B, with no trace of the A run left behind.

I get the same behavior with or without High Concurrency sessions setup.

Is there a way to run the same Notebook twice at the same time with different parameters?

[Anonymous · Accepted solution]

Hi @jisaac ,

I recommend using a single call to a notebook that calls mssparkutils.notebook.runMultiple() in order to execute multiple calls to the notebook in a single Spark session. This approach allows Spark to utilize its ability to efficiently perform concurrent tasks.

For more detailed information, please refer to:

Microsoft Spark Utilities (MSSparkUtils) for Fabric - Microsoft Fabric | Microsoft Learn

Best Regards,
Yang
Community Support Team

If there is any post helps, then please consider Accept it as the solution  to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!

[uselessai_in]

Refer to this blog maybe : https://uselessai.in/how-to-fix-runmultiple-notebook-failure-in-microsoft-fabric-7eb728b8fcb8