Automatically end session configuration

[lsabetta]

Hi community,

I have a pipeline with 2 Notebooks with 1 CopyData between both. I turned on the spark high concurrency session for pipelines and choose a name for a session tag in the Notebooks.

The problem is that my CopyData sometimes lasts 3 o 4 minutes to finish and when this happens a new Spark Session is created again, which is not the behavior I am expecting. To avoid that, I set the Automatically end session to 5 minutes but whenever I refresh Fabric, it returns to 1 minute by default.

[rubayatyasmin]

Hi @lsabetta 

Set the workspace level timeout session. refer: Configure high concurrency mode for notebooks in pipelines - Microsoft Fabric | Microsoft Learn

Keep Spark Sessions Alive in Microsoft Fabric - That Fabric Guy

This post might be helpful: https://community.fabric.microsoft.com/t5/Fabric-platform/High-Concurrency-Sessions-in-Fabric-Pipeli... 

[lsabetta]

Hi @rubayatyasmin ,

The problem is that whenever I set the Automatically end session in more than a minute (the default value) and I refresh Fabric, it returns to a minute. 

[v-tejrama · Accepted solution]

Hi @lsabetta ,

Thanks for sharing this. I understand how frustrating it can be when the Automatically end session setting doesn’t stay at the value you configure. What you’re seeing is actually a known limitation in Microsoft Fabric right now. When you update that setting to anything above the default one minute, the change doesn’t persist after a page refresh or workspace reload, which makes it look like it’s reverting back.

This isn’t a configuration mistake on your end it’s related to how the current Fabric interface handles session timeout values for Spark. The team is aware of this behavior, and it’s expected to be addressed in a future update. For now, you can safely rely on enabling High Concurrency mode at the workspace level and use the same session tag across your notebooks. That setup ensures that the same Spark session is reused during your pipeline execution, even if the timeout display resets in the interface.

If your Copy Data activity is taking several minutes and you notice a new session being created despite high concurrency being enabled, it would be worth confirming that both the workspace-level concurrency setting and the session tag are consistently applied. If everything looks correct but the behavior continues, I’d suggest opening a support ticket with Microsoft Fabric Support so they can verify whether your workspace is affected by this persistence issue.

You can also refer to the official documentation on configuring high concurrency mode for notebooks and pipelines here:
https://learn.microsoft.com/en-us/fabric/data-engineering/configure-high-concurrency-session-notebo…

Best Regards,
Tejaswi.
Community Support