Issues running Spark Notebook

[ronan_b6]

Hi I'm runing spark notebooks on Fabric trial, it's been working great and no time out issues, or severe dalys in sessions starting.

But today, notebooks do not start the spark session, returning this type of error:

"Livy session has failed. Error code: SparkCoreError/SessionDidNotEnterIdle. SessionInfo.State from SparkCore is Error: Session did not enter idle state after 10 minutes. Source: System."

I see from the forum that others have experience similar issues, with severe delays in functionality.

Can anyone shed more light on this?

Thanks,

[yongshao]

Face the same issue Today in West US (California)

for many hours, notebook cannot be run due to the following error:

SparkCoreError/SessionDidNotEnterIdle: Livy session has failed. Error code: SparkCoreError/SessionDidNotEnterIdle. SessionInfo.State from SparkCore is Error: Session did not enter idle state after 10 minutes

[yongshao]

Doubt if we could use Fabric in production????

[Anonymous]

Hi all,

We have escalated this issue to the team, who are investigating the cause of the error and working hard to try and fix the problem. If any progress is made behind the scenes, I'll update you all here.

Best Regards,

Ada Wang

[mrobache]

Same issue in central europe for 24 hours at least. Agree, what if it happen when the system is in production ?

[user45551]

The issue is still up there. Is there any update regarding this?

[ed-freeman]

Also hitting this same error right now in North Europe. It has not been fixed, contrary to screenshots suggesting otherwise.

[danieletiles]

Hi,

still having the issue in North Europe. What is incredible is that there is no acknowledge from support.fabric.microsoft.com!

[Sarien]

Exactly, this is a very important part of their flagship product in general availability and there is pretty much zero support for two days when it is completely broken.

[michaelmll]

Same issue. Can't start spark session... (North Europe)

[PhantomThread]

Had this yesterday. Intermittent.

Got it today and I'm 100% blocked on any Synapse work.

Way to go Fabric, you've put me late on delivering work AGAIN

[PhantomThread]

What is the point of this page in circumstances like this..!

[abm_chris]

At this stage, I'm certain that's just a static table and not connected to any data.

[jonocrotz]

Response from Microsoft relating to my P1 ticket for the issue. They say its mitigated but still have an issue.

Really struggling with Microsoft's shocking support.

North Europe region. Still not working as of 20 mins ago

STATUS:
Mitigated

SUMMARY OF IMPACT:
Our production team has applied the mitigation steps, and the error trend came down around 7/18 1:30 AM UTC. 
We will be continuously monitoring the trend and act accordingly if needed. We'll send an update once we know more. 

In the meanwhile, please contact us if you still encounter the same issue.

[Adlane_Kadri]

We have been experiencing the same problem since Monday. We struggle to launch a notebook, succeeding only once out of 20 attempts, even with multiple retries. We are really having issues with Fabric Notebook over the past two days (France).

[rmoraes1987]

I am having the same issue in Brazil

[kirah2128]

I'm having the same error too , from middle east. 

[ronan_b6]

Issue seems to be resolved for Central Canada.
Hope it's working for you guys too!

[jonocrotz]

Anyone still experiencing this issue in North Europe?

[JFTxJ]

Hey everyone,

I just had some resolution on this issue by enabling runtime 1.3 (Preview) on my workspace.  I have restarted a pipeline with hundreds of Notebook executions and they all seem to be running smoothly with the newer (Preview) runtime.

Of course this could just be a convenient glitch, so I will comment back tomorrow on the progress, but it looks promising to me as I was able to reproduce the issue with a manual execution of a notebook, and that has also been working since I made the config change.

If you are able to reproduce the issue while running Notebooks manually, I recommend experimenting with a custom Notebook environment set to runtime 1.3 and see if you get any better results.

I will follow up tomorrow with a status update.

Thank you

[PhantomThread]

I've just tried the 1.3 change and got a Pandas error. Further reading this morning suggests I'm hitting memory issues based on the size of the data frame I'm outputting... Have edited this post as my code is the issue here, not Fabric 🙂

[PhantomThread]

So just to reply to myself... I've had other pipelines calling notebooks over the weekend - default is now 1.3 - and they've worked ok (and have calls to pandas to output files).

I've not yet reverted to 1.2 to test it, but can confirm that 1.3 is working.

Looks like something strange is going on with this particular notebook...

[JFTxJ]

FYI:  I got an update from MS last friday that this issue should be resolved.   So we should be able to revert back to runtime 1.2.

I have not tested this yet, but just passing on the message.

[kirah2128]

i also have the same solution applied and data pipeline now works successfully on 1.3 spark.

[Anonymous]

Hi @JFTxJ ,

Thank you for sharing and we really look forward to your status updates.

Hi all,

The incident is still under investigation and the team is checking for changes or updates that may have caused the issue. Please be patient.

Best Regards,

Ada Wang

[Anonymous]

Hi all,

We have reported this issue and submitted it to the product team.

They have been aware of the issue and fixed it. Please check it later.

Best Regards,

Ada Wang

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[vdgarcia]

Hi I have the same issue today(30-JUL-2024), in US South Central Region ("Notebook execution failed at Notebook service with http status code - '200', please check the Run logs on Notebook, additional details - 'Error name - Exception, Error value - Failed to create Livy session for executing notebook.")

[daniele_tiles]

Hi, it might be related to the big issues happening world wide on Azure netowrok infrastructure:

https://azure.status.microsoft/it-it/status

[vdgarcia]

Hi Again,

Today(31-JUL-2024), I have the same issue("SparkCoreError/SessionDidNotEnterIdle: Livy session has failed. Error code: SparkCoreError/SessionDidNotEnterIdle. SessionInfo.State from SparkCore is Error: Session did not enter idle state after 10 minutes. Source: System.").

I don't know if this behavior is related to my size capacity(F2), but at run time the capability does not execute any other processes.

Thank you.

[kirah2128]

I'm having the same error too. Hopefully they will fix this issue asap. 

[vdgarcia]

Hi, This problem persists using the stable version of Spark (1.2), however since I adjusted the Spark version to 1.3v (preview) the Notebooks now run without problems.

I sincerely hope that the stable version 1.2 really works stably and we can keep it and run our production processes with confidence.

[ronan_b6 · Accepted solution]

This issue persisted for a week. I updated to runtime 1.3 at the time and it had no effect on the issue. MS stated at the time it was a resource allocation issue. Since 07-18-2024 I have not experienced this issue with my pipelines. For now it seems MS has resolved the issue. (Central Canada)

[smeetsh]

Australia region here, started getting this error in the morning when running a pipeling manually. The automated 5 am run was fine, trying the same pipeline manually around 9 am caused the error.

[Madalina2801]

It seems that this issue is happening again Microsoft Fabric Support and Status | Microsoft Fabric.

Service Outage/Degradation

Fabric Premium customers in the North Europe region may be experiencing issues with submitting Spark jobs, executing notebook operations, or rendering Python and R visuals. Engineers are investigating the issue, and an update will be provided soon.

[jonocrotz]

We noticed this issue which started from 2025/05/13 8PM (UTC+2). What's strange is that we are back at a P1 ticket with Microsoft but on the Service status, its only showing as Degraded.

The other sad part is that its been over 12 hours for us and still no resolution or any ETA on the fix.

[jonocrotz]

Looks like its back up again

[Element115]

There seems to be multiple issues across the whole Fabric platform.  As regards Notebooks, all my Spark SQL stopped working with a bunch of mindboggling errors.  Mind you, Spark SQL code that never triggered an error before, now suddenly the parser complains about syntax errors!

This is not good. This is not a serious platform. Microsoft is not forthcoming. No transparency. Too much complacency. Devs breaking backend code nilly-willy.

Imagine: All your stuff in production grinds to a halt due to no fault of your own. And you are paying for it!  How is this acceptable in any way, shape, or form? 😡😡😡😡😡😡