Issues running Spark Notebook

[ronan_b6]

Hi I'm runing spark notebooks on Fabric trial, it's been working great and no time out issues, or severe dalys in sessions starting.

But today, notebooks do not start the spark session, returning this type of error:

"Livy session has failed. Error code: SparkCoreError/SessionDidNotEnterIdle. SessionInfo.State from SparkCore is Error: Session did not enter idle state after 10 minutes. Source: System."

I see from the forum that others have experience similar issues, with severe delays in functionality.

Can anyone shed more light on this?

Thanks,

[Anonymous]

Hi @ronan_b6 ,

May I ask if you are still facing this issue?
You can try to restart the spark pool to resolve the issue and make sure that the spark is configured correctly and there are enough available resources in the spark pool.
If the issue is still not resolved, can you provide more details like in which region you are facing this issue and attach a screenshot of the issue.

Best Regards,

Ada Wang

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[naveesupply]

Hi Ada, 

How would I restart the spark pool? I am facing the same issue on my trial, I am also based out of Canada

[JFTxJ]

Can you please clarify how to "restart the spark pool"?

Thank you

[Sarien]

Still facing the same issue as well. Tried changing the environment version as well which did not help.

Not sure how I should restart the pool as you are saying.

[benfedit]

hi Ada we have the same issue here in North Europe all our notebooks in Fabric are affected.

[maxmerchant]

I, too, am encountering this issue since yesterday. We were able to run notebooks without any issues until last week and started seeing the error below. Tried several different notebooks in different workspaces and all give the same error.

"Livy session has failed. Error code: SparkCoreError/SessionDidNotEnterIdle. SessionInfo.State from SparkCore is Error: Session did not enter idle state after 10 minutes. Source: System"

We are using Fabric F64 instance. Region Canada Central.

Is anyone else having this issue as well?

[naveesupply]

Same issue, were you able to fix it? 

[maxmerchant]

No. Still facing this issue. The weird thing is that it Livy session starts every once in a while, with no consistency. So can't execute a pipeline with a notebook activity.

[naveesupply]

Ya I am just running a script using the notebook but its not connecting at all. Will have to wait to see if it works tomorrow. 

[dmedeiros]

I am also facing the same issue since yesterday both using the trial capacity and a reserved capacity - both from Canada Central

[dver344]

Having a similar issue. It's not timing out, since I don't do a lot of complex processes, but the spark pools usually takes 3 to 5 min to start (before the issue this was 10sec). This all started 24h ago for me (time of writing is 17/07/2024, 2PM Brussels)

[Anonymous]

Facing the same issue in Ontario, Canada

[ToughItOut]

My organization is encountering this issue today as well, also in Canada Central. 
Ive submitted a support request to Unified Support, will update if I learn more.

[ToughItOut]

Looks like its been identified as an issue now, I was in contact with support briefly and passed on some information, then I received a notice around 10PM MST last night. Currently there is a posting in the admin center Service health - Microsoft 365 admin center:

[ToughItOut]

Latest Update:

[ToughItOut]

Final update - I see more posts from other locations, but the notice has changed to resolved. Hoping every is able to start a session now!

• 

   

[jonocrotz]

Thanks so much for the updates and visibility.

We are still unable to start up a session as all our pipelines with notebooks are still failing.

I'm waiting on feedback on a P1 ticket from a Partner so will post an update if I hear anything else

[MMS_007]

Which region are you? Maybe they fixed it for Canada. I'm North Europe and I'm still facing the issues... 

[ToughItOut]

Edit - sorry thought you were replying to me. 

Canada Central - I've heard from my team that it seems to be working for them in this region, but clearly this is a larger issue. 

[Sarien]

It is not fixed yet, just got the error again.

[Anonymous]

Hi all,

We have reported this issue and submitted it to the team.
They have been aware of the issue and the engineers will do their best to resolve it. So please be patient.

Best Regards,

Ada Wang

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[jgus]

any updates?

[ToughItOut]

Latest update:

[higgy7]

I am also having the same issue, pipelines with a notebook element fail at the notebook stage with the same error as above.

When i try to start a session within a notebook it tries to start for 10 minutes and then fails to start and disconnects. This issue only started yesterday.

[danieletiles]

Facing the same issue, North Europe

[tlanza89_RWL]

I'm having the same issue in the North Central US region as well. The error happens when running a notebook in a pipeline.

"Notebook execution failed at Notebook service with http status code - '200', please check the Run logs on Notebook, additional details - 'Error name - Exception, Error value - Failed to create Livy session for executing notebook"

[daniele_tiles]

In my case is a plain notebook, from scratch. No way to start a spark session

[MMS_007]

Same issue for me. Cannot run pipelines with notebooks (any useful error information), but also cannot run the notebook itself - after around 10 minutes it returns:
SparkCoreError/SessionDidNotEnterIdle: Livy session has failed. Error code: SparkCoreError/SessionDidNotEnterIdle. SessionInfo.State from SparkCore is Error: Session did not enter idle state after 10 minutes. Source: System.

For me it started around 4PM CET yesterday.

In About Power BI section I can see: Your data is stored in North Europe (Ireland) so I guess that's my region (I'm not an admin). It's not a trial but paid Fabric license. 

@Anonymous Ada, what is this issue, what can be done about it, when it will start working?
I cannot develop nor refresh the data in any of workspaces I'm responsible for... It needs to be fixed urgently

[abm_chris]

North Europe, now having the same issue, seem to be running ok for me from 10-11am BST

[Anonymous]

Facing the same issue with a F64 Capacity in North Europe.

[WhiteSox77]

Issue just started last 24 hours, North Europe (UK) region

[jonocrotz]

Also having the same issue, as mentioned above. No outage or awareness on the service so very frustrating that we are facing an issue for so long yet no acknowledgement by Microsoft

[r_mel]

Same issue here (North America). Commenting so I get the solution when it is found.

[CartierSantos]

Same issue here in Norway, North Europe region. Getting the same error. Weirdly, the Azure status says everything is working fine.

[Jaggana]

Same error North Europe

[danieletiles]

It's frustrating that in the status of Fabric everything is "working without any problem", while this problem has been experienced in many regions...

[suriaprakash]

We're also facing the issue - North America Central zone from 17th midnight. 

[kevingauv]

Same issue here.

We created a support ticket to Microsoft yesterday. They asked for some detail regarding the issue, but we still waiting for an answer.

[jade7]

I have been experiencing this same issue in CANADA CENTRAL for the last 2 days.

Just started using my trial and I am starting to wonder if this normal. lol