Synapse Notebook UI thinks cell is running. Notebook won't proceed.

[dbeavon3]

Curious about the notebook ui.  Sometimes I use the command "run this cell and all below".

That is predominently a UI operation that will continue generating user input to advance from one cell to another, right?  IE, it won't submit a background operation and just run it without feedback.

The reason I ask is because sometimes the notebook UI will think a cell is running indefinitely.

The UI behavior is inconsistent with the Spark UI that says this Job (7) had completed 20 minutes ago.

Worst of all, none of the cells that follow will start execution as they should.

What is the best way to recover from this?  I'm guessing I need to open a new browser tab and pray that my session gets reconnected or something along those lines.  This UI is complex, and probably too complex for its own good.  At the end of the day I really hate editing code in a browser.  This notebook coding experience (in a browser) seems like the most silly invention that came out of modern data engineering.  ... I may have to transition to VS code on the desktop, even if it still involves notebooks.

Please let me know how to "kick" this session and run new cells after things get stuck in this confused state.

[v-echaithra · Accepted solution]

Hi @dbeavon3 ,

Manual Recovery

This usually works if the session hasn't fully died.
Click "Stop". This tries to cancel the current cell.
Wait 5-10 seconds to see if the cell resets to idle.

If it doesn’t work:
Refresh the browser tab (press F5).
Synapse will try to reconnect to the Spark session.
Often the notebook will recognize that the session is active and re-sync.
If successful, you can now continue executing from any cell.

Kernel Restart

If refreshing doesn't help:
Use the "Manage Sessions".
Click "Restart session" or "Disconnect & Clear State".
This will kill the Spark session and reset all state.
Re-run cells from the top (or your desired point).

As you mentioned this: "I'm guessing I need to open a new browser tab and pray that my session gets reconnected..." You're not wrong. Opening a new browser tab to the same notebook can sometimes:
Reconnect to the live session
Force UI state to reinitialize

But if the old tab’s UI thread is stuck, the Spark driver may still be considered “active” and resist reconnection. Don’t hit “Run All Below” again while stuck, it just queues new UI commands to an already desynced frontend. Avoid spamming “Stop”, it can sometimes hang even further.

Hope this helps.
Thank you

[dbeavon3]

Thanks for this.  Given the elaborate recovery recipes you shared, I see that the experience is not specific to me or my solution.  If any other IDE (say Visual Studio, or Intellij, or Eclipse, or even VS code) behaved like this, developers would quickly jump ship.  I'm guessing that the majority of Fabric developers must be a lot more patient than I am. 

[tayloramy]

Hi @dbeavon3,

Here’s how to get unstuck when the notebook UI insists a cell is still “Running,” even though Spark says the job finished.

1. Click Run all > Cancel all to flush the client-side run queue. (docs)
2. If cells still show “Running,” hit Stop session (power icon) to tear down the kernel/driver, then Run to start a fresh session. (reference)
3. If the page looks desynced, open the same notebook in a new browser tab (it will offer to reconnect to the active session).
4. As a last resort, run a tiny nudge cell (e.g., spark.version) or cancel jobs from the backend with spark.sparkContext.cancelAllJobs(). (Spark API)

If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.

[dbeavon3]

Stopping the session is not a great option.  It bothers me, since we use custom spark pools and it takes 3 mins to get the session up and running, combined with the time it takes to run all the cells leading up to the one that glitched.   I really want to just pick up where I left off (recover the confused U/I).  If this is just a U/I -related issue then a recovery should be possible.

Given the detailed response, it sounds like these sorts of incidents are not altogether uncommon.  I have only spent about 50 hours or so in Fabric notebooks, and I think it is already enough to last a lifetime.  In general, using notebooks in a browser seem like a very unproductive place to develop complex software.  Even when I'm doing something quick-and-dirty, I'm still very frustrated by this sort of an IDE.  I plan to check and see if the related VS code extensions are any better.