Issue spark resource usage is low & inefficient

[benitovbreugel]

Hi, 

We are running on an f8 Fabric capacity, with our own created spark environment. In our pipeline we are running the same notebook multiple times and it always uses the created environment. 

Next to that we have enabled high concurrancy sharing, and created a session tag to share the spark session accross steps. Bursting is active as well.

Unfortunatelly we still have a long waiting time in the notebook and a low efficiency rate in the notebook for using spark. 

In the picture below you see that the notebook is assigned to spark, but it stil takes 15 minutes before it actually does something. 

According to our own calculation we do not exceed CU's for the F8 capacity. Therefore we are wondering what we can do to optimize this waiting time?

[tayloramy]

Hi @benitovbreugel,

Here’s what “low Spark efficiency + long waiting time” usually means in Fabric: your notebooks are waiting on session startup or the capacity job queue long before any cells execute. This is common when (a) custom environments are used (image build/attach time), (b) the capacity is at its Spark session/CPU limits so jobs get queued, or (c) you’re launching many separate sessions instead of reusing one via High Concurrency.

 

• Confirm if you’re queued or just slow to start. Open the run and check the Spark details page to see “Queued/Waiting” vs “Running” time. Use the Monitoring Hub > Spark application detail to pinpoint where the 15 minutes is spent (doc, overview).
• Minimize custom environment cost. Starter pools normally attach in ~5-10 seconds; custom environments (extra wheels) can push startup to minutes. Keep your custom env lean or try a test run on the starter pool to compare (doc, community example of slow starts with custom env: thread).
• Reuse one session. Turn on High Concurrency for notebooks/pipelines and keep all steps under the same user/workspace/environment so they pack into a single running session (concept, pipelines setup, notebooks setup).
• Reduce “many notebooks” > “one session”. Chain work in one notebook (use %run to call helpers) instead of multiple pipeline steps that each trigger their own session.
• Check capacity pressure. If the F8 is busy, Spark jobs queue even if your CU math says you “should fit.” Verify job queueing and throttling behavior (Spark job queueing, throttling/bursting). If you see persistent queue waits, consider scaling to F16 or staggering schedules.
• If it’s region/tenant specific, open a ticket. If starter pool also takes many minutes or you see unusual queue delays, contact Microsoft Support with run IDs and timestamps.

If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.

[benitovbreugel]

Hi @tayloramy,

Thanks for your clarification. 

We are indeed using a custom environments, so trying to minimize the usage of this in some notebooks. 

Next to that we also try to reuse the same session with high concurrency. 

The last is that we now also challenge the capacity presure. Maybe in total we are ok, but at the moment we run our process it is possible we exceeds limit at that moent. 

Will investigate this further. 

[v-prasare]

Hi @benitovbreugel,

We would like to confirm if our community members answer resolves your query or if you need further help. If you still have any questions or need more support, please feel free to let us know. We are happy to help you.

@tayloramy ,Thanks for your prompt response

Thank you for your patience and look forward to hearing from you.
Best Regards,
Prashanth Are
MS Fabric community support

[v-prasare]

Hi @benitovbreugel,

We would like to confirm if our community members answer resolves your query or if you need further help. If you still have any questions or need more support, please feel free to let us know. We are happy to help you.

Thank you for your patience and look forward to hearing from you.
Best Regards,
Prashanth Are
MS Fabric community support

[benitovbreugel]

Hi, 

At the moment we have increased efficiency from 17% tot 33%. However, still encoutering an additional 10 minutes before the session actually does something. 

We have done the following: 

• Confirm if you’re queued or just slow to start. It seems
• Minimize custom environment cost. We changed some notebooks to the standard environments, but not all according to some installation packages we need.
• Reuse one session. Which is someting we do with High Concurrency
• Reduce “many notebooks” > “one session”.  This is also what we try to enforce. However It seems that everytime a new session is started, which should not be the case.

So in total we are getting better, but are not there yet.