Notebook error- MsSparkUtils parallel run failing with 429 error

[dseym-brb]

Hello,

On Saturday, we began receiving intermittent (but frequent**) errors in our Fabric notebooks that use mssparkutils.runMultiple for parallelizing Spark jobs. The error looks like a 429/500 error trying to hit an internal Microsoft token endpoint to authenticate. The maximum concurrency we are using is 5, so I'm surprised that this would cause a 429 error.

Caused by: java.lang.Exception: Request to https://tokenservice1.northcentralus.trident.azuresynapse.net/api/v1/proxy/preprocessorApi/versions/... failed with status code: 500, response:{"error":"WorkloadApiInternalErrorException","reason":"Internal error 429.Internal error PBIServiceException. (ErrorCode=429) (HTTP 500)"}

We have retries setup in the pipeline so now that the error rate has reduced from 100% to only aboug 25-33%, we are able to complete our pipeline runs, but it is still concerning that a low volume of queries could produce a 429 error. We have an F64 SKU and our compute usage during these jobs stays below 25%.

Has anyone else received this same issue, and if so, were you able to find a root cause?

**From Saturday through Sunday, this was happening for close to 100% of notebooks running mssparkutils.runMultiple, but since then it has become less frequent, about 1/3 or 1/4 notebooks have this failure.

[Anonymous]

Hi @dseym-brb ,

Based on the information you provided, a 429 error usually indicates that the service is receiving too many requests in a short period of time. Here are some potential causes and solutions that can lead to similar issues:

1. Even with low concurrency settings, specific F SKUs and node sizes may still reach the concurrency limit, resulting in a 429 error. You may need to check if your SKUs have any specific limits or quotas.
2. Check your authentication, especially if there have been recent updates or changes in your environment.

Best Regards,

Ada Wang

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.