Getting error Py4JJavaError: An error occurred while calling o36814.save. : org.apache.spark.SparkEx

[Lakssh]

Iam trying to validate image loaded into Lakehouse using API call . The API is in a VM Server . i will be getting 40000 rows /day . When iam trying to call the API in a batch of 50 rows/batch my notebook is failiing in 1000th row( 25th batch ) with the above row . Iam uisng F4 license , what could be the issue . I tried with batch size of 10 , 20 and 50 . even tried using rdd and map partitition but not able to prpcess more than 1000 row every time it fails with some error or the other , what am i missing here 

[Anonymous]

Hi   @Lakssh 

It seems like this might be an issue related to query limits or data volume limits.

You could check the relevant API documentation to see if there are any query limits. For example, some APIs might limit the number of read and write requests per minute. Additionally, some APIs might restrict the maximum amount of data that can be queried per request.

It might also be related to the computing power of the F4 SKU. You could try temporarily upgrading the Fabric capacity to F8 to see if it yields better results.

 

Are you loading all the data into the lakehouse at once after querying from the API, or are you loading it in batches? If you change the loading frequency, would you get the same error?

 

Best Regards,
Jing
Community Support Team

[govindarajan_d]

@Lakssh  Can you provide the way how you are extracting data from REST API? Since Spark does not have a native way to connect REST API and we usually Python request library to fetch data from the API. 

If you are using a python UDF, it is not the most efficient way and can cause errors. I would recommend splitting the extraction logic by implementing it in Python requests or Data factory pipeline or DF g2 and then run the transformation process.