Mismatched minReaderVersion and readerFeatures - Fabric notebook

[moalnorrk]

Hi,

I'm getting this weird error message when trying to run a Spark SQL code in my notebook in Fabric. I have the latest version for my gateway so that wasn't the issue.
I have been struggling with this for the last 3 days, is there any solution or anyone can help? 

The error is like this (and continues with many more lines but I think these are the important ones here): 
Error while decoding: java.lang.IllegalArgumentException: requirement failed: Mismatched minReaderVersion and readerFeatures. newInstance(class scala.Tuple4). org.apache.spark.sql.errors.QueryExecutionErrors$.expressionDecodingError(QueryExecutionErrors.scala:1405) org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$Deserializer.apply(ExpressionEncoder.scala:185) org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$Deserializer.apply(ExpressionEncoder.scala:173)

[Anonymous]

Hi @moalnorrk ,

Can you please share some more detail information about this issue? They should help us clarify your scenario and test to troubleshoot.

For example:

spark SQL query, notebook environment settings, details configuration and operation steps.

Regards,

Xiaoxin Sheng

[moalnorrk · Accepted solution]

Hi,

We've managed to resolve the issue, and it appears to be related to a bug. The problem arose after a new release of the Gateway we were using to read files into the bronze stage. Initially, we had already transformed data to bronze using the older version of the Gateway. However, once the new release was applied, the issue emerged during the transformation process from bronze to silver.

Despite updating the Gateway to the latest version, the issue persisted. To resolve it, we had to delete all the existing tables/files and reload the data into bronze, followed by a transformation to silver using the updated Gateway.

This seems to suggest a compatibility issue between the older and newer Gateway versions during the transformation process.

Regards,
Moalnorrk