[iceberg] Error loading in-memory sorter check class path

[andygrove]

Describe the bug

I am fuzz testing Comet 0.10.0-SNAPSHOT and Iceberg 1.8.1 and running into an error for queries with ORDER BY clauses:

SELECT c5, Sqrt(c5) AS x FROM local.db.test1 ORDER BY c5;

[ERROR] Query failed in Comet: Job aborted due to stage failure: Task 7 in stage 27.0 failed 1 times, most recent failure: Lost task 7.0 in stage 27.0 (TID 78) (10.0.0.118 executor driver): java.lang.RuntimeException: Error loading in-memory sorter check class path -- see https://github.com/apache/arrow-datafusion-comet?tab=readme-ov-file#enable-comet-shuffle
...
Caused by: java.lang.IllegalAccessError: failed to access class org.apache.spark.shuffle.sort.ShuffleInMemorySorter from class org.apache.spark.shuffle.sort.CometShuffleExternalSorter$SpillSorter (org.apache.spark.shuffle.sort.ShuffleInMemorySorter is in unnamed module of loader 'app'; org.apache.spark.shuffle.sort.CometShuffleExternalSorter$SpillSorter is in unnamed module of loader org.apache.spark.util.MutableURLClassLoader @2a49fe)
        at org.apache.spark.shuffle.sort.CometShuffleExternalSorter$SpillSorter.<init>(CometShuffleExternalSorter.java:438)

The URL in the error message does not exist.

Steps to reproduce

No response

Expected behavior

No response

Additional context

No response

[andygrove]

Related: #221

[hsiang-c]

take

[parthchandra]

How does one reproduce this? I tried -

scala> spark.sql(s"CREATE TABLE IF NOT EXISTS t1 (c0 INT, c1 STRING) USING iceberg")
...
scala> spark.sql(s"INSERT INTO t1 VALUES ${(0 until 10000).map(i => (i, i)).mkString(",")}")
...
scala> spark.sql(s"select c0, sqrt(c0) from t1 ORDER BY c0").show()
25/08/13 13:04:39 WARN CometExecRule: Comet cannot execute some parts of this plan natively (set spark.comet.explainFallback.enabled=false to disable this logging):
 TakeOrderedAndProject [COMET: ]
+- Project
   +-  BatchScan spark_catalog.default.t1 [COMET: Unsupported scan: org.apache.iceberg.spark.source.SparkBatchQueryScan. Comet Scan only supports Parquet and Iceberg Parquet file formats, BatchScan spark_catalog.default.t1 is not supported]

The query falls back to Spark.

[parthchandra]

Nvm. This is now fixed by the diff in #1987
The error is no longer reproducible.