API, Core, Spark: Pass FileIO on Spark's read path

[nastra]

The downside of this approach is that the changes are much bigger than in #15368 and we also need

When accessing/reading data files, the codebase is using the Table's FileIO instance through table.io() on Spark's read path. With remote scan planning the FileIO instance is configured with a PlanID + custom storage credentials inside RESTTableScan, but that instance is never propagated to the place(s) that actually perform the read., thus leading to errors.

This PR passes the FileIO obtained during remote/distributed scan planning next to the Table instance on Spark's read path.

This is an alternative to #15368 and requires SerializableFileIOWithSize, which makes sure that the FileIO instance is only closed on the driver and not on executor nodes (similar to SerializableTableWithSize).

[nastra]

this makes sure that the FileIO instance is only closed on the driver and not on executor nodes and is similar to SerializableTableWithSize

[rdblue]

I'm looking into when we can return a FileIO from the scan and I was surprised to see that storage-credentials is returned for any CompletedPlanningResult rather than CompletedPlanningWithIDResult. The one with ID is used for completed responses from the plan endpoint, while the generic result is returned from both plan and fetch endpoints.

Why can we return storage credentials when fetching tasks? Wouldn't it make more sense to return credentials once per planning operation? That simplifies when we know we have credentials.

That wouldn't solve the problem of needing to wait until after planFiles is called to get the FileIO, but it would at least simplify the protocol so we don't need to try to create a new FileIO after each call to fetch more tasks. (@danielcweeks, any thoughts on this?)

[nastra]

I guess if we want to change this we would at the very least need to update the Spec that we added in #14563

[rdblue]

This is a supplier because it will be a broadcast?

[nastra]

this is a supplier because planFiles() will be called later in the read path and so the actual FileIO with the right credentials will only be available later