[blog post] Spark compatibility testing

[andygrove]

What is the problem the feature request solves?

There have been some discussions around writing technical deep dive blog posts on the DataFusion blog. I would like to see if there is interest in a blog post focussing on Spark compatibility testing.

Possible topics:

• Challenges (different floating point implementation, regexp, unicode differences, date/time range differences)
• QueryPlanSerde overview - framework for context-specific tagging as Compatible, Incompatible, Unsupported
• Lessons learned from Fuzz testing approach
• Standalone fuzz testing vs fuzz testing in unit/integration tests
• Lessons learned from agentic coding
• Testing against multiple Spark versions
• Shims
• SQL-based testing, inspired by sqllogictest

Describe the potential solution

No response

Additional context

No response

[Shekharrajak]

are we expecting this blog post to be added here https://github.com/apache/datafusion-comet/tree/main/docs/source/user-guide/latest ? Let me have some draft ready and them update the content as per review.

[andygrove]

are we expecting this blog post to be added here https://github.com/apache/datafusion-comet/tree/main/docs/source/user-guide/latest ? Let me have some draft ready and them update the content as per review.

I was thinking that we would publish on the main DataFusion blog at https://datafusion.apache.org/blog/.

In the past, we have either created a Google document where people can collaborate on posts, or we have created a draft PR against the blog repo https://github.com/apache/datafusion-site where discussions can happen.