[FAQ] Why does Spark write multiple parquet files after repartitioning a dataset?

[AsherJD-io]

Course

data-engineering-zoomcamp

Question

Why does writing a Spark DataFrame after repartitioning create multiple parquet files instead of a single file?

Answer

Spark processes data in partitions. When a DataFrame is written to disk, each partition is written as a separate output file.

For example:

trips.repartition(4).write.parquet("output/")

This creates four parquet files because the DataFrame now has four partitions.

This behavior allows Spark to write data in parallel and improves performance when working with large datasets.

Checklist

• I have searched existing FAQs and this question is not already answered
• The answer provides accurate, helpful information
• I have included any relevant code examples or links

[github-actions]

✅ FAQ NEW proposal created in PR #238

Please review and approve the changes.