1

Spark 3.0 new feature

databricks blog

Spark SQL

Improving engine
Adaptive Query Execution(AQE)

improves performance and simplifies tuning by generating a better execution plan at runtime :

  • Dynamically coalesce shuffle partitions
  • Dynamically switching join strategies
  • Dynamically optimizing skew joins
  • Dynamically Partition Pruning
  • ANSI SQL compliance
  • Join hints