On-demand state repartitioning for Spark Structured Streaming on Databricks

On-demand state repartitioning (Public Preview) in Databricks Runtime 18+ removes that constraint. You can change the number of state partitions and keep the existing checkpointed state. The feature redistributes the state on disk to match the new partition count so the hashing and layout line up after a restart.

Requirements are short and concrete: run Databricks Runtime 18 or later and use the RocksDB state store provider. Databricks set RocksDB as the default state store on DBR 17.3+, so most new stateful queries will already use it.

The operation follows a familiar pattern: stop the streaming query, set a dedicated configuration, and restart. Specifically, set spark.sql.streaming.stateStore.partitions to your desired partition count and restart the query. For stateful queries, spark.sql.streaming.stateStore.partitions takes precedence over spark.sql.shuffle.partitions. On restart, Structured Streaming physically redistributes the RocksDB-backed state files to match the new partition layout so existing keys remain discoverable by the query.

• Tune long-running streams without checkpoint rebuilds. If your initial partitioning becomes a bottleneck as data grows, you can resize to reduce skew and hot partitions.
• Scale resource usage to workload patterns. Shrink or expand partition count as traffic falls or spikes to better match compute and storage characteristics.
• Apply to any stateful operation: aggregations, stream-stream joins, deduplication, session windowing, and transformWithState.

1) Confirm your cluster runs Databricks Runtime 18 or later. 2) Verify RocksDB is the state store (default on DBR 17.3+). 3) Stop the stateful streaming query you want to resize. 4) Set spark.sql.streaming.stateStore.partitions to the target partition count. 5) Restart the query and monitor query progress metrics to confirm the repartition operation completed and the new partition layout is in use.

The repartitioning operation physically moves state files to match the new layout, so monitor query progress and metrics during the restart. The blog emphasizes that the mechanism is designed to be safe: it preserves existing state rather than ignoring updated settings, avoiding silent state corruption.

On-demand state repartitioning removes the long-standing operational trade-off for stateful Structured Streaming: you no longer must choose between overprovisioning and rebuilding checkpoints when traffic changes. If you run stateful streaming at scale on Databricks, enable the RocksDB state store and use spark.sql.streaming.stateStore.partitions to right-size your streams without losing accumulated state.