RDD Storage Info for AdaptiveSparkPlan isFinalPlan=false +- Filter ((NOT is_ride_id_valid#474 OR NOT is_duration_valid#475) OR NOT is_station_valid#476) +- Window [row_number() windowspecdefinition(start_station_id#31, started_at#28 ASC NULLS FIRST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS _start_station_ride_num#536], [start_station_id#31], [started_at#28 ASC NULLS FIRST] +- Sort [start_station_id#31 ASC NULLS FIRST, started_at#28 ASC NULLS FIRST], false, 0 +- Exchange hashpartitioning(start_station_id#31, 200), ENSURE_REQUIREMENTS, [plan_id=91] +- Project [ride_id#26, rideable_type#27, started_at#28, ended_at#29, start_station_name#30, start_station_id#31, end_station_name#32, end_station_id#33, start_lat#34, start_lng#35, end_lat#36, end_lng#37, member_casual#38, is_ride_id_valid#474, is_duration_valid#475, is_station_valid#476, _source_file#493, 2026-08-18 07:26:21.350521 AS _processed_dttm#494, year(cast(started_at#28 as date)) AS year#513, month(cast(started_a...

  • Storage Level: Disk Memory Deserialized 1x Replicated
  • Cached Partitions: 6
  • Total Partitions: 6
  • Memory Size: 8.1 MiB
  • Disk Size: 0.0 B

Data Distribution on 1 Executors

HostOn Heap Memory UsageOff Heap Memory UsageDisk Usage
172.20.0.29:37075 8.1 MiB (1013.0 MiB Remaining) 0.0 B (0.0 B Remaining) 0.0 B

6 Partitions

Page:
Block Name ▾ Storage Level Size in Memory Size on Disk Executors
rdd_49_5 Memory Deserialized 1x Replicated 1004.5 KiB 0.0 B 172.20.0.29:37075
rdd_49_4 Memory Deserialized 1x Replicated 1272.3 KiB 0.0 B 172.20.0.29:37075
rdd_49_3 Memory Deserialized 1x Replicated 1398.0 KiB 0.0 B 172.20.0.29:37075
rdd_49_2 Memory Deserialized 1x Replicated 1519.8 KiB 0.0 B 172.20.0.29:37075
rdd_49_1 Memory Deserialized 1x Replicated 1496.0 KiB 0.0 B 172.20.0.29:37075
rdd_49_0 Memory Deserialized 1x Replicated 1558.7 KiB 0.0 B 172.20.0.29:37075
Page: