RDD Storage Info for AdaptiveSparkPlan isFinalPlan=false +- Project [ride_id#3716, rideable_type#3717, started_at#3718, ended_at#3719, start_station_name#3720, start_station_id#3721, end_station_name#3722, end_station_id#3723, start_lat#3724, start_lng#3725, end_lat#3726, end_lng#3727, member_casual#3728, _corrupt_record#3279, _source_file#3294, _is_ride_id_not_empty#12608, _is_ended_after_started#12609, _is_different_stations#12610, _is_valid#12611, 2026-11-01 20:50:07.641085 AS _processed_dttm#14037] +- Filter (isnotnull(_is_valid#12611) AND NOT _is_valid#12611) +- InMemoryTableScan [_corrupt_record#3279, _is_different_stations#12610, _is_ended_after_started#12609, _is_ride_id_not_empty#12608, _is_valid#12611, _source_file#3294, end_lat#3726, end_lng#3727, end_station_id#3723, end_station_name#3722, ended_at#3719, member_casual#3728, ride_id#3716, rideable_type#3717, start_lat#3724, start_lng#3725, start_station_id#3721, start_station_name#3720, started_at#3718], [isnotnull(_is_valid#12611), NOT _is_valid#12611] ...

  • Storage Level: Disk Memory Deserialized 1x Replicated
  • Cached Partitions: 2
  • Total Partitions: 2
  • Memory Size: 2.7 MiB
  • Disk Size: 0.0 B

Data Distribution on 1 Executors

HostOn Heap Memory UsageOff Heap Memory UsageDisk Usage
172.19.0.22:35015 2.7 MiB (1336.4 MiB Remaining) 0.0 B (0.0 B Remaining) 0.0 B

2 Partitions

Page:
Block Name ▾ Storage Level Size in Memory Size on Disk Executors
rdd_380_1 Memory Deserialized 1x Replicated 1352.6 KiB 0.0 B 172.19.0.22:35015
rdd_380_0 Memory Deserialized 1x Replicated 1369.3 KiB 0.0 B 172.19.0.22:35015
Page: