== Physical Plan ==
AdaptiveSparkPlan (22)
+- == Final Plan ==
Execute InsertIntoHadoopFsRelationCommand (18)
+- WriteFiles (17)
+- Coalesce (16)
+- TableCacheQueryStage (15), Statistics(sizeInBytes=5.9 MiB, rowCount=2.94E+4)
+- InMemoryTableScan (1)
+- InMemoryRelation (2)
+- AdaptiveSparkPlan (14)
+- Project (13)
+- Filter (12)
+- InMemoryTableScan (3)
+- InMemoryRelation (4)
+- AdaptiveSparkPlan (11)
+- Project (10)
+- Project (9)
+- InMemoryTableScan (5)
+- InMemoryRelation (6)
+- * Project (8)
+- Scan csv (7)
+- == Initial Plan ==
Execute InsertIntoHadoopFsRelationCommand (21)
+- WriteFiles (20)
+- Coalesce (19)
+- InMemoryTableScan (1)
+- InMemoryRelation (2)
+- AdaptiveSparkPlan (14)
+- Project (13)
+- Filter (12)
+- InMemoryTableScan (3)
+- InMemoryRelation (4)
+- AdaptiveSparkPlan (11)
+- Project (10)
+- Project (9)
+- InMemoryTableScan (5)
+- InMemoryRelation (6)
+- * Project (8)
+- Scan csv (7)
(1) InMemoryTableScan
Output [20]: [ride_id#3716, rideable_type#3717, started_at#3718, ended_at#3719, start_station_name#3720, start_station_id#3721, end_station_name#3722, end_station_id#3723, start_lat#3724, start_lng#3725, end_lat#3726, end_lng#3727, member_casual#3728, _corrupt_record#3279, _source_file#3294, _is_ride_id_not_empty#12608, _is_ended_after_started#12609, _is_different_stations#12610, _is_valid#12611, _processed_dttm#14037]
Arguments: [ride_id#3716, rideable_type#3717, started_at#3718, ended_at#3719, start_station_name#3720, start_station_id#3721, end_station_name#3722, end_station_id#3723, start_lat#3724, start_lng#3725, end_lat#3726, end_lng#3727, member_casual#3728, _corrupt_record#3279, _source_file#3294, _is_ride_id_not_empty#12608, _is_ended_after_started#12609, _is_different_stations#12610, _is_valid#12611, _processed_dttm#14037]
(2) InMemoryRelation
Arguments: [ride_id#3716, rideable_type#3717, started_at#3718, ended_at#3719, start_station_name#3720, start_station_id#3721, end_station_name#3722, end_station_id#3723, start_lat#3724, start_lng#3725, end_lat#3726, end_lng#3727, member_casual#3728, _corrupt_record#3279, _source_file#3294, _is_ride_id_not_empty#12608, _is_ended_after_started#12609, _is_different_stations#12610, _is_valid#12611, _processed_dttm#14037], CachedRDDBuilder(org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer@28f1e8bc,StorageLevel(disk, memory, deserialized, 1 replicas),AdaptiveSparkPlan isFinalPlan=true
+- == Final Plan ==
*(1) Project [ride_id#3716, rideable_type#3717, started_at#3718, ended_at#3719, start_station_name#3720, start_station_id#3721, end_station_name#3722, end_station_id#3723, start_lat#3724, start_lng#3725, end_lat#3726, end_lng#3727, member_casual#3728, _corrupt_record#3279, _source_file#3294, _is_ride_id_not_empty#12608, _is_ended_after_started#12609, _is_different_stations#12610, _is_valid#12611, 2026-11-01 20:50:07.641085 AS _processed_dttm#14037]
+- *(1) Filter (isnotnull(_is_valid#12611) AND NOT _is_valid#12611)
+- TableCacheQueryStage 0
+- InMemoryTableScan [_corrupt_record#3279, _is_different_stations#12610, _is_ended_after_started#12609, _is_ride_id_not_empty#12608, _is_valid#12611, _source_file#3294, end_lat#3726, end_lng#3727, end_station_id#3723, end_station_name#3722, ended_at#3719, member_casual#3728, ride_id#3716, rideable_type#3717, start_lat#3724, start_lng#3725, start_station_id#3721, start_station_name#3720, started_at#3718], [isnotnull(_is_valid#12611), NOT _is_valid#12611]
+- InMemoryRelation [ride_id#3716, rideable_type#3717, started_at#3718, ended_at#3719, start_station_name#3720, start_station_id#3721, end_station_name#3722, end_station_id#3723, start_lat#3724, start_lng#3725, end_lat#3726, end_lng#3727, member_casual#3728, _corrupt_record#3279, _source_file#3294, _is_ride_id_not_empty#12608, _is_ended_after_started#12609, _is_different_stations#12610, _is_valid#12611], StorageLevel(disk, memory, deserialized, 1 replicas)
+- AdaptiveSparkPlan isFinalPlan=false
+- Project [ride_id#128, rideable_type#129, started_at#669, ended_at#670, start_station_name#132, start_station_id#133, end_station_name#134, end_station_id#135, start_lat#675, start_lng#676, end_lat#677, end_lng#678, member_casual#140, _corrupt_record#141, _source_file#156, (isnotnull(ride_id#128) AND NOT (trim(ride_id#128, None) = )) AS _is_ride_id_not_empty#695, coalesce((ended_at#670 > started_at#669), false) AS _is_ended_after_started#696, coalesce(NOT (end_station_id#135 = start_station_id#133), false) AS _is_different_stations#697, ((((isnotnull(ride_id#128) AND NOT (trim(ride_id#128, None) = )) AND coalesce((ended_at#670 > started_at#669), false)) AND coalesce(NOT (end_station_id#135 = start_station_id#133), false)) AND isnull(_corrupt_record#141)) AS _is_valid#698]
+- Project [ride_id#128, rideable_type#129, cast(started_at#130 as timestamp) AS started_at#669, cast(ended_at#131 as timestamp) AS ended_at#670, start_station_name#132, start_station_id#133, end_station_name#134, end_station_id#135, cast(start_lat#136 as double) AS start_lat#675, cast(start_lng#137 as double) AS start_lng#676, cast(end_lat#138 as double) AS end_lat#677, cast(end_lng#139 as double) AS end_lng#678, member_casual#140, _corrupt_record#141, _source_file#156]
+- InMemoryTableScan [_corrupt_record#141, _source_file#156, end_lat#138, end_lng#139, end_station_id#135, end_station_name#134, ended_at#131, member_casual#140, ride_id#128, rideable_type#129, start_lat#136, start_lng#137, start_station_id#133, start_station_name#132, started_at#130]
+- InMemoryRelation [ride_id#128, rideable_type#129, started_at#130, ended_at#131, start_station_name#132, start_station_id#133, end_station_name#134, end_station_id#135, start_lat#136, start_lng#137, end_lat#138, end_lng#139, member_casual#140, _corrupt_record#141, _source_file#156], StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(1) Project [ride_id#128, rideable_type#129, started_at#130, ended_at#131, start_station_name#132, start_station_id#133, end_station_name#134, end_station_id#135, start_lat#136, start_lng#137, end_lat#138, end_lng#139, member_casual#140, _corrupt_record#141, regexp_extract(input_file_name(), [^/]+$, 0) AS _source_file#156]
+- FileScan csv [ride_id#128,rideable_type#129,started_at#130,ended_at#131,start_station_name#132,start_station_id#133,end_station_name#134,end_station_id#135,start_lat#136,start_lng#137,end_lat#138,end_lng#139,member_casual#140,_corrupt_record#141] Batched: false, DataFilters: [], Format: CSV, Location: InMemoryFileIndex(2 paths)[s3a://rzvde-g11-durnev-pavel/raw/citibike_data/202602/202602-citibike-..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<ride_id:string,rideable_type:string,started_at:string,ended_at:string,start_station_name:s...
+- == Initial Plan ==
Project [ride_id#3716, rideable_type#3717, started_at#3718, ended_at#3719, start_station_name#3720, start_station_id#3721, end_station_name#3722, end_station_id#3723, start_lat#3724, start_lng#3725, end_lat#3726, end_lng#3727, member_casual#3728, _corrupt_record#3279, _source_file#3294, _is_ride_id_not_empty#12608, _is_ended_after_started#12609, _is_different_stations#12610, _is_valid#12611, 2026-11-01 20:50:07.641085 AS _processed_dttm#14037]
+- Filter (isnotnull(_is_valid#12611) AND NOT _is_valid#12611)
+- InMemoryTableScan [_corrupt_record#3279, _is_different_stations#12610, _is_ended_after_started#12609, _is_ride_id_not_empty#12608, _is_valid#12611, _source_file#3294, end_lat#3726, end_lng#3727, end_station_id#3723, end_station_name#3722, ended_at#3719, member_casual#3728, ride_id#3716, rideable_type#3717, start_lat#3724, start_lng#3725, start_station_id#3721, start_station_name#3720, started_at#3718], [isnotnull(_is_valid#12611), NOT _is_valid#12611]
+- InMemoryRelation [ride_id#3716, rideable_type#3717, started_at#3718, ended_at#3719, start_station_name#3720, start_station_id#3721, end_station_name#3722, end_station_id#3723, start_lat#3724, start_lng#3725, end_lat#3726, end_lng#3727, member_casual#3728, _corrupt_record#3279, _source_file#3294, _is_ride_id_not_empty#12608, _is_ended_after_started#12609, _is_different_stations#12610, _is_valid#12611], StorageLevel(disk, memory, deserialized, 1 replicas)
+- AdaptiveSparkPlan isFinalPlan=false
+- Project [ride_id#128, rideable_type#129, started_at#669, ended_at#670, start_station_name#132, start_station_id#133, end_station_name#134, end_station_id#135, start_lat#675, start_lng#676, end_lat#677, end_lng#678, member_casual#140, _corrupt_record#141, _source_file#156, (isnotnull(ride_id#128) AND NOT (trim(ride_id#128, None) = )) AS _is_ride_id_not_empty#695, coalesce((ended_at#670 > started_at#669), false) AS _is_ended_after_started#696, coalesce(NOT (end_station_id#135 = start_station_id#133), false) AS _is_different_stations#697, ((((isnotnull(ride_id#128) AND NOT (trim(ride_id#128, None) = )) AND coalesce((ended_at#670 > started_at#669), false)) AND coalesce(NOT (end_station_id#135 = start_station_id#133), false)) AND isnull(_corrupt_record#141)) AS _is_valid#698]
+- Project [ride_id#128, rideable_type#129, cast(started_at#130 as timestamp) AS started_at#669, cast(ended_at#131 as timestamp) AS ended_at#670, start_station_name#132, start_station_id#133, end_station_name#134, end_station_id#135, cast(start_lat#136 as double) AS start_lat#675, cast(start_lng#137 as double) AS start_lng#676, cast(end_lat#138 as double) AS end_lat#677, cast(end_lng#139 as double) AS end_lng#678, member_casual#140, _corrupt_record#141, _source_file#156]
+- InMemoryTableScan [_corrupt_record#141, _source_file#156, end_lat#138, end_lng#139, end_station_id#135, end_station_name#134, ended_at#131, member_casual#140, ride_id#128, rideable_type#129, start_lat#136, start_lng#137, start_station_id#133, start_station_name#132, started_at#130]
+- InMemoryRelation [ride_id#128, rideable_type#129, started_at#130, ended_at#131, start_station_name#132, start_station_id#133, end_station_name#134, end_station_id#135, start_lat#136, start_lng#137, end_lat#138, end_lng#139, member_casual#140, _corrupt_record#141, _source_file#156], StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(1) Project [ride_id#128, rideable_type#129, started_at#130, ended_at#131, start_station_name#132, start_station_id#133, end_station_name#134, end_station_id#135, start_lat#136, start_lng#137, end_lat#138, end_lng#139, member_casual#140, _corrupt_record#141, regexp_extract(input_file_name(), [^/]+$, 0) AS _source_file#156]
+- FileScan csv [ride_id#128,rideable_type#129,started_at#130,ended_at#131,start_station_name#132,start_station_id#133,end_station_name#134,end_station_id#135,start_lat#136,start_lng#137,end_lat#138,end_lng#139,member_casual#140,_corrupt_record#141] Batched: false, DataFilters: [], Format: CSV, Location: InMemoryFileIndex(2 paths)[s3a://rzvde-g11-durnev-pavel/raw/citibike_data/202602/202602-citibike-..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<ride_id:string,rideable_type:string,started_at:string,ended_at:string,start_station_name:s...
,None)
(3) InMemoryTableScan
Output [19]: [_corrupt_record#3279, _is_different_stations#12610, _is_ended_after_started#12609, _is_ride_id_not_empty#12608, _is_valid#12611, _source_file#3294, end_lat#3726, end_lng#3727, end_station_id#3723, end_station_name#3722, ended_at#3719, member_casual#3728, ride_id#3716, rideable_type#3717, start_lat#3724, start_lng#3725, start_station_id#3721, start_station_name#3720, started_at#3718]
Arguments: [_corrupt_record#3279, _is_different_stations#12610, _is_ended_after_started#12609, _is_ride_id_not_empty#12608, _is_valid#12611, _source_file#3294, end_lat#3726, end_lng#3727, end_station_id#3723, end_station_name#3722, ended_at#3719, member_casual#3728, ride_id#3716, rideable_type#3717, start_lat#3724, start_lng#3725, start_station_id#3721, start_station_name#3720, started_at#3718], [isnotnull(_is_valid#12611), NOT _is_valid#12611]
(4) InMemoryRelation
Arguments: [ride_id#3716, rideable_type#3717, started_at#3718, ended_at#3719, start_station_name#3720, start_station_id#3721, end_station_name#3722, end_station_id#3723, start_lat#3724, start_lng#3725, end_lat#3726, end_lng#3727, member_casual#3728, _corrupt_record#3279, _source_file#3294, _is_ride_id_not_empty#12608, _is_ended_after_started#12609, _is_different_stations#12610, _is_valid#12611], CachedRDDBuilder(org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer@28f1e8bc,StorageLevel(disk, memory, deserialized, 1 replicas),AdaptiveSparkPlan isFinalPlan=true
+- == Final Plan ==
*(1) Project [ride_id#128, rideable_type#129, started_at#669, ended_at#670, start_station_name#132, start_station_id#133, end_station_name#134, end_station_id#135, start_lat#675, start_lng#676, end_lat#677, end_lng#678, member_casual#140, _corrupt_record#141, _source_file#156, (isnotnull(ride_id#128) AND NOT (trim(ride_id#128, None) = )) AS _is_ride_id_not_empty#695, coalesce((ended_at#670 > started_at#669), false) AS _is_ended_after_started#696, coalesce(NOT (end_station_id#135 = start_station_id#133), false) AS _is_different_stations#697, ((((isnotnull(ride_id#128) AND NOT (trim(ride_id#128, None) = )) AND coalesce((ended_at#670 > started_at#669), false)) AND coalesce(NOT (end_station_id#135 = start_station_id#133), false)) AND isnull(_corrupt_record#141)) AS _is_valid#698]
+- *(1) Project [ride_id#128, rideable_type#129, cast(started_at#130 as timestamp) AS started_at#669, cast(ended_at#131 as timestamp) AS ended_at#670, start_station_name#132, start_station_id#133, end_station_name#134, end_station_id#135, cast(start_lat#136 as double) AS start_lat#675, cast(start_lng#137 as double) AS start_lng#676, cast(end_lat#138 as double) AS end_lat#677, cast(end_lng#139 as double) AS end_lng#678, member_casual#140, _corrupt_record#141, _source_file#156]
+- TableCacheQueryStage 0
+- InMemoryTableScan [_corrupt_record#141, _source_file#156, end_lat#138, end_lng#139, end_station_id#135, end_station_name#134, ended_at#131, member_casual#140, ride_id#128, rideable_type#129, start_lat#136, start_lng#137, start_station_id#133, start_station_name#132, started_at#130]
+- InMemoryRelation [ride_id#128, rideable_type#129, started_at#130, ended_at#131, start_station_name#132, start_station_id#133, end_station_name#134, end_station_id#135, start_lat#136, start_lng#137, end_lat#138, end_lng#139, member_casual#140, _corrupt_record#141, _source_file#156], StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(1) Project [ride_id#128, rideable_type#129, started_at#130, ended_at#131, start_station_name#132, start_station_id#133, end_station_name#134, end_station_id#135, start_lat#136, start_lng#137, end_lat#138, end_lng#139, member_casual#140, _corrupt_record#141, regexp_extract(input_file_name(), [^/]+$, 0) AS _source_file#156]
+- FileScan csv [ride_id#128,rideable_type#129,started_at#130,ended_at#131,start_station_name#132,start_station_id#133,end_station_name#134,end_station_id#135,start_lat#136,start_lng#137,end_lat#138,end_lng#139,member_casual#140,_corrupt_record#141] Batched: false, DataFilters: [], Format: CSV, Location: InMemoryFileIndex(2 paths)[s3a://rzvde-g11-durnev-pavel/raw/citibike_data/202602/202602-citibike-..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<ride_id:string,rideable_type:string,started_at:string,ended_at:string,start_station_name:s...
+- == Initial Plan ==
Project [ride_id#128, rideable_type#129, started_at#669, ended_at#670, start_station_name#132, start_station_id#133, end_station_name#134, end_station_id#135, start_lat#675, start_lng#676, end_lat#677, end_lng#678, member_casual#140, _corrupt_record#141, _source_file#156, (isnotnull(ride_id#128) AND NOT (trim(ride_id#128, None) = )) AS _is_ride_id_not_empty#695, coalesce((ended_at#670 > started_at#669), false) AS _is_ended_after_started#696, coalesce(NOT (end_station_id#135 = start_station_id#133), false) AS _is_different_stations#697, ((((isnotnull(ride_id#128) AND NOT (trim(ride_id#128, None) = )) AND coalesce((ended_at#670 > started_at#669), false)) AND coalesce(NOT (end_station_id#135 = start_station_id#133), false)) AND isnull(_corrupt_record#141)) AS _is_valid#698]
+- Project [ride_id#128, rideable_type#129, cast(started_at#130 as timestamp) AS started_at#669, cast(ended_at#131 as timestamp) AS ended_at#670, start_station_name#132, start_station_id#133, end_station_name#134, end_station_id#135, cast(start_lat#136 as double) AS start_lat#675, cast(start_lng#137 as double) AS start_lng#676, cast(end_lat#138 as double) AS end_lat#677, cast(end_lng#139 as double) AS end_lng#678, member_casual#140, _corrupt_record#141, _source_file#156]
+- InMemoryTableScan [_corrupt_record#141, _source_file#156, end_lat#138, end_lng#139, end_station_id#135, end_station_name#134, ended_at#131, member_casual#140, ride_id#128, rideable_type#129, start_lat#136, start_lng#137, start_station_id#133, start_station_name#132, started_at#130]
+- InMemoryRelation [ride_id#128, rideable_type#129, started_at#130, ended_at#131, start_station_name#132, start_station_id#133, end_station_name#134, end_station_id#135, start_lat#136, start_lng#137, end_lat#138, end_lng#139, member_casual#140, _corrupt_record#141, _source_file#156], StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(1) Project [ride_id#128, rideable_type#129, started_at#130, ended_at#131, start_station_name#132, start_station_id#133, end_station_name#134, end_station_id#135, start_lat#136, start_lng#137, end_lat#138, end_lng#139, member_casual#140, _corrupt_record#141, regexp_extract(input_file_name(), [^/]+$, 0) AS _source_file#156]
+- FileScan csv [ride_id#128,rideable_type#129,started_at#130,ended_at#131,start_station_name#132,start_station_id#133,end_station_name#134,end_station_id#135,start_lat#136,start_lng#137,end_lat#138,end_lng#139,member_casual#140,_corrupt_record#141] Batched: false, DataFilters: [], Format: CSV, Location: InMemoryFileIndex(2 paths)[s3a://rzvde-g11-durnev-pavel/raw/citibike_data/202602/202602-citibike-..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<ride_id:string,rideable_type:string,started_at:string,ended_at:string,start_station_name:s...
,None)
(5) InMemoryTableScan
Output [15]: [_corrupt_record#141, _source_file#156, end_lat#138, end_lng#139, end_station_id#135, end_station_name#134, ended_at#131, member_casual#140, ride_id#128, rideable_type#129, start_lat#136, start_lng#137, start_station_id#133, start_station_name#132, started_at#130]
Arguments: [_corrupt_record#141, _source_file#156, end_lat#138, end_lng#139, end_station_id#135, end_station_name#134, ended_at#131, member_casual#140, ride_id#128, rideable_type#129, start_lat#136, start_lng#137, start_station_id#133, start_station_name#132, started_at#130]
(6) InMemoryRelation
Arguments: [ride_id#128, rideable_type#129, started_at#130, ended_at#131, start_station_name#132, start_station_id#133, end_station_name#134, end_station_id#135, start_lat#136, start_lng#137, end_lat#138, end_lng#139, member_casual#140, _corrupt_record#141, _source_file#156], CachedRDDBuilder(org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer@28f1e8bc,StorageLevel(disk, memory, deserialized, 1 replicas),*(1) Project [ride_id#128, rideable_type#129, started_at#130, ended_at#131, start_station_name#132, start_station_id#133, end_station_name#134, end_station_id#135, start_lat#136, start_lng#137, end_lat#138, end_lng#139, member_casual#140, _corrupt_record#141, regexp_extract(input_file_name(), [^/]+$, 0) AS _source_file#156]
+- FileScan csv [ride_id#128,rideable_type#129,started_at#130,ended_at#131,start_station_name#132,start_station_id#133,end_station_name#134,end_station_id#135,start_lat#136,start_lng#137,end_lat#138,end_lng#139,member_casual#140,_corrupt_record#141] Batched: false, DataFilters: [], Format: CSV, Location: InMemoryFileIndex(2 paths)[s3a://rzvde-g11-durnev-pavel/raw/citibike_data/202602/202602-citibike-..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<ride_id:string,rideable_type:string,started_at:string,ended_at:string,start_station_name:s...
,None)
(7) Scan csv
Output [14]: [ride_id#128, rideable_type#129, started_at#130, ended_at#131, start_station_name#132, start_station_id#133, end_station_name#134, end_station_id#135, start_lat#136, start_lng#137, end_lat#138, end_lng#139, member_casual#140, _corrupt_record#141]
Batched: false
Location: InMemoryFileIndex [s3a://rzvde-g11-durnev-pavel/raw/citibike_data/202602/202602-citibike-tripdata-part00.csv, ... 1 entries]
ReadSchema: struct<ride_id:string,rideable_type:string,started_at:string,ended_at:string,start_station_name:string,start_station_id:string,end_station_name:string,end_station_id:string,start_lat:string,start_lng:string,end_lat:string,end_lng:string,member_casual:string,_corrupt_record:string>
(8) Project [codegen id : 1]
Output [15]: [ride_id#128, rideable_type#129, started_at#130, ended_at#131, start_station_name#132, start_station_id#133, end_station_name#134, end_station_id#135, start_lat#136, start_lng#137, end_lat#138, end_lng#139, member_casual#140, _corrupt_record#141, regexp_extract(input_file_name(), [^/]+$, 0) AS _source_file#156]
Input [14]: [ride_id#128, rideable_type#129, started_at#130, ended_at#131, start_station_name#132, start_station_id#133, end_station_name#134, end_station_id#135, start_lat#136, start_lng#137, end_lat#138, end_lng#139, member_casual#140, _corrupt_record#141]
(9) Project
Output [15]: [ride_id#128, rideable_type#129, cast(started_at#130 as timestamp) AS started_at#669, cast(ended_at#131 as timestamp) AS ended_at#670, start_station_name#132, start_station_id#133, end_station_name#134, end_station_id#135, cast(start_lat#136 as double) AS start_lat#675, cast(start_lng#137 as double) AS start_lng#676, cast(end_lat#138 as double) AS end_lat#677, cast(end_lng#139 as double) AS end_lng#678, member_casual#140, _corrupt_record#141, _source_file#156]
Input [15]: [_corrupt_record#141, _source_file#156, end_lat#138, end_lng#139, end_station_id#135, end_station_name#134, ended_at#131, member_casual#140, ride_id#128, rideable_type#129, start_lat#136, start_lng#137, start_station_id#133, start_station_name#132, started_at#130]
(10) Project
Output [19]: [ride_id#128, rideable_type#129, started_at#669, ended_at#670, start_station_name#132, start_station_id#133, end_station_name#134, end_station_id#135, start_lat#675, start_lng#676, end_lat#677, end_lng#678, member_casual#140, _corrupt_record#141, _source_file#156, (isnotnull(ride_id#128) AND NOT (trim(ride_id#128, None) = )) AS _is_ride_id_not_empty#695, coalesce((ended_at#670 > started_at#669), false) AS _is_ended_after_started#696, coalesce(NOT (end_station_id#135 = start_station_id#133), false) AS _is_different_stations#697, ((((isnotnull(ride_id#128) AND NOT (trim(ride_id#128, None) = )) AND coalesce((ended_at#670 > started_at#669), false)) AND coalesce(NOT (end_station_id#135 = start_station_id#133), false)) AND isnull(_corrupt_record#141)) AS _is_valid#698]
Input [15]: [ride_id#128, rideable_type#129, started_at#669, ended_at#670, start_station_name#132, start_station_id#133, end_station_name#134, end_station_id#135, start_lat#675, start_lng#676, end_lat#677, end_lng#678, member_casual#140, _corrupt_record#141, _source_file#156]
(11) AdaptiveSparkPlan
Output [19]: [ride_id#128, rideable_type#129, started_at#669, ended_at#670, start_station_name#132, start_station_id#133, end_station_name#134, end_station_id#135, start_lat#675, start_lng#676, end_lat#677, end_lng#678, member_casual#140, _corrupt_record#141, _source_file#156, _is_ride_id_not_empty#695, _is_ended_after_started#696, _is_different_stations#697, _is_valid#698]
Arguments: isFinalPlan=false
(12) Filter
Input [19]: [_corrupt_record#3279, _is_different_stations#12610, _is_ended_after_started#12609, _is_ride_id_not_empty#12608, _is_valid#12611, _source_file#3294, end_lat#3726, end_lng#3727, end_station_id#3723, end_station_name#3722, ended_at#3719, member_casual#3728, ride_id#3716, rideable_type#3717, start_lat#3724, start_lng#3725, start_station_id#3721, start_station_name#3720, started_at#3718]
Condition : (isnotnull(_is_valid#12611) AND NOT _is_valid#12611)
(13) Project
Output [20]: [ride_id#3716, rideable_type#3717, started_at#3718, ended_at#3719, start_station_name#3720, start_station_id#3721, end_station_name#3722, end_station_id#3723, start_lat#3724, start_lng#3725, end_lat#3726, end_lng#3727, member_casual#3728, _corrupt_record#3279, _source_file#3294, _is_ride_id_not_empty#12608, _is_ended_after_started#12609, _is_different_stations#12610, _is_valid#12611, 2026-11-01 20:50:07.641085 AS _processed_dttm#14037]
Input [19]: [_corrupt_record#3279, _is_different_stations#12610, _is_ended_after_started#12609, _is_ride_id_not_empty#12608, _is_valid#12611, _source_file#3294, end_lat#3726, end_lng#3727, end_station_id#3723, end_station_name#3722, ended_at#3719, member_casual#3728, ride_id#3716, rideable_type#3717, start_lat#3724, start_lng#3725, start_station_id#3721, start_station_name#3720, started_at#3718]
(14) AdaptiveSparkPlan
Output [20]: [ride_id#3716, rideable_type#3717, started_at#3718, ended_at#3719, start_station_name#3720, start_station_id#3721, end_station_name#3722, end_station_id#3723, start_lat#3724, start_lng#3725, end_lat#3726, end_lng#3727, member_casual#3728, _corrupt_record#3279, _source_file#3294, _is_ride_id_not_empty#12608, _is_ended_after_started#12609, _is_different_stations#12610, _is_valid#12611, _processed_dttm#14037]
Arguments: isFinalPlan=false
(15) TableCacheQueryStage
Output [20]: [ride_id#3716, rideable_type#3717, started_at#3718, ended_at#3719, start_station_name#3720, start_station_id#3721, end_station_name#3722, end_station_id#3723, start_lat#3724, start_lng#3725, end_lat#3726, end_lng#3727, member_casual#3728, _corrupt_record#3279, _source_file#3294, _is_ride_id_not_empty#12608, _is_ended_after_started#12609, _is_different_stations#12610, _is_valid#12611, _processed_dttm#14037]
Arguments: 0
(16) Coalesce
Input [20]: [ride_id#3716, rideable_type#3717, started_at#3718, ended_at#3719, start_station_name#3720, start_station_id#3721, end_station_name#3722, end_station_id#3723, start_lat#3724, start_lng#3725, end_lat#3726, end_lng#3727, member_casual#3728, _corrupt_record#3279, _source_file#3294, _is_ride_id_not_empty#12608, _is_ended_after_started#12609, _is_different_stations#12610, _is_valid#12611, _processed_dttm#14037]
Arguments: 1
(17) WriteFiles
Input [20]: [ride_id#3716, rideable_type#3717, started_at#3718, ended_at#3719, start_station_name#3720, start_station_id#3721, end_station_name#3722, end_station_id#3723, start_lat#3724, start_lng#3725, end_lat#3726, end_lng#3727, member_casual#3728, _corrupt_record#3279, _source_file#3294, _is_ride_id_not_empty#12608, _is_ended_after_started#12609, _is_different_stations#12610, _is_valid#12611, _processed_dttm#14037]
(18) Execute InsertIntoHadoopFsRelationCommand
Input: []
Arguments: s3a://rzvde-g11-durnev-pavel/raw_invalid/citibike_data/202602, false, CSV, [header=true, path=s3a://rzvde-g11-durnev-pavel/raw_invalid/citibike_data/202602], Overwrite, [ride_id, rideable_type, started_at, ended_at, start_station_name, start_station_id, end_station_name, end_station_id, start_lat, start_lng, end_lat, end_lng, member_casual, _corrupt_record, _source_file, _is_ride_id_not_empty, _is_ended_after_started, _is_different_stations, _is_valid, _processed_dttm]
(19) Coalesce
Input [20]: [ride_id#3716, rideable_type#3717, started_at#3718, ended_at#3719, start_station_name#3720, start_station_id#3721, end_station_name#3722, end_station_id#3723, start_lat#3724, start_lng#3725, end_lat#3726, end_lng#3727, member_casual#3728, _corrupt_record#3279, _source_file#3294, _is_ride_id_not_empty#12608, _is_ended_after_started#12609, _is_different_stations#12610, _is_valid#12611, _processed_dttm#14037]
Arguments: 1
(20) WriteFiles
Input [20]: [ride_id#3716, rideable_type#3717, started_at#3718, ended_at#3719, start_station_name#3720, start_station_id#3721, end_station_name#3722, end_station_id#3723, start_lat#3724, start_lng#3725, end_lat#3726, end_lng#3727, member_casual#3728, _corrupt_record#3279, _source_file#3294, _is_ride_id_not_empty#12608, _is_ended_after_started#12609, _is_different_stations#12610, _is_valid#12611, _processed_dttm#14037]
(21) Execute InsertIntoHadoopFsRelationCommand
Input: []
Arguments: s3a://rzvde-g11-durnev-pavel/raw_invalid/citibike_data/202602, false, CSV, [header=true, path=s3a://rzvde-g11-durnev-pavel/raw_invalid/citibike_data/202602], Overwrite, [ride_id, rideable_type, started_at, ended_at, start_station_name, start_station_id, end_station_name, end_station_id, start_lat, start_lng, end_lat, end_lng, member_casual, _corrupt_record, _source_file, _is_ride_id_not_empty, _is_ended_after_started, _is_different_stations, _is_valid, _processed_dttm]
(22) AdaptiveSparkPlan
Output: []
Arguments: isFinalPlan=true