g11.durnev.pavel-CitiBike-WAP - Details for Query 40

Submitted Time: 2026/11/01 20:50:07
Duration: 9 s
Succeeded Jobs: 81 82 83 84

Show the Stage ID and Task ID that corresponds to the max metric

digraph G { 0 [labelType="html" label=" AdaptiveSparkPlan "]; subgraph cluster1 { isCluster="true"; label="WholeStageCodegen (2)\n \nduration: 0 ms"; 2 [labelType="html" label="HashAggregate time in aggregation build: 0 ms number of output rows: 1"]; } 3 [labelType="html" label="Exchange shuffle records written: 2 local merged chunks fetched: 0 shuffle write time total (min, med, max (stageId: taskId)) 0 ms (0 ms, 0 ms, 0 ms (stage 128.0: task 125)) remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 2 local bytes read: 118.0 B fetch wait time: 0 ms remote bytes read: 0.0 B merged fetch fallback count: 0 local blocks read: 2 remote merged chunks fetched: 0 remote blocks read: 0 data size total (min, med, max (stageId: taskId)) 32.0 B (0.0 B, 16.0 B, 16.0 B (stage 128.0: task 124)) local merged bytes read: 0.0 B number of partitions: 1 remote reqs duration: 0 ms remote bytes read to disk: 0.0 B shuffle bytes written total (min, med, max (stageId: taskId)) 118.0 B (0.0 B, 59.0 B, 59.0 B (stage 128.0: task 124))"]; subgraph cluster4 { isCluster="true"; label="WholeStageCodegen (1)\n \nduration: total (min, med, max (stageId: taskId))\n6 ms (3 ms, 3 ms, 3 ms (stage 128.0: task 124))"; 5 [labelType="html" label="HashAggregate time in aggregation build total (min, med, max (stageId: taskId)) 6 ms (3 ms, 3 ms, 3 ms (stage 128.0: task 124)) number of output rows: 2"]; } 6 [labelType="html" label="InMemoryTableScan number of output rows: 1,190,091"]; 7 [labelType="html" label=" AdaptiveSparkPlan "]; subgraph cluster8 { isCluster="true"; label="WholeStageCodegen (3)\n \nduration: total (min, med, max (stageId: taskId))\n21.2 s (0 ms, 10.3 s, 10.9 s (stage 126.0: task 122))"; 9 [labelType="html" label=" Project "]; } 10 [labelType="html" label="Window spill size total (min, med, max (stageId: taskId)) 0.0 B (0.0 B, 0.0 B, 0.0 B (stage 128.0: task 124))"]; subgraph cluster11 { isCluster="true"; label="WholeStageCodegen (2)\n \nduration: total (min, med, max (stageId: taskId))\n13.0 s (0 ms, 6.4 s, 6.7 s (stage 126.0: task 122))"; 12 [labelType="html" label="Sort sort time total (min, med, max (stageId: taskId)) 1.6 s (0 ms, 790 ms, 797 ms (stage 126.0: task 123)) peak memory total (min, med, max (stageId: taskId)) 416.0 MiB (0.0 B, 208.0 MiB, 208.0 MiB (stage 126.0: task 123)) spill size total (min, med, max (stageId: taskId)) 0.0 B (0.0 B, 0.0 B, 0.0 B (stage 128.0: task 124))"]; } 13 [labelType="html" label="AQEShuffleRead number of partitions: 2 partition data size total (min, med, max (stageId: taskId)) 91.1 MiB (45.4 MiB, 45.6 MiB, 45.6 MiB (driver)) number of coalesced partitions: 2"]; 14 [labelType="html" label="Exchange shuffle records written: 1,190,091 local merged chunks fetched: 0 shuffle write time total (min, med, max (stageId: taskId)) 269 ms (0 ms, 0 ms, 138 ms (stage 124.0: task 120)) remote merged bytes read total (min, med, max (stageId: taskId)) 0.0 B (0.0 B, 0.0 B, 0.0 B (stage 128.0: task 124)) local merged blocks fetched: 0 corrupt merged block chunks: 0 remote merged reqs duration total (min, med, max (stageId: taskId)) 0 ms (0 ms, 0 ms, 0 ms (stage 128.0: task 124)) remote merged blocks fetched: 0 records read: 1,190,091 local bytes read total (min, med, max (stageId: taskId)) 86.9 MiB (0.0 B, 43.4 MiB, 43.5 MiB (stage 126.0: task 123)) fetch wait time total (min, med, max (stageId: taskId)) 0 ms (0 ms, 0 ms, 0 ms (stage 128.0: task 124)) remote bytes read total (min, med, max (stageId: taskId)) 0.0 B (0.0 B, 0.0 B, 0.0 B (stage 128.0: task 124)) merged fetch fallback count: 0 local blocks read: 4 remote merged chunks fetched: 0 remote blocks read: 0 data size total (min, med, max (stageId: taskId)) 368.9 MiB (0.0 B, 0.0 B, 191.0 MiB (stage 124.0: task 120)) local merged bytes read total (min, med, max (stageId: taskId)) 0.0 B (0.0 B, 0.0 B, 0.0 B (stage 128.0: task 124)) number of partitions: 200 remote reqs duration total (min, med, max (stageId: taskId)) 0 ms (0 ms, 0 ms, 0 ms (stage 128.0: task 124)) remote bytes read to disk total (min, med, max (stageId: taskId)) 0.0 B (0.0 B, 0.0 B, 0.0 B (stage 128.0: task 124)) shuffle bytes written total (min, med, max (stageId: taskId)) 86.9 MiB (0.0 B, 0.0 B, 44.6 MiB (stage 124.0: task 120))"]; subgraph cluster15 { isCluster="true"; label="WholeStageCodegen (1)\n \nduration: total (min, med, max (stageId: taskId))\n3.6 s (0 ms, 0 ms, 1.8 s (stage 124.0: task 120))"; 16 [labelType="html" label=" Project "]; 17 [labelType="html" label="Filter number of output rows: 1,190,091"]; } 18 [labelType="html" label="InMemoryTableScan number of output rows: 1,219,444"]; 19 [labelType="html" label=" AdaptiveSparkPlan "]; subgraph cluster20 { isCluster="true"; label="WholeStageCodegen (1)\n \nduration: total (min, med, max (stageId: taskId))\n0 ms (0 ms, 0 ms, 0 ms (stage 124.0: task 121))"; 21 [labelType="html" label=" Project "]; 22 [labelType="html" label=" Project "]; } 23 [labelType="html" label="InMemoryTableScan number of output rows: 0"]; subgraph cluster24 { isCluster="true"; label="WholeStageCodegen (1)\n \nduration: total (min, med, max (stageId: taskId))\n0 ms (0 ms, 0 ms, 0 ms (stage 124.0: task 121))"; 25 [labelType="html" label=" Project "]; } 26 [labelType="html" label="Scan csv number of output rows: 0"]; 2->0; 3->2; 5->3; 6->5; 7->6; 9->7; 10->9; 12->10; 13->12; 14->13; 16->14; 17->16; 18->17; 19->18; 21->19; 22->21; 23->22; 25->23; 26->25; }

AdaptiveSparkPlan isFinalPlan=true

HashAggregate(keys=[], functions=[count(1)])

WholeStageCodegen (2)

Exchange SinglePartition, ENSURE_REQUIREMENTS, [plan_id=3003]

HashAggregate(keys=[], functions=[partial_count(1)])

WholeStageCodegen (1)

InMemoryTableScan

AdaptiveSparkPlan isFinalPlan=true

Project [ride_id#3716, rideable_type#3717, started_at#3718, ended_at#3719, start_station_name#3720, start_station_id#3721, end_station_name#3722, end_station_id#3723, start_lat#3724, start_lng#3725, end_lat#3726, end_lng#3727, member_casual#3728, _source_file#3294, 2026-11-01 20:50:07.599499 AS _processed_dttm#13637, _start_station_ride_num#13639, year#13640, month#13641]

WholeStageCodegen (3)

Window [row_number() windowspecdefinition(start_station_id#3721, _w0#13642, _w1#13643, started_at#3718 ASC NULLS FIRST, ride_id#3716 ASC NULLS FIRST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS _start_station_ride_num#13639], [start_station_id#3721, _w0#13642, _w1#13643], [started_at#3718 ASC NULLS FIRST, ride_id#3716 ASC NULLS FIRST]

Sort [start_station_id#3721 ASC NULLS FIRST, _w0#13642 ASC NULLS FIRST, _w1#13643 ASC NULLS FIRST, started_at#3718 ASC NULLS FIRST, ride_id#3716 ASC NULLS FIRST], false, 0

WholeStageCodegen (2)

AQEShuffleRead coalesced

Exchange hashpartitioning(start_station_id#3721, _w0#13642, _w1#13643, 200), ENSURE_REQUIREMENTS, [plan_id=2956]

Project [ride_id#3716, rideable_type#3717, started_at#3718, ended_at#3719, start_station_name#3720, start_station_id#3721, end_station_name#3722, end_station_id#3723, start_lat#3724, start_lng#3725, end_lat#3726, end_lng#3727, member_casual#3728, _source_file#3294, date_format(started_at#3718, yyyy, Some(UTC)) AS year#13640, date_format(started_at#3718, MM, Some(UTC)) AS month#13641, date_format(started_at#3718, yyyy, Some(UTC)) AS _w0#13642, date_format(started_at#3718, MM, Some(UTC)) AS _w1#13643]

Filter (isnotnull(_is_valid#12611) AND _is_valid#12611)

WholeStageCodegen (1)

InMemoryTableScan [_is_valid#12611, _source_file#3294, end_lat#3726, end_lng#3727, end_station_id#3723, end_station_name#3722, ended_at#3719, member_casual#3728, ride_id#3716, rideable_type#3717, start_lat#3724, start_lng#3725, start_station_id#3721, start_station_name#3720, started_at#3718], [isnotnull(_is_valid#12611), _is_valid#12611]

AdaptiveSparkPlan isFinalPlan=true

Project [ride_id#128, rideable_type#129, started_at#669, ended_at#670, start_station_name#132, start_station_id#133, end_station_name#134, end_station_id#135, start_lat#675, start_lng#676, end_lat#677, end_lng#678, member_casual#140, _corrupt_record#141, _source_file#156, (isnotnull(ride_id#128) AND NOT (trim(ride_id#128, None) = )) AS _is_ride_id_not_empty#695, coalesce((ended_at#670 > started_at#669), false) AS _is_ended_after_started#696, coalesce(NOT (end_station_id#135 = start_station_id#133), false) AS _is_different_stations#697, ((((isnotnull(ride_id#128) AND NOT (trim(ride_id#128, None) = )) AND coalesce((ended_at#670 > started_at#669), false)) AND coalesce(NOT (end_station_id#135 = start_station_id#133), false)) AND isnull(_corrupt_record#141)) AS _is_valid#698]

Project [ride_id#128, rideable_type#129, cast(started_at#130 as timestamp) AS started_at#669, cast(ended_at#131 as timestamp) AS ended_at#670, start_station_name#132, start_station_id#133, end_station_name#134, end_station_id#135, cast(start_lat#136 as double) AS start_lat#675, cast(start_lng#137 as double) AS start_lng#676, cast(end_lat#138 as double) AS end_lat#677, cast(end_lng#139 as double) AS end_lng#678, member_casual#140, _corrupt_record#141, _source_file#156]

WholeStageCodegen (1)

InMemoryTableScan [_corrupt_record#141, _source_file#156, end_lat#138, end_lng#139, end_station_id#135, end_station_name#134, ended_at#131, member_casual#140, ride_id#128, rideable_type#129, start_lat#136, start_lng#137, start_station_id#133, start_station_name#132, started_at#130]

Project [ride_id#128, rideable_type#129, started_at#130, ended_at#131, start_station_name#132, start_station_id#133, end_station_name#134, end_station_id#135, start_lat#136, start_lng#137, end_lat#138, end_lng#139, member_casual#140, _corrupt_record#141, regexp_extract(input_file_name(), [^/]+$, 0) AS _source_file#156]

WholeStageCodegen (1)

FileScan csv [ride_id#128,rideable_type#129,started_at#130,ended_at#131,start_station_name#132,start_station_id#133,end_station_name#134,end_station_id#135,start_lat#136,start_lng#137,end_lat#138,end_lng#139,member_casual#140,_corrupt_record#141] Batched: false, DataFilters: [], Format: CSV, Location: InMemoryFileIndex(2 paths)[s3a://rzvde-g11-durnev-pavel/raw/citibike_data/202602/202602-citibike-..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<ride_id:string,rideable_type:string,started_at:string,ended_at:string,start_station_name:s...

Details

== Physical Plan ==
AdaptiveSparkPlan (27)
+- == Final Plan ==
   * HashAggregate (23)
   +- ShuffleQueryStage (22), Statistics(sizeInBytes=32.0 B, rowCount=2)
      +- Exchange (21)
         +- * HashAggregate (20)
            +- TableCacheQueryStage (19), Statistics(sizeInBytes=260.4 MiB, rowCount=1.19E+6)
               +- InMemoryTableScan (1)
                     +- InMemoryRelation (2)
                           +- AdaptiveSparkPlan (18)
                              +- Project (17)
                                 +- Window (16)
                                    +- Sort (15)
                                       +- Exchange (14)
                                          +- Project (13)
                                             +- Filter (12)
                                                +- InMemoryTableScan (3)
                                                      +- InMemoryRelation (4)
                                                            +- AdaptiveSparkPlan (11)
                                                               +- Project (10)
                                                                  +- Project (9)
                                                                     +- InMemoryTableScan (5)
                                                                           +- InMemoryRelation (6)
                                                                                 +- * Project (8)
                                                                                    +- Scan csv  (7)
+- == Initial Plan ==
   HashAggregate (26)
   +- Exchange (25)
      +- HashAggregate (24)
         +- InMemoryTableScan (1)
               +- InMemoryRelation (2)
                     +- AdaptiveSparkPlan (18)
                        +- Project (17)
                           +- Window (16)
                              +- Sort (15)
                                 +- Exchange (14)
                                    +- Project (13)
                                       +- Filter (12)
                                          +- InMemoryTableScan (3)
                                                +- InMemoryRelation (4)
                                                      +- AdaptiveSparkPlan (11)
                                                         +- Project (10)
                                                            +- Project (9)
                                                               +- InMemoryTableScan (5)
                                                                     +- InMemoryRelation (6)
                                                                           +- * Project (8)
                                                                              +- Scan csv  (7)


(1) InMemoryTableScan
Output: []

(2) InMemoryRelation
Arguments: [ride_id#3716, rideable_type#3717, started_at#3718, ended_at#3719, start_station_name#3720, start_station_id#3721, end_station_name#3722, end_station_id#3723, start_lat#3724, start_lng#3725, end_lat#3726, end_lng#3727, member_casual#3728, _source_file#3294, _processed_dttm#13637, _start_station_ride_num#13639, year#13640, month#13641], CachedRDDBuilder(org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer@28f1e8bc,StorageLevel(disk, memory, deserialized, 1 replicas),AdaptiveSparkPlan isFinalPlan=true
+- == Final Plan ==
   *(3) Project [ride_id#3716, rideable_type#3717, started_at#3718, ended_at#3719, start_station_name#3720, start_station_id#3721, end_station_name#3722, end_station_id#3723, start_lat#3724, start_lng#3725, end_lat#3726, end_lng#3727, member_casual#3728, _source_file#3294, 2026-11-01 20:50:07.599499 AS _processed_dttm#13637, _start_station_ride_num#13639, year#13640, month#13641]
   +- Window [row_number() windowspecdefinition(start_station_id#3721, _w0#13642, _w1#13643, started_at#3718 ASC NULLS FIRST, ride_id#3716 ASC NULLS FIRST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS _start_station_ride_num#13639], [start_station_id#3721, _w0#13642, _w1#13643], [started_at#3718 ASC NULLS FIRST, ride_id#3716 ASC NULLS FIRST]
      +- *(2) Sort [start_station_id#3721 ASC NULLS FIRST, _w0#13642 ASC NULLS FIRST, _w1#13643 ASC NULLS FIRST, started_at#3718 ASC NULLS FIRST, ride_id#3716 ASC NULLS FIRST], false, 0
         +- AQEShuffleRead coalesced
            +- ShuffleQueryStage 1
               +- Exchange hashpartitioning(start_station_id#3721, _w0#13642, _w1#13643, 200), ENSURE_REQUIREMENTS, [plan_id=2956]
                  +- *(1) Project [ride_id#3716, rideable_type#3717, started_at#3718, ended_at#3719, start_station_name#3720, start_station_id#3721, end_station_name#3722, end_station_id#3723, start_lat#3724, start_lng#3725, end_lat#3726, end_lng#3727, member_casual#3728, _source_file#3294, date_format(started_at#3718, yyyy, Some(UTC)) AS year#13640, date_format(started_at#3718, MM, Some(UTC)) AS month#13641, date_format(started_at#3718, yyyy, Some(UTC)) AS _w0#13642, date_format(started_at#3718, MM, Some(UTC)) AS _w1#13643]
                     +- *(1) Filter (isnotnull(_is_valid#12611) AND _is_valid#12611)
                        +- TableCacheQueryStage 0
                           +- InMemoryTableScan [_is_valid#12611, _source_file#3294, end_lat#3726, end_lng#3727, end_station_id#3723, end_station_name#3722, ended_at#3719, member_casual#3728, ride_id#3716, rideable_type#3717, start_lat#3724, start_lng#3725, start_station_id#3721, start_station_name#3720, started_at#3718], [isnotnull(_is_valid#12611), _is_valid#12611]
                                 +- InMemoryRelation [ride_id#3716, rideable_type#3717, started_at#3718, ended_at#3719, start_station_name#3720, start_station_id#3721, end_station_name#3722, end_station_id#3723, start_lat#3724, start_lng#3725, end_lat#3726, end_lng#3727, member_casual#3728, _corrupt_record#3279, _source_file#3294, _is_ride_id_not_empty#12608, _is_ended_after_started#12609, _is_different_stations#12610, _is_valid#12611], StorageLevel(disk, memory, deserialized, 1 replicas)
                                       +- AdaptiveSparkPlan isFinalPlan=false
                                          +- Project [ride_id#128, rideable_type#129, started_at#669, ended_at#670, start_station_name#132, start_station_id#133, end_station_name#134, end_station_id#135, start_lat#675, start_lng#676, end_lat#677, end_lng#678, member_casual#140, _corrupt_record#141, _source_file#156, (isnotnull(ride_id#128) AND NOT (trim(ride_id#128, None) = )) AS _is_ride_id_not_empty#695, coalesce((ended_at#670 > started_at#669), false) AS _is_ended_after_started#696, coalesce(NOT (end_station_id#135 = start_station_id#133), false) AS _is_different_stations#697, ((((isnotnull(ride_id#128) AND NOT (trim(ride_id#128, None) = )) AND coalesce((ended_at#670 > started_at#669), false)) AND coalesce(NOT (end_station_id#135 = start_station_id#133), false)) AND isnull(_corrupt_record#141)) AS _is_valid#698]
                                             +- Project [ride_id#128, rideable_type#129, cast(started_at#130 as timestamp) AS started_at#669, cast(ended_at#131 as timestamp) AS ended_at#670, start_station_name#132, start_station_id#133, end_station_name#134, end_station_id#135, cast(start_lat#136 as double) AS start_lat#675, cast(start_lng#137 as double) AS start_lng#676, cast(end_lat#138 as double) AS end_lat#677, cast(end_lng#139 as double) AS end_lng#678, member_casual#140, _corrupt_record#141, _source_file#156]
                                                +- InMemoryTableScan [_corrupt_record#141, _source_file#156, end_lat#138, end_lng#139, end_station_id#135, end_station_name#134, ended_at#131, member_casual#140, ride_id#128, rideable_type#129, start_lat#136, start_lng#137, start_station_id#133, start_station_name#132, started_at#130]
                                                      +- InMemoryRelation [ride_id#128, rideable_type#129, started_at#130, ended_at#131, start_station_name#132, start_station_id#133, end_station_name#134, end_station_id#135, start_lat#136, start_lng#137, end_lat#138, end_lng#139, member_casual#140, _corrupt_record#141, _source_file#156], StorageLevel(disk, memory, deserialized, 1 replicas)
                                                            +- *(1) Project [ride_id#128, rideable_type#129, started_at#130, ended_at#131, start_station_name#132, start_station_id#133, end_station_name#134, end_station_id#135, start_lat#136, start_lng#137, end_lat#138, end_lng#139, member_casual#140, _corrupt_record#141, regexp_extract(input_file_name(), [^/]+$, 0) AS _source_file#156]
                                                               +- FileScan csv [ride_id#128,rideable_type#129,started_at#130,ended_at#131,start_station_name#132,start_station_id#133,end_station_name#134,end_station_id#135,start_lat#136,start_lng#137,end_lat#138,end_lng#139,member_casual#140,_corrupt_record#141] Batched: false, DataFilters: [], Format: CSV, Location: InMemoryFileIndex(2 paths)[s3a://rzvde-g11-durnev-pavel/raw/citibike_data/202602/202602-citibike-..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<ride_id:string,rideable_type:string,started_at:string,ended_at:string,start_station_name:s...
+- == Initial Plan ==
   Project [ride_id#3716, rideable_type#3717, started_at#3718, ended_at#3719, start_station_name#3720, start_station_id#3721, end_station_name#3722, end_station_id#3723, start_lat#3724, start_lng#3725, end_lat#3726, end_lng#3727, member_casual#3728, _source_file#3294, 2026-11-01 20:50:07.599499 AS _processed_dttm#13637, _start_station_ride_num#13639, year#13640, month#13641]
   +- Window [row_number() windowspecdefinition(start_station_id#3721, _w0#13642, _w1#13643, started_at#3718 ASC NULLS FIRST, ride_id#3716 ASC NULLS FIRST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS _start_station_ride_num#13639], [start_station_id#3721, _w0#13642, _w1#13643], [started_at#3718 ASC NULLS FIRST, ride_id#3716 ASC NULLS FIRST]
      +- Sort [start_station_id#3721 ASC NULLS FIRST, _w0#13642 ASC NULLS FIRST, _w1#13643 ASC NULLS FIRST, started_at#3718 ASC NULLS FIRST, ride_id#3716 ASC NULLS FIRST], false, 0
         +- Exchange hashpartitioning(start_station_id#3721, _w0#13642, _w1#13643, 200), ENSURE_REQUIREMENTS, [plan_id=2881]
            +- Project [ride_id#3716, rideable_type#3717, started_at#3718, ended_at#3719, start_station_name#3720, start_station_id#3721, end_station_name#3722, end_station_id#3723, start_lat#3724, start_lng#3725, end_lat#3726, end_lng#3727, member_casual#3728, _source_file#3294, date_format(started_at#3718, yyyy, Some(UTC)) AS year#13640, date_format(started_at#3718, MM, Some(UTC)) AS month#13641, date_format(started_at#3718, yyyy, Some(UTC)) AS _w0#13642, date_format(started_at#3718, MM, Some(UTC)) AS _w1#13643]
               +- Filter (isnotnull(_is_valid#12611) AND _is_valid#12611)
                  +- InMemoryTableScan [_is_valid#12611, _source_file#3294, end_lat#3726, end_lng#3727, end_station_id#3723, end_station_name#3722, ended_at#3719, member_casual#3728, ride_id#3716, rideable_type#3717, start_lat#3724, start_lng#3725, start_station_id#3721, start_station_name#3720, started_at#3718], [isnotnull(_is_valid#12611), _is_valid#12611]
                        +- InMemoryRelation [ride_id#3716, rideable_type#3717, started_at#3718, ended_at#3719, start_station_name#3720, start_station_id#3721, end_station_name#3722, end_station_id#3723, start_lat#3724, start_lng#3725, end_lat#3726, end_lng#3727, member_casual#3728, _corrupt_record#3279, _source_file#3294, _is_ride_id_not_empty#12608, _is_ended_after_started#12609, _is_different_stations#12610, _is_valid#12611], StorageLevel(disk, memory, deserialized, 1 replicas)
                              +- AdaptiveSparkPlan isFinalPlan=false
                                 +- Project [ride_id#128, rideable_type#129, started_at#669, ended_at#670, start_station_name#132, start_station_id#133, end_station_name#134, end_station_id#135, start_lat#675, start_lng#676, end_lat#677, end_lng#678, member_casual#140, _corrupt_record#141, _source_file#156, (isnotnull(ride_id#128) AND NOT (trim(ride_id#128, None) = )) AS _is_ride_id_not_empty#695, coalesce((ended_at#670 > started_at#669), false) AS _is_ended_after_started#696, coalesce(NOT (end_station_id#135 = start_station_id#133), false) AS _is_different_stations#697, ((((isnotnull(ride_id#128) AND NOT (trim(ride_id#128, None) = )) AND coalesce((ended_at#670 > started_at#669), false)) AND coalesce(NOT (end_station_id#135 = start_station_id#133), false)) AND isnull(_corrupt_record#141)) AS _is_valid#698]
                                    +- Project [ride_id#128, rideable_type#129, cast(started_at#130 as timestamp) AS started_at#669, cast(ended_at#131 as timestamp) AS ended_at#670, start_station_name#132, start_station_id#133, end_station_name#134, end_station_id#135, cast(start_lat#136 as double) AS start_lat#675, cast(start_lng#137 as double) AS start_lng#676, cast(end_lat#138 as double) AS end_lat#677, cast(end_lng#139 as double) AS end_lng#678, member_casual#140, _corrupt_record#141, _source_file#156]
                                       +- InMemoryTableScan [_corrupt_record#141, _source_file#156, end_lat#138, end_lng#139, end_station_id#135, end_station_name#134, ended_at#131, member_casual#140, ride_id#128, rideable_type#129, start_lat#136, start_lng#137, start_station_id#133, start_station_name#132, started_at#130]
                                             +- InMemoryRelation [ride_id#128, rideable_type#129, started_at#130, ended_at#131, start_station_name#132, start_station_id#133, end_station_name#134, end_station_id#135, start_lat#136, start_lng#137, end_lat#138, end_lng#139, member_casual#140, _corrupt_record#141, _source_file#156], StorageLevel(disk, memory, deserialized, 1 replicas)
                                                   +- *(1) Project [ride_id#128, rideable_type#129, started_at#130, ended_at#131, start_station_name#132, start_station_id#133, end_station_name#134, end_station_id#135, start_lat#136, start_lng#137, end_lat#138, end_lng#139, member_casual#140, _corrupt_record#141, regexp_extract(input_file_name(), [^/]+$, 0) AS _source_file#156]
                                                      +- FileScan csv [ride_id#128,rideable_type#129,started_at#130,ended_at#131,start_station_name#132,start_station_id#133,end_station_name#134,end_station_id#135,start_lat#136,start_lng#137,end_lat#138,end_lng#139,member_casual#140,_corrupt_record#141] Batched: false, DataFilters: [], Format: CSV, Location: InMemoryFileIndex(2 paths)[s3a://rzvde-g11-durnev-pavel/raw/citibike_data/202602/202602-citibike-..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<ride_id:string,rideable_type:string,started_at:string,ended_at:string,start_station_name:s...
,None)

(3) InMemoryTableScan
Output [15]: [_is_valid#12611, _source_file#3294, end_lat#3726, end_lng#3727, end_station_id#3723, end_station_name#3722, ended_at#3719, member_casual#3728, ride_id#3716, rideable_type#3717, start_lat#3724, start_lng#3725, start_station_id#3721, start_station_name#3720, started_at#3718]
Arguments: [_is_valid#12611, _source_file#3294, end_lat#3726, end_lng#3727, end_station_id#3723, end_station_name#3722, ended_at#3719, member_casual#3728, ride_id#3716, rideable_type#3717, start_lat#3724, start_lng#3725, start_station_id#3721, start_station_name#3720, started_at#3718], [isnotnull(_is_valid#12611), _is_valid#12611]

(4) InMemoryRelation
Arguments: [ride_id#3716, rideable_type#3717, started_at#3718, ended_at#3719, start_station_name#3720, start_station_id#3721, end_station_name#3722, end_station_id#3723, start_lat#3724, start_lng#3725, end_lat#3726, end_lng#3727, member_casual#3728, _corrupt_record#3279, _source_file#3294, _is_ride_id_not_empty#12608, _is_ended_after_started#12609, _is_different_stations#12610, _is_valid#12611], CachedRDDBuilder(org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer@28f1e8bc,StorageLevel(disk, memory, deserialized, 1 replicas),AdaptiveSparkPlan isFinalPlan=true
+- == Final Plan ==
   *(1) Project [ride_id#128, rideable_type#129, started_at#669, ended_at#670, start_station_name#132, start_station_id#133, end_station_name#134, end_station_id#135, start_lat#675, start_lng#676, end_lat#677, end_lng#678, member_casual#140, _corrupt_record#141, _source_file#156, (isnotnull(ride_id#128) AND NOT (trim(ride_id#128, None) = )) AS _is_ride_id_not_empty#695, coalesce((ended_at#670 > started_at#669), false) AS _is_ended_after_started#696, coalesce(NOT (end_station_id#135 = start_station_id#133), false) AS _is_different_stations#697, ((((isnotnull(ride_id#128) AND NOT (trim(ride_id#128, None) = )) AND coalesce((ended_at#670 > started_at#669), false)) AND coalesce(NOT (end_station_id#135 = start_station_id#133), false)) AND isnull(_corrupt_record#141)) AS _is_valid#698]
   +- *(1) Project [ride_id#128, rideable_type#129, cast(started_at#130 as timestamp) AS started_at#669, cast(ended_at#131 as timestamp) AS ended_at#670, start_station_name#132, start_station_id#133, end_station_name#134, end_station_id#135, cast(start_lat#136 as double) AS start_lat#675, cast(start_lng#137 as double) AS start_lng#676, cast(end_lat#138 as double) AS end_lat#677, cast(end_lng#139 as double) AS end_lng#678, member_casual#140, _corrupt_record#141, _source_file#156]
      +- TableCacheQueryStage 0
         +- InMemoryTableScan [_corrupt_record#141, _source_file#156, end_lat#138, end_lng#139, end_station_id#135, end_station_name#134, ended_at#131, member_casual#140, ride_id#128, rideable_type#129, start_lat#136, start_lng#137, start_station_id#133, start_station_name#132, started_at#130]
               +- InMemoryRelation [ride_id#128, rideable_type#129, started_at#130, ended_at#131, start_station_name#132, start_station_id#133, end_station_name#134, end_station_id#135, start_lat#136, start_lng#137, end_lat#138, end_lng#139, member_casual#140, _corrupt_record#141, _source_file#156], StorageLevel(disk, memory, deserialized, 1 replicas)
                     +- *(1) Project [ride_id#128, rideable_type#129, started_at#130, ended_at#131, start_station_name#132, start_station_id#133, end_station_name#134, end_station_id#135, start_lat#136, start_lng#137, end_lat#138, end_lng#139, member_casual#140, _corrupt_record#141, regexp_extract(input_file_name(), [^/]+$, 0) AS _source_file#156]
                        +- FileScan csv [ride_id#128,rideable_type#129,started_at#130,ended_at#131,start_station_name#132,start_station_id#133,end_station_name#134,end_station_id#135,start_lat#136,start_lng#137,end_lat#138,end_lng#139,member_casual#140,_corrupt_record#141] Batched: false, DataFilters: [], Format: CSV, Location: InMemoryFileIndex(2 paths)[s3a://rzvde-g11-durnev-pavel/raw/citibike_data/202602/202602-citibike-..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<ride_id:string,rideable_type:string,started_at:string,ended_at:string,start_station_name:s...
+- == Initial Plan ==
   Project [ride_id#128, rideable_type#129, started_at#669, ended_at#670, start_station_name#132, start_station_id#133, end_station_name#134, end_station_id#135, start_lat#675, start_lng#676, end_lat#677, end_lng#678, member_casual#140, _corrupt_record#141, _source_file#156, (isnotnull(ride_id#128) AND NOT (trim(ride_id#128, None) = )) AS _is_ride_id_not_empty#695, coalesce((ended_at#670 > started_at#669), false) AS _is_ended_after_started#696, coalesce(NOT (end_station_id#135 = start_station_id#133), false) AS _is_different_stations#697, ((((isnotnull(ride_id#128) AND NOT (trim(ride_id#128, None) = )) AND coalesce((ended_at#670 > started_at#669), false)) AND coalesce(NOT (end_station_id#135 = start_station_id#133), false)) AND isnull(_corrupt_record#141)) AS _is_valid#698]
   +- Project [ride_id#128, rideable_type#129, cast(started_at#130 as timestamp) AS started_at#669, cast(ended_at#131 as timestamp) AS ended_at#670, start_station_name#132, start_station_id#133, end_station_name#134, end_station_id#135, cast(start_lat#136 as double) AS start_lat#675, cast(start_lng#137 as double) AS start_lng#676, cast(end_lat#138 as double) AS end_lat#677, cast(end_lng#139 as double) AS end_lng#678, member_casual#140, _corrupt_record#141, _source_file#156]
      +- InMemoryTableScan [_corrupt_record#141, _source_file#156, end_lat#138, end_lng#139, end_station_id#135, end_station_name#134, ended_at#131, member_casual#140, ride_id#128, rideable_type#129, start_lat#136, start_lng#137, start_station_id#133, start_station_name#132, started_at#130]
            +- InMemoryRelation [ride_id#128, rideable_type#129, started_at#130, ended_at#131, start_station_name#132, start_station_id#133, end_station_name#134, end_station_id#135, start_lat#136, start_lng#137, end_lat#138, end_lng#139, member_casual#140, _corrupt_record#141, _source_file#156], StorageLevel(disk, memory, deserialized, 1 replicas)
                  +- *(1) Project [ride_id#128, rideable_type#129, started_at#130, ended_at#131, start_station_name#132, start_station_id#133, end_station_name#134, end_station_id#135, start_lat#136, start_lng#137, end_lat#138, end_lng#139, member_casual#140, _corrupt_record#141, regexp_extract(input_file_name(), [^/]+$, 0) AS _source_file#156]
                     +- FileScan csv [ride_id#128,rideable_type#129,started_at#130,ended_at#131,start_station_name#132,start_station_id#133,end_station_name#134,end_station_id#135,start_lat#136,start_lng#137,end_lat#138,end_lng#139,member_casual#140,_corrupt_record#141] Batched: false, DataFilters: [], Format: CSV, Location: InMemoryFileIndex(2 paths)[s3a://rzvde-g11-durnev-pavel/raw/citibike_data/202602/202602-citibike-..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<ride_id:string,rideable_type:string,started_at:string,ended_at:string,start_station_name:s...
,None)

(5) InMemoryTableScan
Output [15]: [_corrupt_record#141, _source_file#156, end_lat#138, end_lng#139, end_station_id#135, end_station_name#134, ended_at#131, member_casual#140, ride_id#128, rideable_type#129, start_lat#136, start_lng#137, start_station_id#133, start_station_name#132, started_at#130]
Arguments: [_corrupt_record#141, _source_file#156, end_lat#138, end_lng#139, end_station_id#135, end_station_name#134, ended_at#131, member_casual#140, ride_id#128, rideable_type#129, start_lat#136, start_lng#137, start_station_id#133, start_station_name#132, started_at#130]

(6) InMemoryRelation
Arguments: [ride_id#128, rideable_type#129, started_at#130, ended_at#131, start_station_name#132, start_station_id#133, end_station_name#134, end_station_id#135, start_lat#136, start_lng#137, end_lat#138, end_lng#139, member_casual#140, _corrupt_record#141, _source_file#156], CachedRDDBuilder(org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer@28f1e8bc,StorageLevel(disk, memory, deserialized, 1 replicas),*(1) Project [ride_id#128, rideable_type#129, started_at#130, ended_at#131, start_station_name#132, start_station_id#133, end_station_name#134, end_station_id#135, start_lat#136, start_lng#137, end_lat#138, end_lng#139, member_casual#140, _corrupt_record#141, regexp_extract(input_file_name(), [^/]+$, 0) AS _source_file#156]
+- FileScan csv [ride_id#128,rideable_type#129,started_at#130,ended_at#131,start_station_name#132,start_station_id#133,end_station_name#134,end_station_id#135,start_lat#136,start_lng#137,end_lat#138,end_lng#139,member_casual#140,_corrupt_record#141] Batched: false, DataFilters: [], Format: CSV, Location: InMemoryFileIndex(2 paths)[s3a://rzvde-g11-durnev-pavel/raw/citibike_data/202602/202602-citibike-..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<ride_id:string,rideable_type:string,started_at:string,ended_at:string,start_station_name:s...
,None)

(7) Scan csv 
Output [14]: [ride_id#128, rideable_type#129, started_at#130, ended_at#131, start_station_name#132, start_station_id#133, end_station_name#134, end_station_id#135, start_lat#136, start_lng#137, end_lat#138, end_lng#139, member_casual#140, _corrupt_record#141]
Batched: false
Location: InMemoryFileIndex [s3a://rzvde-g11-durnev-pavel/raw/citibike_data/202602/202602-citibike-tripdata-part00.csv, ... 1 entries]
ReadSchema: struct<ride_id:string,rideable_type:string,started_at:string,ended_at:string,start_station_name:string,start_station_id:string,end_station_name:string,end_station_id:string,start_lat:string,start_lng:string,end_lat:string,end_lng:string,member_casual:string,_corrupt_record:string>

(8) Project [codegen id : 1]
Output [15]: [ride_id#128, rideable_type#129, started_at#130, ended_at#131, start_station_name#132, start_station_id#133, end_station_name#134, end_station_id#135, start_lat#136, start_lng#137, end_lat#138, end_lng#139, member_casual#140, _corrupt_record#141, regexp_extract(input_file_name(), [^/]+$, 0) AS _source_file#156]
Input [14]: [ride_id#128, rideable_type#129, started_at#130, ended_at#131, start_station_name#132, start_station_id#133, end_station_name#134, end_station_id#135, start_lat#136, start_lng#137, end_lat#138, end_lng#139, member_casual#140, _corrupt_record#141]

(9) Project
Output [15]: [ride_id#128, rideable_type#129, cast(started_at#130 as timestamp) AS started_at#669, cast(ended_at#131 as timestamp) AS ended_at#670, start_station_name#132, start_station_id#133, end_station_name#134, end_station_id#135, cast(start_lat#136 as double) AS start_lat#675, cast(start_lng#137 as double) AS start_lng#676, cast(end_lat#138 as double) AS end_lat#677, cast(end_lng#139 as double) AS end_lng#678, member_casual#140, _corrupt_record#141, _source_file#156]
Input [15]: [_corrupt_record#141, _source_file#156, end_lat#138, end_lng#139, end_station_id#135, end_station_name#134, ended_at#131, member_casual#140, ride_id#128, rideable_type#129, start_lat#136, start_lng#137, start_station_id#133, start_station_name#132, started_at#130]

(10) Project
Output [19]: [ride_id#128, rideable_type#129, started_at#669, ended_at#670, start_station_name#132, start_station_id#133, end_station_name#134, end_station_id#135, start_lat#675, start_lng#676, end_lat#677, end_lng#678, member_casual#140, _corrupt_record#141, _source_file#156, (isnotnull(ride_id#128) AND NOT (trim(ride_id#128, None) = )) AS _is_ride_id_not_empty#695, coalesce((ended_at#670 > started_at#669), false) AS _is_ended_after_started#696, coalesce(NOT (end_station_id#135 = start_station_id#133), false) AS _is_different_stations#697, ((((isnotnull(ride_id#128) AND NOT (trim(ride_id#128, None) = )) AND coalesce((ended_at#670 > started_at#669), false)) AND coalesce(NOT (end_station_id#135 = start_station_id#133), false)) AND isnull(_corrupt_record#141)) AS _is_valid#698]
Input [15]: [ride_id#128, rideable_type#129, started_at#669, ended_at#670, start_station_name#132, start_station_id#133, end_station_name#134, end_station_id#135, start_lat#675, start_lng#676, end_lat#677, end_lng#678, member_casual#140, _corrupt_record#141, _source_file#156]

(11) AdaptiveSparkPlan
Output [19]: [ride_id#128, rideable_type#129, started_at#669, ended_at#670, start_station_name#132, start_station_id#133, end_station_name#134, end_station_id#135, start_lat#675, start_lng#676, end_lat#677, end_lng#678, member_casual#140, _corrupt_record#141, _source_file#156, _is_ride_id_not_empty#695, _is_ended_after_started#696, _is_different_stations#697, _is_valid#698]
Arguments: isFinalPlan=false

(12) Filter
Input [15]: [_is_valid#12611, _source_file#3294, end_lat#3726, end_lng#3727, end_station_id#3723, end_station_name#3722, ended_at#3719, member_casual#3728, ride_id#3716, rideable_type#3717, start_lat#3724, start_lng#3725, start_station_id#3721, start_station_name#3720, started_at#3718]
Condition : (isnotnull(_is_valid#12611) AND _is_valid#12611)

(13) Project
Output [18]: [ride_id#3716, rideable_type#3717, started_at#3718, ended_at#3719, start_station_name#3720, start_station_id#3721, end_station_name#3722, end_station_id#3723, start_lat#3724, start_lng#3725, end_lat#3726, end_lng#3727, member_casual#3728, _source_file#3294, date_format(started_at#3718, yyyy, Some(UTC)) AS year#13640, date_format(started_at#3718, MM, Some(UTC)) AS month#13641, date_format(started_at#3718, yyyy, Some(UTC)) AS _w0#13642, date_format(started_at#3718, MM, Some(UTC)) AS _w1#13643]
Input [15]: [_is_valid#12611, _source_file#3294, end_lat#3726, end_lng#3727, end_station_id#3723, end_station_name#3722, ended_at#3719, member_casual#3728, ride_id#3716, rideable_type#3717, start_lat#3724, start_lng#3725, start_station_id#3721, start_station_name#3720, started_at#3718]

(14) Exchange
Input [18]: [ride_id#3716, rideable_type#3717, started_at#3718, ended_at#3719, start_station_name#3720, start_station_id#3721, end_station_name#3722, end_station_id#3723, start_lat#3724, start_lng#3725, end_lat#3726, end_lng#3727, member_casual#3728, _source_file#3294, year#13640, month#13641, _w0#13642, _w1#13643]
Arguments: hashpartitioning(start_station_id#3721, _w0#13642, _w1#13643, 200), ENSURE_REQUIREMENTS, [plan_id=2909]

(15) Sort
Input [18]: [ride_id#3716, rideable_type#3717, started_at#3718, ended_at#3719, start_station_name#3720, start_station_id#3721, end_station_name#3722, end_station_id#3723, start_lat#3724, start_lng#3725, end_lat#3726, end_lng#3727, member_casual#3728, _source_file#3294, year#13640, month#13641, _w0#13642, _w1#13643]
Arguments: [start_station_id#3721 ASC NULLS FIRST, _w0#13642 ASC NULLS FIRST, _w1#13643 ASC NULLS FIRST, started_at#3718 ASC NULLS FIRST, ride_id#3716 ASC NULLS FIRST], false, 0

(16) Window
Input [18]: [ride_id#3716, rideable_type#3717, started_at#3718, ended_at#3719, start_station_name#3720, start_station_id#3721, end_station_name#3722, end_station_id#3723, start_lat#3724, start_lng#3725, end_lat#3726, end_lng#3727, member_casual#3728, _source_file#3294, year#13640, month#13641, _w0#13642, _w1#13643]
Arguments: [row_number() windowspecdefinition(start_station_id#3721, _w0#13642, _w1#13643, started_at#3718 ASC NULLS FIRST, ride_id#3716 ASC NULLS FIRST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS _start_station_ride_num#13639], [start_station_id#3721, _w0#13642, _w1#13643], [started_at#3718 ASC NULLS FIRST, ride_id#3716 ASC NULLS FIRST]

(17) Project
Output [18]: [ride_id#3716, rideable_type#3717, started_at#3718, ended_at#3719, start_station_name#3720, start_station_id#3721, end_station_name#3722, end_station_id#3723, start_lat#3724, start_lng#3725, end_lat#3726, end_lng#3727, member_casual#3728, _source_file#3294, 2026-11-01 20:50:07.599499 AS _processed_dttm#13637, _start_station_ride_num#13639, year#13640, month#13641]
Input [19]: [ride_id#3716, rideable_type#3717, started_at#3718, ended_at#3719, start_station_name#3720, start_station_id#3721, end_station_name#3722, end_station_id#3723, start_lat#3724, start_lng#3725, end_lat#3726, end_lng#3727, member_casual#3728, _source_file#3294, year#13640, month#13641, _w0#13642, _w1#13643, _start_station_ride_num#13639]

(18) AdaptiveSparkPlan
Output [18]: [ride_id#3716, rideable_type#3717, started_at#3718, ended_at#3719, start_station_name#3720, start_station_id#3721, end_station_name#3722, end_station_id#3723, start_lat#3724, start_lng#3725, end_lat#3726, end_lng#3727, member_casual#3728, _source_file#3294, _processed_dttm#13637, _start_station_ride_num#13639, year#13640, month#13641]
Arguments: isFinalPlan=false

(19) TableCacheQueryStage
Output: []
Arguments: 0

(20) HashAggregate [codegen id : 1]
Input: []
Keys: []
Functions [1]: [partial_count(1)]
Aggregate Attributes [1]: [count#14734L]
Results [1]: [count#14735L]

(21) Exchange
Input [1]: [count#14735L]
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=3003]

(22) ShuffleQueryStage
Output [1]: [count#14735L]
Arguments: 1

(23) HashAggregate [codegen id : 2]
Input [1]: [count#14735L]
Keys: []
Functions [1]: [count(1)]
Aggregate Attributes [1]: [count(1)#14461L]
Results [1]: [count(1)#14461L AS count#14462L]

(24) HashAggregate
Input: []
Keys: []
Functions [1]: [partial_count(1)]
Aggregate Attributes [1]: [count#14734L]
Results [1]: [count#14735L]

(25) Exchange
Input [1]: [count#14735L]
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=2906]

(26) HashAggregate
Input [1]: [count#14735L]
Keys: []
Functions [1]: [count(1)]
Aggregate Attributes [1]: [count(1)#14461L]
Results [1]: [count(1)#14461L AS count#14462L]

(27) AdaptiveSparkPlan
Output [1]: [count#14462L]
Arguments: isFinalPlan=true