citibike-kryukov-egor-st2 - Details for Query 3

Submitted Time: 2026/11/06 14:13:39
Duration: 20 s
Succeeded Jobs: 4 5

Show the Stage ID and Task ID that corresponds to the max metric

digraph G { 0 [labelType="html" label=" AdaptiveSparkPlan "]; 1 [labelType="html" label=" CollectLimit "]; subgraph cluster2 { isCluster="true"; label="WholeStageCodegen (3)\n \nduration: 0 ms"; 3 [labelType="html" label=" Project "]; } 4 [labelType="html" label="Window spill size: 0.0 B"]; subgraph cluster5 { isCluster="true"; label="WholeStageCodegen (2)\n \nduration: 0 ms"; 6 [labelType="html" label="Sort sort time: 5.2 s peak memory: 512.0 MiB spill size: 0.0 B"]; } 7 [labelType="html" label="AQEShuffleRead number of partitions: 2 partition data size total (min, med, max (stageId: taskId)) 128.1 MiB (63.3 MiB, 64.8 MiB, 64.8 MiB (driver)) number of coalesced partitions: 2"]; 8 [labelType="html" label="Exchange shuffle records written: 4,230,360 local merged chunks fetched: 0 shuffle write time total (min, med, max (stageId: taskId)) 657 ms (0 ms, 104 ms, 125 ms (stage 5.0: task 12)) remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 2,092,252 local bytes read: 60.4 MiB fetch wait time: 0 ms remote bytes read: 0.0 B merged fetch fallback count: 0 local blocks read: 7 remote merged chunks fetched: 0 remote blocks read: 0 data size total (min, med, max (stageId: taskId)) 871.4 MiB (0.0 B, 146.5 MiB, 147.6 MiB (stage 5.0: task 12)) local merged bytes read: 0.0 B number of partitions: 200 remote reqs duration: 0 ms remote bytes read to disk: 0.0 B shuffle bytes written total (min, med, max (stageId: taskId)) 122.0 MiB (0.0 B, 20.5 MiB, 20.7 MiB (stage 5.0: task 12))"]; subgraph cluster9 { isCluster="true"; label="WholeStageCodegen (1)\n \nduration: total (min, med, max (stageId: taskId))\n24.2 s (0 ms, 3.9 s, 4.2 s (stage 5.0: task 11))"; 10 [labelType="html" label=" Project "]; 11 [labelType="html" label=" Project "]; } 12 [labelType="html" label="Scan csv number of output rows: 4,230,360 number of files read: 5 metadata time total (min, med, max (stageId: taskId)) 0 ms (0 ms, 0 ms, 0 ms (stage 7.0: task 17)) size of files read total (min, med, max (stageId: taskId)) 760.1 MiB (0.0 B, 760.1 MiB, 760.1 MiB (driver))"]; 1->0; 3->1; 4->3; 6->4; 7->6; 8->7; 10->8; 11->10; 12->11; }

AdaptiveSparkPlan isFinalPlan=true

CollectLimit 6

Project [toprettystring(ride_id#17, Some(Etc/UTC)) AS toprettystring(ride_id)#204, toprettystring(is_valid_ride_id#154, Some(Etc/UTC)) AS toprettystring(is_valid_ride_id)#205, toprettystring(is_valid_duration#155, Some(Etc/UTC)) AS toprettystring(is_valid_duration)#206, toprettystring(is_valid_stations#156, Some(Etc/UTC)) AS toprettystring(is_valid_stations)#207, toprettystring(_source_file#157, Some(Etc/UTC)) AS toprettystring(_source_file)#208, 2026-11-06 14:13:39.7842 AS toprettystring(_processed_dttm)#209, toprettystring(_start_station_ride_num#160, Some(Etc/UTC)) AS toprettystring(_start_station_ride_num)#210, toprettystring(year#161, Some(Etc/UTC)) AS toprettystring(year)#211, toprettystring(month#162, Some(Etc/UTC)) AS toprettystring(month)#212]

WholeStageCodegen (3)

Window [row_number() windowspecdefinition(start_station_id#22, _w0#163, _w1#164, started_at#72 ASC NULLS FIRST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS _start_station_ride_num#160], [start_station_id#22, _w0#163, _w1#164], [started_at#72 ASC NULLS FIRST]

Sort [start_station_id#22 ASC NULLS FIRST, _w0#163 ASC NULLS FIRST, _w1#164 ASC NULLS FIRST, started_at#72 ASC NULLS FIRST], false, 0

WholeStageCodegen (2)

AQEShuffleRead coalesced

Exchange hashpartitioning(start_station_id#22, _w0#163, _w1#164, 200), ENSURE_REQUIREMENTS, [plan_id=98]

Project [ride_id#17, started_at#72, start_station_id#22, (isnotnull(ride_id#17) AND NOT (trim(ride_id#17, None) = )) AS is_valid_ride_id#154, ((isnotnull(started_at#72) AND isnotnull(ended_at#73)) AND (ended_at#73 > started_at#72)) AS is_valid_duration#155, ((isnotnull(start_station_id#22) AND isnotnull(end_station_id#24)) AND NOT (end_station_id#24 = start_station_id#22)) AS is_valid_stations#156, input_file_name() AS _source_file#157, year(cast(started_at#72 as date)) AS year#161, month(cast(started_at#72 as date)) AS month#162, year(cast(started_at#72 as date)) AS _w0#163, month(cast(started_at#72 as date)) AS _w1#164]

Project [ride_id#17, cast(started_at#19 as timestamp) AS started_at#72, cast(ended_at#20 as timestamp) AS ended_at#73, start_station_id#22, end_station_id#24]

WholeStageCodegen (1)

FileScan csv [ride_id#17,started_at#19,ended_at#20,start_station_id#22,end_station_id#24] Batched: false, DataFilters: [], Format: CSV, Location: InMemoryFileIndex(5 paths)[s3a://rzvde-g12-kryukov-egor/raw/citibike_data/202405/202405-citibike-..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<ride_id:string,started_at:string,ended_at:string,start_station_id:string,end_station_id:st...

Details

== Physical Plan ==
AdaptiveSparkPlan (18)
+- == Final Plan ==
   CollectLimit (10)
   +- * Project (9)
      +- Window (8)
         +- * Sort (7)
            +- AQEShuffleRead (6)
               +- ShuffleQueryStage (5), Statistics(sizeInBytes=871.4 MiB, rowCount=4.23E+6)
                  +- Exchange (4)
                     +- * Project (3)
                        +- * Project (2)
                           +- Scan csv  (1)
+- == Initial Plan ==
   CollectLimit (17)
   +- Project (16)
      +- Window (15)
         +- Sort (14)
            +- Exchange (13)
               +- Project (12)
                  +- Project (11)
                     +- Scan csv  (1)


(1) Scan csv 
Output [5]: [ride_id#17, started_at#19, ended_at#20, start_station_id#22, end_station_id#24]
Batched: false
Location: InMemoryFileIndex [s3a://rzvde-g12-kryukov-egor/raw/citibike_data/202405/202405-citibike-tripdata-part00.csv, ... 4 entries]
ReadSchema: struct<ride_id:string,started_at:string,ended_at:string,start_station_id:string,end_station_id:string>

(2) Project [codegen id : 1]
Output [5]: [ride_id#17, cast(started_at#19 as timestamp) AS started_at#72, cast(ended_at#20 as timestamp) AS ended_at#73, start_station_id#22, end_station_id#24]
Input [5]: [ride_id#17, started_at#19, ended_at#20, start_station_id#22, end_station_id#24]

(3) Project [codegen id : 1]
Output [11]: [ride_id#17, started_at#72, start_station_id#22, (isnotnull(ride_id#17) AND NOT (trim(ride_id#17, None) = )) AS is_valid_ride_id#154, ((isnotnull(started_at#72) AND isnotnull(ended_at#73)) AND (ended_at#73 > started_at#72)) AS is_valid_duration#155, ((isnotnull(start_station_id#22) AND isnotnull(end_station_id#24)) AND NOT (end_station_id#24 = start_station_id#22)) AS is_valid_stations#156, input_file_name() AS _source_file#157, year(cast(started_at#72 as date)) AS year#161, month(cast(started_at#72 as date)) AS month#162, year(cast(started_at#72 as date)) AS _w0#163, month(cast(started_at#72 as date)) AS _w1#164]
Input [5]: [ride_id#17, started_at#72, ended_at#73, start_station_id#22, end_station_id#24]

(4) Exchange
Input [11]: [ride_id#17, started_at#72, start_station_id#22, is_valid_ride_id#154, is_valid_duration#155, is_valid_stations#156, _source_file#157, year#161, month#162, _w0#163, _w1#164]
Arguments: hashpartitioning(start_station_id#22, _w0#163, _w1#164, 200), ENSURE_REQUIREMENTS, [plan_id=98]

(5) ShuffleQueryStage
Output [11]: [ride_id#17, started_at#72, start_station_id#22, is_valid_ride_id#154, is_valid_duration#155, is_valid_stations#156, _source_file#157, year#161, month#162, _w0#163, _w1#164]
Arguments: 0

(6) AQEShuffleRead
Input [11]: [ride_id#17, started_at#72, start_station_id#22, is_valid_ride_id#154, is_valid_duration#155, is_valid_stations#156, _source_file#157, year#161, month#162, _w0#163, _w1#164]
Arguments: coalesced

(7) Sort [codegen id : 2]
Input [11]: [ride_id#17, started_at#72, start_station_id#22, is_valid_ride_id#154, is_valid_duration#155, is_valid_stations#156, _source_file#157, year#161, month#162, _w0#163, _w1#164]
Arguments: [start_station_id#22 ASC NULLS FIRST, _w0#163 ASC NULLS FIRST, _w1#164 ASC NULLS FIRST, started_at#72 ASC NULLS FIRST], false, 0

(8) Window
Input [11]: [ride_id#17, started_at#72, start_station_id#22, is_valid_ride_id#154, is_valid_duration#155, is_valid_stations#156, _source_file#157, year#161, month#162, _w0#163, _w1#164]
Arguments: [row_number() windowspecdefinition(start_station_id#22, _w0#163, _w1#164, started_at#72 ASC NULLS FIRST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS _start_station_ride_num#160], [start_station_id#22, _w0#163, _w1#164], [started_at#72 ASC NULLS FIRST]

(9) Project [codegen id : 3]
Output [9]: [toprettystring(ride_id#17, Some(Etc/UTC)) AS toprettystring(ride_id)#204, toprettystring(is_valid_ride_id#154, Some(Etc/UTC)) AS toprettystring(is_valid_ride_id)#205, toprettystring(is_valid_duration#155, Some(Etc/UTC)) AS toprettystring(is_valid_duration)#206, toprettystring(is_valid_stations#156, Some(Etc/UTC)) AS toprettystring(is_valid_stations)#207, toprettystring(_source_file#157, Some(Etc/UTC)) AS toprettystring(_source_file)#208, 2026-11-06 14:13:39.7842 AS toprettystring(_processed_dttm)#209, toprettystring(_start_station_ride_num#160, Some(Etc/UTC)) AS toprettystring(_start_station_ride_num)#210, toprettystring(year#161, Some(Etc/UTC)) AS toprettystring(year)#211, toprettystring(month#162, Some(Etc/UTC)) AS toprettystring(month)#212]
Input [12]: [ride_id#17, started_at#72, start_station_id#22, is_valid_ride_id#154, is_valid_duration#155, is_valid_stations#156, _source_file#157, year#161, month#162, _w0#163, _w1#164, _start_station_ride_num#160]

(10) CollectLimit
Input [9]: [toprettystring(ride_id)#204, toprettystring(is_valid_ride_id)#205, toprettystring(is_valid_duration)#206, toprettystring(is_valid_stations)#207, toprettystring(_source_file)#208, toprettystring(_processed_dttm)#209, toprettystring(_start_station_ride_num)#210, toprettystring(year)#211, toprettystring(month)#212]
Arguments: 6

(11) Project
Output [5]: [ride_id#17, cast(started_at#19 as timestamp) AS started_at#72, cast(ended_at#20 as timestamp) AS ended_at#73, start_station_id#22, end_station_id#24]
Input [5]: [ride_id#17, started_at#19, ended_at#20, start_station_id#22, end_station_id#24]

(12) Project
Output [11]: [ride_id#17, started_at#72, start_station_id#22, (isnotnull(ride_id#17) AND NOT (trim(ride_id#17, None) = )) AS is_valid_ride_id#154, ((isnotnull(started_at#72) AND isnotnull(ended_at#73)) AND (ended_at#73 > started_at#72)) AS is_valid_duration#155, ((isnotnull(start_station_id#22) AND isnotnull(end_station_id#24)) AND NOT (end_station_id#24 = start_station_id#22)) AS is_valid_stations#156, input_file_name() AS _source_file#157, year(cast(started_at#72 as date)) AS year#161, month(cast(started_at#72 as date)) AS month#162, year(cast(started_at#72 as date)) AS _w0#163, month(cast(started_at#72 as date)) AS _w1#164]
Input [5]: [ride_id#17, started_at#72, ended_at#73, start_station_id#22, end_station_id#24]

(13) Exchange
Input [11]: [ride_id#17, started_at#72, start_station_id#22, is_valid_ride_id#154, is_valid_duration#155, is_valid_stations#156, _source_file#157, year#161, month#162, _w0#163, _w1#164]
Arguments: hashpartitioning(start_station_id#22, _w0#163, _w1#164, 200), ENSURE_REQUIREMENTS, [plan_id=82]

(14) Sort
Input [11]: [ride_id#17, started_at#72, start_station_id#22, is_valid_ride_id#154, is_valid_duration#155, is_valid_stations#156, _source_file#157, year#161, month#162, _w0#163, _w1#164]
Arguments: [start_station_id#22 ASC NULLS FIRST, _w0#163 ASC NULLS FIRST, _w1#164 ASC NULLS FIRST, started_at#72 ASC NULLS FIRST], false, 0

(15) Window
Input [11]: [ride_id#17, started_at#72, start_station_id#22, is_valid_ride_id#154, is_valid_duration#155, is_valid_stations#156, _source_file#157, year#161, month#162, _w0#163, _w1#164]
Arguments: [row_number() windowspecdefinition(start_station_id#22, _w0#163, _w1#164, started_at#72 ASC NULLS FIRST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS _start_station_ride_num#160], [start_station_id#22, _w0#163, _w1#164], [started_at#72 ASC NULLS FIRST]

(16) Project
Output [9]: [toprettystring(ride_id#17, Some(Etc/UTC)) AS toprettystring(ride_id)#204, toprettystring(is_valid_ride_id#154, Some(Etc/UTC)) AS toprettystring(is_valid_ride_id)#205, toprettystring(is_valid_duration#155, Some(Etc/UTC)) AS toprettystring(is_valid_duration)#206, toprettystring(is_valid_stations#156, Some(Etc/UTC)) AS toprettystring(is_valid_stations)#207, toprettystring(_source_file#157, Some(Etc/UTC)) AS toprettystring(_source_file)#208, 2026-11-06 14:13:39.7842 AS toprettystring(_processed_dttm)#209, toprettystring(_start_station_ride_num#160, Some(Etc/UTC)) AS toprettystring(_start_station_ride_num)#210, toprettystring(year#161, Some(Etc/UTC)) AS toprettystring(year)#211, toprettystring(month#162, Some(Etc/UTC)) AS toprettystring(month)#212]
Input [12]: [ride_id#17, started_at#72, start_station_id#22, is_valid_ride_id#154, is_valid_duration#155, is_valid_stations#156, _source_file#157, year#161, month#162, _w0#163, _w1#164, _start_station_ride_num#160]

(17) CollectLimit
Input [9]: [toprettystring(ride_id)#204, toprettystring(is_valid_ride_id)#205, toprettystring(is_valid_duration)#206, toprettystring(is_valid_stations)#207, toprettystring(_source_file)#208, toprettystring(_processed_dttm)#209, toprettystring(_start_station_ride_num)#210, toprettystring(year)#211, toprettystring(month)#212]
Arguments: 6

(18) AdaptiveSparkPlan
Output [9]: [toprettystring(ride_id)#204, toprettystring(is_valid_ride_id)#205, toprettystring(is_valid_duration)#206, toprettystring(is_valid_stations)#207, toprettystring(_source_file)#208, toprettystring(_processed_dttm)#209, toprettystring(_start_station_ride_num)#210, toprettystring(year)#211, toprettystring(month)#212]
Arguments: isFinalPlan=true