citibike-kryukov-egor-st2 - Details for Query 8

Submitted Time: 2026/11/06 14:16:48
Duration: 1.1 min
Succeeded Jobs: 14 15

Show the Stage ID and Task ID that corresponds to the max metric

digraph G { 0 [labelType="html" label=" AdaptiveSparkPlan "]; 1 [labelType="html" label="Execute InsertIntoHadoopFsRelationCommand task commit time total (min, med, max (stageId: taskId)) 11.3 s (1.6 s, 1.9 s, 2.0 s (stage 22.0: task 65)) number of written files: 6 job commit time: 19.4 s number of output rows: 182,197 number of dynamic part: 0 written output: 58.9 MiB"]; 2 [labelType="html" label=" WriteFiles "]; subgraph cluster3 { isCluster="true"; label="WholeStageCodegen (3)\n \nduration: total (min, med, max (stageId: taskId))\n9.0 s (614 ms, 1.6 s, 2.0 s (stage 22.0: task 62))"; 4 [labelType="html" label=" Project "]; 5 [labelType="html" label="Filter number of output rows: 182,197"]; } 6 [labelType="html" label="Window spill size total (min, med, max (stageId: taskId)) 0.0 B (0.0 B, 0.0 B, 0.0 B (stage 22.0: task 62))"]; subgraph cluster7 { isCluster="true"; label="WholeStageCodegen (2)\n \nduration: total (min, med, max (stageId: taskId))\n21.0 s (1.4 s, 3.7 s, 4.5 s (stage 22.0: task 62))"; 8 [labelType="html" label="Sort sort time total (min, med, max (stageId: taskId)) 9.0 s (610 ms, 1.6 s, 1.8 s (stage 22.0: task 63)) peak memory total (min, med, max (stageId: taskId)) 1736.0 MiB (136.0 MiB, 320.0 MiB, 320.0 MiB (stage 22.0: task 62)) spill size total (min, med, max (stageId: taskId)) 0.0 B (0.0 B, 0.0 B, 0.0 B (stage 22.0: task 62))"]; } 9 [labelType="html" label="AQEShuffleRead number of partitions: 6 partition data size total (min, med, max (stageId: taskId)) 339.4 MiB (24.2 MiB, 63.5 MiB, 63.8 MiB (driver)) number of coalesced partitions: 6"]; 10 [labelType="html" label="Exchange shuffle records written: 4,230,360 local merged chunks fetched: 0 shuffle write time total (min, med, max (stageId: taskId)) 1.2 s (0 ms, 96 ms, 216 ms (stage 20.0: task 56)) remote merged bytes read total (min, med, max (stageId: taskId)) 0.0 B (0.0 B, 0.0 B, 0.0 B (stage 22.0: task 62)) local merged blocks fetched: 0 corrupt merged block chunks: 0 remote merged reqs duration total (min, med, max (stageId: taskId)) 0 ms (0 ms, 0 ms, 0 ms (stage 22.0: task 62)) remote merged blocks fetched: 0 records read: 4,230,360 local bytes read total (min, med, max (stageId: taskId)) 323.4 MiB (23.1 MiB, 60.5 MiB, 60.7 MiB (stage 22.0: task 62)) fetch wait time total (min, med, max (stageId: taskId)) 0 ms (0 ms, 0 ms, 0 ms (stage 22.0: task 62)) remote bytes read total (min, med, max (stageId: taskId)) 0.0 B (0.0 B, 0.0 B, 0.0 B (stage 22.0: task 62)) merged fetch fallback count: 0 local blocks read: 42 remote merged chunks fetched: 0 remote blocks read: 0 data size total (min, med, max (stageId: taskId)) 1507.1 MiB (0.0 B, 82.1 MiB, 255.1 MiB (stage 20.0: task 57)) local merged bytes read total (min, med, max (stageId: taskId)) 0.0 B (0.0 B, 0.0 B, 0.0 B (stage 22.0: task 62)) number of partitions: 200 remote reqs duration total (min, med, max (stageId: taskId)) 0 ms (0 ms, 0 ms, 0 ms (stage 22.0: task 62)) remote bytes read to disk total (min, med, max (stageId: taskId)) 0.0 B (0.0 B, 0.0 B, 0.0 B (stage 22.0: task 62)) shuffle bytes written total (min, med, max (stageId: taskId)) 323.4 MiB (0.0 B, 17.5 MiB, 58.5 MiB (stage 20.0: task 55))"]; subgraph cluster11 { isCluster="true"; label="WholeStageCodegen (1)\n \nduration: total (min, med, max (stageId: taskId))\n36.4 s (0 ms, 2.0 s, 6.1 s (stage 20.0: task 58))"; 12 [labelType="html" label=" Project "]; 13 [labelType="html" label=" Project "]; } 14 [labelType="html" label="Scan csv number of output rows: 4,230,360 number of files read: 5 metadata time total (min, med, max (stageId: taskId)) 0 ms (0 ms, 0 ms, 0 ms (stage 22.0: task 62)) size of files read total (min, med, max (stageId: taskId)) 760.1 MiB (0.0 B, 0.0 B, 760.1 MiB (driver))"]; 1->0; 2->1; 4->2; 5->4; 6->5; 8->6; 9->8; 10->9; 12->10; 13->12; 14->13; }

AdaptiveSparkPlan isFinalPlan=true

Execute InsertIntoHadoopFsRelationCommand s3a://rzvde-g12-kryukov-egor/raw_invalid/citibike_data/202405, false, CSV, [header=true, path=s3a://rzvde-g12-kryukov-egor/raw_invalid/citibike_data/202405/], Overwrite, [ride_id, rideable_type, started_at, ended_at, start_station_name, start_station_id, end_station_name, end_station_id, start_lat, start_lng, end_lat, end_lng, member_casual, is_valid_ride_id, is_valid_duration, is_valid_stations, _source_file, _processed_dttm, _start_station_ride_num, year, month]

WriteFiles

Filter ((NOT is_valid_ride_id#154 OR NOT is_valid_duration#155) OR NOT is_valid_stations#156)

WholeStageCodegen (3)

Window [row_number() windowspecdefinition(start_station_id#22, _w0#163, _w1#164, started_at#72 ASC NULLS FIRST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS _start_station_ride_num#160], [start_station_id#22, _w0#163, _w1#164], [started_at#72 ASC NULLS FIRST]

Sort [start_station_id#22 ASC NULLS FIRST, _w0#163 ASC NULLS FIRST, _w1#164 ASC NULLS FIRST, started_at#72 ASC NULLS FIRST], false, 0

WholeStageCodegen (2)

AQEShuffleRead coalesced

Exchange hashpartitioning(start_station_id#22, _w0#163, _w1#164, 200), ENSURE_REQUIREMENTS, [plan_id=420]

Project [ride_id#17, rideable_type#18, started_at#72, ended_at#73, start_station_name#21, start_station_id#22, end_station_name#23, end_station_id#24, start_lat#67, start_lng#68, end_lat#69, end_lng#70, member_casual#29, (isnotnull(ride_id#17) AND NOT (trim(ride_id#17, None) = )) AS is_valid_ride_id#154, ((isnotnull(started_at#72) AND isnotnull(ended_at#73)) AND (ended_at#73 > started_at#72)) AS is_valid_duration#155, ((isnotnull(start_station_id#22) AND isnotnull(end_station_id#24)) AND NOT (end_station_id#24 = start_station_id#22)) AS is_valid_stations#156, input_file_name() AS _source_file#157, year(cast(started_at#72 as date)) AS year#161, month(cast(started_at#72 as date)) AS month#162, year(cast(started_at#72 as date)) AS _w0#163, month(cast(started_at#72 as date)) AS _w1#164]

Project [ride_id#17, rideable_type#18, cast(started_at#19 as timestamp) AS started_at#72, cast(ended_at#20 as timestamp) AS ended_at#73, start_station_name#21, start_station_id#22, end_station_name#23, end_station_id#24, cast(start_lat#25 as double) AS start_lat#67, cast(start_lng#26 as double) AS start_lng#68, cast(end_lat#27 as double) AS end_lat#69, cast(end_lng#28 as double) AS end_lng#70, member_casual#29]

WholeStageCodegen (1)

FileScan csv [ride_id#17,rideable_type#18,started_at#19,ended_at#20,start_station_name#21,start_station_id#22,end_station_name#23,end_station_id#24,start_lat#25,start_lng#26,end_lat#27,end_lng#28,member_casual#29] Batched: false, DataFilters: [], Format: CSV, Location: InMemoryFileIndex(5 paths)[s3a://rzvde-g12-kryukov-egor/raw/citibike_data/202405/202405-citibike-..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<ride_id:string,rideable_type:string,started_at:string,ended_at:string,start_station_name:s...

Details

== Physical Plan ==
AdaptiveSparkPlan (22)
+- == Final Plan ==
   Execute InsertIntoHadoopFsRelationCommand (12)
   +- WriteFiles (11)
      +- * Project (10)
         +- * Filter (9)
            +- Window (8)
               +- * Sort (7)
                  +- AQEShuffleRead (6)
                     +- ShuffleQueryStage (5), Statistics(sizeInBytes=1507.1 MiB, rowCount=4.23E+6)
                        +- Exchange (4)
                           +- * Project (3)
                              +- * Project (2)
                                 +- Scan csv  (1)
+- == Initial Plan ==
   Execute InsertIntoHadoopFsRelationCommand (21)
   +- WriteFiles (20)
      +- Project (19)
         +- Filter (18)
            +- Window (17)
               +- Sort (16)
                  +- Exchange (15)
                     +- Project (14)
                        +- Project (13)
                           +- Scan csv  (1)


(1) Scan csv 
Output [13]: [ride_id#17, rideable_type#18, started_at#19, ended_at#20, start_station_name#21, start_station_id#22, end_station_name#23, end_station_id#24, start_lat#25, start_lng#26, end_lat#27, end_lng#28, member_casual#29]
Batched: false
Location: InMemoryFileIndex [s3a://rzvde-g12-kryukov-egor/raw/citibike_data/202405/202405-citibike-tripdata-part00.csv, ... 4 entries]
ReadSchema: struct<ride_id:string,rideable_type:string,started_at:string,ended_at:string,start_station_name:string,start_station_id:string,end_station_name:string,end_station_id:string,start_lat:string,start_lng:string,end_lat:string,end_lng:string,member_casual:string>

(2) Project [codegen id : 1]
Output [13]: [ride_id#17, rideable_type#18, cast(started_at#19 as timestamp) AS started_at#72, cast(ended_at#20 as timestamp) AS ended_at#73, start_station_name#21, start_station_id#22, end_station_name#23, end_station_id#24, cast(start_lat#25 as double) AS start_lat#67, cast(start_lng#26 as double) AS start_lng#68, cast(end_lat#27 as double) AS end_lat#69, cast(end_lng#28 as double) AS end_lng#70, member_casual#29]
Input [13]: [ride_id#17, rideable_type#18, started_at#19, ended_at#20, start_station_name#21, start_station_id#22, end_station_name#23, end_station_id#24, start_lat#25, start_lng#26, end_lat#27, end_lng#28, member_casual#29]

(3) Project [codegen id : 1]
Output [21]: [ride_id#17, rideable_type#18, started_at#72, ended_at#73, start_station_name#21, start_station_id#22, end_station_name#23, end_station_id#24, start_lat#67, start_lng#68, end_lat#69, end_lng#70, member_casual#29, (isnotnull(ride_id#17) AND NOT (trim(ride_id#17, None) = )) AS is_valid_ride_id#154, ((isnotnull(started_at#72) AND isnotnull(ended_at#73)) AND (ended_at#73 > started_at#72)) AS is_valid_duration#155, ((isnotnull(start_station_id#22) AND isnotnull(end_station_id#24)) AND NOT (end_station_id#24 = start_station_id#22)) AS is_valid_stations#156, input_file_name() AS _source_file#157, year(cast(started_at#72 as date)) AS year#161, month(cast(started_at#72 as date)) AS month#162, year(cast(started_at#72 as date)) AS _w0#163, month(cast(started_at#72 as date)) AS _w1#164]
Input [13]: [ride_id#17, rideable_type#18, started_at#72, ended_at#73, start_station_name#21, start_station_id#22, end_station_name#23, end_station_id#24, start_lat#67, start_lng#68, end_lat#69, end_lng#70, member_casual#29]

(4) Exchange
Input [21]: [ride_id#17, rideable_type#18, started_at#72, ended_at#73, start_station_name#21, start_station_id#22, end_station_name#23, end_station_id#24, start_lat#67, start_lng#68, end_lat#69, end_lng#70, member_casual#29, is_valid_ride_id#154, is_valid_duration#155, is_valid_stations#156, _source_file#157, year#161, month#162, _w0#163, _w1#164]
Arguments: hashpartitioning(start_station_id#22, _w0#163, _w1#164, 200), ENSURE_REQUIREMENTS, [plan_id=420]

(5) ShuffleQueryStage
Output [21]: [ride_id#17, rideable_type#18, started_at#72, ended_at#73, start_station_name#21, start_station_id#22, end_station_name#23, end_station_id#24, start_lat#67, start_lng#68, end_lat#69, end_lng#70, member_casual#29, is_valid_ride_id#154, is_valid_duration#155, is_valid_stations#156, _source_file#157, year#161, month#162, _w0#163, _w1#164]
Arguments: 0

(6) AQEShuffleRead
Input [21]: [ride_id#17, rideable_type#18, started_at#72, ended_at#73, start_station_name#21, start_station_id#22, end_station_name#23, end_station_id#24, start_lat#67, start_lng#68, end_lat#69, end_lng#70, member_casual#29, is_valid_ride_id#154, is_valid_duration#155, is_valid_stations#156, _source_file#157, year#161, month#162, _w0#163, _w1#164]
Arguments: coalesced

(7) Sort [codegen id : 2]
Input [21]: [ride_id#17, rideable_type#18, started_at#72, ended_at#73, start_station_name#21, start_station_id#22, end_station_name#23, end_station_id#24, start_lat#67, start_lng#68, end_lat#69, end_lng#70, member_casual#29, is_valid_ride_id#154, is_valid_duration#155, is_valid_stations#156, _source_file#157, year#161, month#162, _w0#163, _w1#164]
Arguments: [start_station_id#22 ASC NULLS FIRST, _w0#163 ASC NULLS FIRST, _w1#164 ASC NULLS FIRST, started_at#72 ASC NULLS FIRST], false, 0

(8) Window
Input [21]: [ride_id#17, rideable_type#18, started_at#72, ended_at#73, start_station_name#21, start_station_id#22, end_station_name#23, end_station_id#24, start_lat#67, start_lng#68, end_lat#69, end_lng#70, member_casual#29, is_valid_ride_id#154, is_valid_duration#155, is_valid_stations#156, _source_file#157, year#161, month#162, _w0#163, _w1#164]
Arguments: [row_number() windowspecdefinition(start_station_id#22, _w0#163, _w1#164, started_at#72 ASC NULLS FIRST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS _start_station_ride_num#160], [start_station_id#22, _w0#163, _w1#164], [started_at#72 ASC NULLS FIRST]

(9) Filter [codegen id : 3]
Input [22]: [ride_id#17, rideable_type#18, started_at#72, ended_at#73, start_station_name#21, start_station_id#22, end_station_name#23, end_station_id#24, start_lat#67, start_lng#68, end_lat#69, end_lng#70, member_casual#29, is_valid_ride_id#154, is_valid_duration#155, is_valid_stations#156, _source_file#157, year#161, month#162, _w0#163, _w1#164, _start_station_ride_num#160]
Condition : ((NOT is_valid_ride_id#154 OR NOT is_valid_duration#155) OR NOT is_valid_stations#156)

(10) Project [codegen id : 3]
Output [21]: [ride_id#17, rideable_type#18, started_at#72, ended_at#73, start_station_name#21, start_station_id#22, end_station_name#23, end_station_id#24, start_lat#67, start_lng#68, end_lat#69, end_lng#70, member_casual#29, is_valid_ride_id#154, is_valid_duration#155, is_valid_stations#156, _source_file#157, 2026-11-06 14:16:48.23583 AS _processed_dttm#158, _start_station_ride_num#160, year#161, month#162]
Input [22]: [ride_id#17, rideable_type#18, started_at#72, ended_at#73, start_station_name#21, start_station_id#22, end_station_name#23, end_station_id#24, start_lat#67, start_lng#68, end_lat#69, end_lng#70, member_casual#29, is_valid_ride_id#154, is_valid_duration#155, is_valid_stations#156, _source_file#157, year#161, month#162, _w0#163, _w1#164, _start_station_ride_num#160]

(11) WriteFiles
Input [21]: [ride_id#17, rideable_type#18, started_at#72, ended_at#73, start_station_name#21, start_station_id#22, end_station_name#23, end_station_id#24, start_lat#67, start_lng#68, end_lat#69, end_lng#70, member_casual#29, is_valid_ride_id#154, is_valid_duration#155, is_valid_stations#156, _source_file#157, _processed_dttm#158, _start_station_ride_num#160, year#161, month#162]

(12) Execute InsertIntoHadoopFsRelationCommand
Input: []
Arguments: s3a://rzvde-g12-kryukov-egor/raw_invalid/citibike_data/202405, false, CSV, [header=true, path=s3a://rzvde-g12-kryukov-egor/raw_invalid/citibike_data/202405/], Overwrite, [ride_id, rideable_type, started_at, ended_at, start_station_name, start_station_id, end_station_name, end_station_id, start_lat, start_lng, end_lat, end_lng, member_casual, is_valid_ride_id, is_valid_duration, is_valid_stations, _source_file, _processed_dttm, _start_station_ride_num, year, month]

(13) Project
Output [13]: [ride_id#17, rideable_type#18, cast(started_at#19 as timestamp) AS started_at#72, cast(ended_at#20 as timestamp) AS ended_at#73, start_station_name#21, start_station_id#22, end_station_name#23, end_station_id#24, cast(start_lat#25 as double) AS start_lat#67, cast(start_lng#26 as double) AS start_lng#68, cast(end_lat#27 as double) AS end_lat#69, cast(end_lng#28 as double) AS end_lng#70, member_casual#29]
Input [13]: [ride_id#17, rideable_type#18, started_at#19, ended_at#20, start_station_name#21, start_station_id#22, end_station_name#23, end_station_id#24, start_lat#25, start_lng#26, end_lat#27, end_lng#28, member_casual#29]

(14) Project
Output [21]: [ride_id#17, rideable_type#18, started_at#72, ended_at#73, start_station_name#21, start_station_id#22, end_station_name#23, end_station_id#24, start_lat#67, start_lng#68, end_lat#69, end_lng#70, member_casual#29, (isnotnull(ride_id#17) AND NOT (trim(ride_id#17, None) = )) AS is_valid_ride_id#154, ((isnotnull(started_at#72) AND isnotnull(ended_at#73)) AND (ended_at#73 > started_at#72)) AS is_valid_duration#155, ((isnotnull(start_station_id#22) AND isnotnull(end_station_id#24)) AND NOT (end_station_id#24 = start_station_id#22)) AS is_valid_stations#156, input_file_name() AS _source_file#157, year(cast(started_at#72 as date)) AS year#161, month(cast(started_at#72 as date)) AS month#162, year(cast(started_at#72 as date)) AS _w0#163, month(cast(started_at#72 as date)) AS _w1#164]
Input [13]: [ride_id#17, rideable_type#18, started_at#72, ended_at#73, start_station_name#21, start_station_id#22, end_station_name#23, end_station_id#24, start_lat#67, start_lng#68, end_lat#69, end_lng#70, member_casual#29]

(15) Exchange
Input [21]: [ride_id#17, rideable_type#18, started_at#72, ended_at#73, start_station_name#21, start_station_id#22, end_station_name#23, end_station_id#24, start_lat#67, start_lng#68, end_lat#69, end_lng#70, member_casual#29, is_valid_ride_id#154, is_valid_duration#155, is_valid_stations#156, _source_file#157, year#161, month#162, _w0#163, _w1#164]
Arguments: hashpartitioning(start_station_id#22, _w0#163, _w1#164, 200), ENSURE_REQUIREMENTS, [plan_id=402]

(16) Sort
Input [21]: [ride_id#17, rideable_type#18, started_at#72, ended_at#73, start_station_name#21, start_station_id#22, end_station_name#23, end_station_id#24, start_lat#67, start_lng#68, end_lat#69, end_lng#70, member_casual#29, is_valid_ride_id#154, is_valid_duration#155, is_valid_stations#156, _source_file#157, year#161, month#162, _w0#163, _w1#164]
Arguments: [start_station_id#22 ASC NULLS FIRST, _w0#163 ASC NULLS FIRST, _w1#164 ASC NULLS FIRST, started_at#72 ASC NULLS FIRST], false, 0

(17) Window
Input [21]: [ride_id#17, rideable_type#18, started_at#72, ended_at#73, start_station_name#21, start_station_id#22, end_station_name#23, end_station_id#24, start_lat#67, start_lng#68, end_lat#69, end_lng#70, member_casual#29, is_valid_ride_id#154, is_valid_duration#155, is_valid_stations#156, _source_file#157, year#161, month#162, _w0#163, _w1#164]
Arguments: [row_number() windowspecdefinition(start_station_id#22, _w0#163, _w1#164, started_at#72 ASC NULLS FIRST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS _start_station_ride_num#160], [start_station_id#22, _w0#163, _w1#164], [started_at#72 ASC NULLS FIRST]

(18) Filter
Input [22]: [ride_id#17, rideable_type#18, started_at#72, ended_at#73, start_station_name#21, start_station_id#22, end_station_name#23, end_station_id#24, start_lat#67, start_lng#68, end_lat#69, end_lng#70, member_casual#29, is_valid_ride_id#154, is_valid_duration#155, is_valid_stations#156, _source_file#157, year#161, month#162, _w0#163, _w1#164, _start_station_ride_num#160]
Condition : ((NOT is_valid_ride_id#154 OR NOT is_valid_duration#155) OR NOT is_valid_stations#156)

(19) Project
Output [21]: [ride_id#17, rideable_type#18, started_at#72, ended_at#73, start_station_name#21, start_station_id#22, end_station_name#23, end_station_id#24, start_lat#67, start_lng#68, end_lat#69, end_lng#70, member_casual#29, is_valid_ride_id#154, is_valid_duration#155, is_valid_stations#156, _source_file#157, 2026-11-06 14:16:48.23583 AS _processed_dttm#158, _start_station_ride_num#160, year#161, month#162]
Input [22]: [ride_id#17, rideable_type#18, started_at#72, ended_at#73, start_station_name#21, start_station_id#22, end_station_name#23, end_station_id#24, start_lat#67, start_lng#68, end_lat#69, end_lng#70, member_casual#29, is_valid_ride_id#154, is_valid_duration#155, is_valid_stations#156, _source_file#157, year#161, month#162, _w0#163, _w1#164, _start_station_ride_num#160]

(20) WriteFiles
Input [21]: [ride_id#17, rideable_type#18, started_at#72, ended_at#73, start_station_name#21, start_station_id#22, end_station_name#23, end_station_id#24, start_lat#67, start_lng#68, end_lat#69, end_lng#70, member_casual#29, is_valid_ride_id#154, is_valid_duration#155, is_valid_stations#156, _source_file#157, _processed_dttm#158, _start_station_ride_num#160, year#161, month#162]

(21) Execute InsertIntoHadoopFsRelationCommand
Input: []
Arguments: s3a://rzvde-g12-kryukov-egor/raw_invalid/citibike_data/202405, false, CSV, [header=true, path=s3a://rzvde-g12-kryukov-egor/raw_invalid/citibike_data/202405/], Overwrite, [ride_id, rideable_type, started_at, ended_at, start_station_name, start_station_id, end_station_name, end_station_id, start_lat, start_lng, end_lat, end_lng, member_casual, is_valid_ride_id, is_valid_duration, is_valid_stations, _source_file, _processed_dttm, _start_station_ride_num, year, month]

(22) AdaptiveSparkPlan
Output: []
Arguments: isFinalPlan=true