CitiBike-External-S3 - Details for Query 11

Submitted Time: 2026/11/04 19:02:25
Duration: 48 s
Succeeded Jobs: 18 19

Show the Stage ID and Task ID that corresponds to the max metric

digraph G { 0 [labelType="html" label=" AdaptiveSparkPlan "]; 1 [labelType="html" label="Execute InsertIntoHadoopFsRelationCommand task commit time total (min, med, max (stageId: taskId)) 22.8 s (1.3 s, 1.6 s, 1.7 s (stage 27.0: task 121)) number of written files: 15 job commit time: 22.4 s number of output rows: 99,666 number of dynamic part: 0 written output: 33.0 MiB"]; 2 [labelType="html" label=" WriteFiles "]; subgraph cluster3 { isCluster="true"; label="WholeStageCodegen (3)\n \nduration: total (min, med, max (stageId: taskId))\n41.2 s (566 ms, 2.9 s, 4.2 s (stage 27.0: task 117))"; 4 [labelType="html" label=" Project "]; 5 [labelType="html" label="Filter number of output rows: 99,666"]; } 6 [labelType="html" label="Window spill size total (min, med, max (stageId: taskId)) 0.0 B (0.0 B, 0.0 B, 0.0 B (stage 27.0: task 116))"]; subgraph cluster7 { isCluster="true"; label="WholeStageCodegen (2)\n \nduration: total (min, med, max (stageId: taskId))\n1.3 m (1.4 s, 5.5 s, 6.5 s (stage 27.0: task 117))"; 8 [labelType="html" label="Sort sort time total (min, med, max (stageId: taskId)) 18.6 s (637 ms, 1.3 s, 1.6 s (stage 27.0: task 129)) peak memory total (min, med, max (stageId: taskId)) 1684.0 MiB (84.0 MiB, 108.0 MiB, 128.0 MiB (stage 27.0: task 128)) spill size total (min, med, max (stageId: taskId)) 700.0 MiB (0.0 B, 0.0 B, 100.0 MiB (stage 27.0: task 118))"]; } 9 [labelType="html" label="AQEShuffleRead number of partitions: 15 partition data size total (min, med, max (stageId: taskId)) 340.8 MiB (15.7 MiB, 23.4 MiB, 24.3 MiB (driver)) number of coalesced partitions: 15"]; 10 [labelType="html" label="Exchange shuffle records written: 4,731,900 local merged chunks fetched: 0 shuffle write time total (min, med, max (stageId: taskId)) 3.0 s (0 ms, 36 ms, 268 ms (stage 25.0: task 112)) remote merged bytes read total (min, med, max (stageId: taskId)) 0.0 B (0.0 B, 0.0 B, 0.0 B (stage 27.0: task 118)) local merged blocks fetched: 0 corrupt merged block chunks: 0 remote merged reqs duration total (min, med, max (stageId: taskId)) 0 ms (0 ms, 0 ms, 0 ms (stage 27.0: task 118)) remote merged blocks fetched: 0 records read: 4,731,900 local bytes read total (min, med, max (stageId: taskId)) 85.8 MiB (3.3 MiB, 6.3 MiB, 6.8 MiB (stage 27.0: task 127)) fetch wait time total (min, med, max (stageId: taskId)) 0 ms (0 ms, 0 ms, 0 ms (stage 27.0: task 116)) remote bytes read total (min, med, max (stageId: taskId)) 238.8 MiB (10.8 MiB, 16.2 MiB, 18.7 MiB (stage 27.0: task 126)) merged fetch fallback count: 0 local blocks read: 58 remote merged chunks fetched: 0 remote blocks read: 167 data size total (min, med, max (stageId: taskId)) 1612.1 MiB (0.0 B, 13.7 MiB, 118.3 MiB (stage 25.0: task 106)) local merged bytes read total (min, med, max (stageId: taskId)) 0.0 B (0.0 B, 0.0 B, 0.0 B (stage 27.0: task 118)) number of partitions: 200 remote reqs duration total (min, med, max (stageId: taskId)) 5.1 s (168 ms, 301 ms, 624 ms (stage 27.0: task 128)) remote bytes read to disk total (min, med, max (stageId: taskId)) 0.0 B (0.0 B, 0.0 B, 0.0 B (stage 27.0: task 118)) shuffle bytes written total (min, med, max (stageId: taskId)) 324.7 MiB (0.0 B, 2.7 MiB, 24.5 MiB (stage 25.0: task 110))"]; subgraph cluster11 { isCluster="true"; label="WholeStageCodegen (1)\n \nduration: total (min, med, max (stageId: taskId))\n1.5 m (0 ms, 1.1 s, 8.4 s (stage 25.0: task 101))"; 12 [labelType="html" label=" Project "]; 13 [labelType="html" label=" Project "]; } 14 [labelType="html" label="Scan csv number of output rows: 4,731,900 number of files read: 5 metadata time total (min, med, max (stageId: taskId)) 0 ms (0 ms, 0 ms, 0 ms (stage 27.0: task 118)) size of files read total (min, med, max (stageId: taskId)) 881.3 MiB (0.0 B, 0.0 B, 881.3 MiB (driver))"]; 1->0; 2->1; 4->2; 5->4; 6->5; 8->6; 9->8; 10->9; 12->10; 13->12; 14->13; }

AdaptiveSparkPlan isFinalPlan=true

Execute InsertIntoHadoopFsRelationCommand s3a://rzvde-g11-terentyeva-kristina/raw_invalid/citibike_data/202510, false, CSV, [header=true, path=s3a://rzvde-g11-terentyeva-kristina/raw_invalid/citibike_data/202510/], Overwrite, [ride_id, rideable_type, started_at, ended_at, start_station_name, start_station_id, end_station_name, end_station_id, start_lat, start_lng, end_lat, end_lng, member_casual, flag_ride_id_valid, flag_time_valid, flag_station_valid, _source_file, _processed_dttm, year, month, _start_station_ride_num, is_valid_row]

WriteFiles

Project [ride_id#171, rideable_type#172, started_at#208, ended_at#209, start_station_name#175, start_station_id#176, end_station_name#177, end_station_id#178, start_lat#203, start_lng#204, end_lat#205, end_lng#206, member_casual#183, flag_ride_id_valid#308, flag_time_valid#309, flag_station_valid#310, _source_file#311, 2026-11-04 19:02:24.909156 AS _processed_dttm#312, year#313, month#314, _start_station_ride_num#336, ((flag_ride_id_valid#308 AND flag_time_valid#309) AND flag_station_valid#310) AS is_valid_row#482]

Filter ((NOT flag_ride_id_valid#308 OR NOT flag_time_valid#309) OR NOT flag_station_valid#310)

WholeStageCodegen (3)

Window [row_number() windowspecdefinition(start_station_id#176, started_at#208 ASC NULLS FIRST, ride_id#171 ASC NULLS FIRST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS _start_station_ride_num#336], [start_station_id#176], [started_at#208 ASC NULLS FIRST, ride_id#171 ASC NULLS FIRST]

Sort [start_station_id#176 ASC NULLS FIRST, started_at#208 ASC NULLS FIRST, ride_id#171 ASC NULLS FIRST], false, 0

WholeStageCodegen (2)

AQEShuffleRead coalesced

Exchange hashpartitioning(start_station_id#176, 200), ENSURE_REQUIREMENTS, [plan_id=395]

Project [ride_id#171, rideable_type#172, started_at#208, ended_at#209, start_station_name#175, start_station_id#176, end_station_name#177, end_station_id#178, start_lat#203, start_lng#204, end_lat#205, end_lng#206, member_casual#183, (isnotnull(ride_id#171) AND NOT (trim(ride_id#171, None) = )) AS flag_ride_id_valid#308, ((isnotnull(ended_at#209) AND isnotnull(started_at#208)) AND (ended_at#209 > started_at#208)) AS flag_time_valid#309, ((isnotnull(end_station_id#178) AND isnotnull(start_station_id#176)) AND NOT (end_station_id#178 = start_station_id#176)) AS flag_station_valid#310, input_file_name() AS _source_file#311, year(cast(started_at#208 as date)) AS year#313, month(cast(started_at#208 as date)) AS month#314]

Project [ride_id#171, rideable_type#172, cast(started_at#173 as timestamp) AS started_at#208, cast(ended_at#174 as timestamp) AS ended_at#209, start_station_name#175, start_station_id#176, end_station_name#177, end_station_id#178, cast(start_lat#179 as double) AS start_lat#203, cast(start_lng#180 as double) AS start_lng#204, cast(end_lat#181 as double) AS end_lat#205, cast(end_lng#182 as double) AS end_lng#206, member_casual#183]

WholeStageCodegen (1)

FileScan csv [ride_id#171,rideable_type#172,started_at#173,ended_at#174,start_station_name#175,start_station_id#176,end_station_name#177,end_station_id#178,start_lat#179,start_lng#180,end_lat#181,end_lng#182,member_casual#183] Batched: false, DataFilters: [], Format: CSV, Location: InMemoryFileIndex(5 paths)[s3a://rzvde-g11-terentyeva-kristina/raw/citibike_data/202510/202510-ci..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<ride_id:string,rideable_type:string,started_at:string,ended_at:string,start_station_name:s...

Details

== Physical Plan ==
AdaptiveSparkPlan (22)
+- == Final Plan ==
   Execute InsertIntoHadoopFsRelationCommand (12)
   +- WriteFiles (11)
      +- * Project (10)
         +- * Filter (9)
            +- Window (8)
               +- * Sort (7)
                  +- AQEShuffleRead (6)
                     +- ShuffleQueryStage (5), Statistics(sizeInBytes=1612.1 MiB, rowCount=4.73E+6)
                        +- Exchange (4)
                           +- * Project (3)
                              +- * Project (2)
                                 +- Scan csv  (1)
+- == Initial Plan ==
   Execute InsertIntoHadoopFsRelationCommand (21)
   +- WriteFiles (20)
      +- Project (19)
         +- Filter (18)
            +- Window (17)
               +- Sort (16)
                  +- Exchange (15)
                     +- Project (14)
                        +- Project (13)
                           +- Scan csv  (1)


(1) Scan csv 
Output [13]: [ride_id#171, rideable_type#172, started_at#173, ended_at#174, start_station_name#175, start_station_id#176, end_station_name#177, end_station_id#178, start_lat#179, start_lng#180, end_lat#181, end_lng#182, member_casual#183]
Batched: false
Location: InMemoryFileIndex [s3a://rzvde-g11-terentyeva-kristina/raw/citibike_data/202510/202510-citibike-tripdata-part00.csv, ... 4 entries]
ReadSchema: struct<ride_id:string,rideable_type:string,started_at:string,ended_at:string,start_station_name:string,start_station_id:string,end_station_name:string,end_station_id:string,start_lat:string,start_lng:string,end_lat:string,end_lng:string,member_casual:string>

(2) Project [codegen id : 1]
Output [13]: [ride_id#171, rideable_type#172, cast(started_at#173 as timestamp) AS started_at#208, cast(ended_at#174 as timestamp) AS ended_at#209, start_station_name#175, start_station_id#176, end_station_name#177, end_station_id#178, cast(start_lat#179 as double) AS start_lat#203, cast(start_lng#180 as double) AS start_lng#204, cast(end_lat#181 as double) AS end_lat#205, cast(end_lng#182 as double) AS end_lng#206, member_casual#183]
Input [13]: [ride_id#171, rideable_type#172, started_at#173, ended_at#174, start_station_name#175, start_station_id#176, end_station_name#177, end_station_id#178, start_lat#179, start_lng#180, end_lat#181, end_lng#182, member_casual#183]

(3) Project [codegen id : 1]
Output [19]: [ride_id#171, rideable_type#172, started_at#208, ended_at#209, start_station_name#175, start_station_id#176, end_station_name#177, end_station_id#178, start_lat#203, start_lng#204, end_lat#205, end_lng#206, member_casual#183, (isnotnull(ride_id#171) AND NOT (trim(ride_id#171, None) = )) AS flag_ride_id_valid#308, ((isnotnull(ended_at#209) AND isnotnull(started_at#208)) AND (ended_at#209 > started_at#208)) AS flag_time_valid#309, ((isnotnull(end_station_id#178) AND isnotnull(start_station_id#176)) AND NOT (end_station_id#178 = start_station_id#176)) AS flag_station_valid#310, input_file_name() AS _source_file#311, year(cast(started_at#208 as date)) AS year#313, month(cast(started_at#208 as date)) AS month#314]
Input [13]: [ride_id#171, rideable_type#172, started_at#208, ended_at#209, start_station_name#175, start_station_id#176, end_station_name#177, end_station_id#178, start_lat#203, start_lng#204, end_lat#205, end_lng#206, member_casual#183]

(4) Exchange
Input [19]: [ride_id#171, rideable_type#172, started_at#208, ended_at#209, start_station_name#175, start_station_id#176, end_station_name#177, end_station_id#178, start_lat#203, start_lng#204, end_lat#205, end_lng#206, member_casual#183, flag_ride_id_valid#308, flag_time_valid#309, flag_station_valid#310, _source_file#311, year#313, month#314]
Arguments: hashpartitioning(start_station_id#176, 200), ENSURE_REQUIREMENTS, [plan_id=395]

(5) ShuffleQueryStage
Output [19]: [ride_id#171, rideable_type#172, started_at#208, ended_at#209, start_station_name#175, start_station_id#176, end_station_name#177, end_station_id#178, start_lat#203, start_lng#204, end_lat#205, end_lng#206, member_casual#183, flag_ride_id_valid#308, flag_time_valid#309, flag_station_valid#310, _source_file#311, year#313, month#314]
Arguments: 0

(6) AQEShuffleRead
Input [19]: [ride_id#171, rideable_type#172, started_at#208, ended_at#209, start_station_name#175, start_station_id#176, end_station_name#177, end_station_id#178, start_lat#203, start_lng#204, end_lat#205, end_lng#206, member_casual#183, flag_ride_id_valid#308, flag_time_valid#309, flag_station_valid#310, _source_file#311, year#313, month#314]
Arguments: coalesced

(7) Sort [codegen id : 2]
Input [19]: [ride_id#171, rideable_type#172, started_at#208, ended_at#209, start_station_name#175, start_station_id#176, end_station_name#177, end_station_id#178, start_lat#203, start_lng#204, end_lat#205, end_lng#206, member_casual#183, flag_ride_id_valid#308, flag_time_valid#309, flag_station_valid#310, _source_file#311, year#313, month#314]
Arguments: [start_station_id#176 ASC NULLS FIRST, started_at#208 ASC NULLS FIRST, ride_id#171 ASC NULLS FIRST], false, 0

(8) Window
Input [19]: [ride_id#171, rideable_type#172, started_at#208, ended_at#209, start_station_name#175, start_station_id#176, end_station_name#177, end_station_id#178, start_lat#203, start_lng#204, end_lat#205, end_lng#206, member_casual#183, flag_ride_id_valid#308, flag_time_valid#309, flag_station_valid#310, _source_file#311, year#313, month#314]
Arguments: [row_number() windowspecdefinition(start_station_id#176, started_at#208 ASC NULLS FIRST, ride_id#171 ASC NULLS FIRST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS _start_station_ride_num#336], [start_station_id#176], [started_at#208 ASC NULLS FIRST, ride_id#171 ASC NULLS FIRST]

(9) Filter [codegen id : 3]
Input [20]: [ride_id#171, rideable_type#172, started_at#208, ended_at#209, start_station_name#175, start_station_id#176, end_station_name#177, end_station_id#178, start_lat#203, start_lng#204, end_lat#205, end_lng#206, member_casual#183, flag_ride_id_valid#308, flag_time_valid#309, flag_station_valid#310, _source_file#311, year#313, month#314, _start_station_ride_num#336]
Condition : ((NOT flag_ride_id_valid#308 OR NOT flag_time_valid#309) OR NOT flag_station_valid#310)

(10) Project [codegen id : 3]
Output [22]: [ride_id#171, rideable_type#172, started_at#208, ended_at#209, start_station_name#175, start_station_id#176, end_station_name#177, end_station_id#178, start_lat#203, start_lng#204, end_lat#205, end_lng#206, member_casual#183, flag_ride_id_valid#308, flag_time_valid#309, flag_station_valid#310, _source_file#311, 2026-11-04 19:02:24.909156 AS _processed_dttm#312, year#313, month#314, _start_station_ride_num#336, ((flag_ride_id_valid#308 AND flag_time_valid#309) AND flag_station_valid#310) AS is_valid_row#482]
Input [20]: [ride_id#171, rideable_type#172, started_at#208, ended_at#209, start_station_name#175, start_station_id#176, end_station_name#177, end_station_id#178, start_lat#203, start_lng#204, end_lat#205, end_lng#206, member_casual#183, flag_ride_id_valid#308, flag_time_valid#309, flag_station_valid#310, _source_file#311, year#313, month#314, _start_station_ride_num#336]

(11) WriteFiles
Input [22]: [ride_id#171, rideable_type#172, started_at#208, ended_at#209, start_station_name#175, start_station_id#176, end_station_name#177, end_station_id#178, start_lat#203, start_lng#204, end_lat#205, end_lng#206, member_casual#183, flag_ride_id_valid#308, flag_time_valid#309, flag_station_valid#310, _source_file#311, _processed_dttm#312, year#313, month#314, _start_station_ride_num#336, is_valid_row#482]

(12) Execute InsertIntoHadoopFsRelationCommand
Input: []
Arguments: s3a://rzvde-g11-terentyeva-kristina/raw_invalid/citibike_data/202510, false, CSV, [header=true, path=s3a://rzvde-g11-terentyeva-kristina/raw_invalid/citibike_data/202510/], Overwrite, [ride_id, rideable_type, started_at, ended_at, start_station_name, start_station_id, end_station_name, end_station_id, start_lat, start_lng, end_lat, end_lng, member_casual, flag_ride_id_valid, flag_time_valid, flag_station_valid, _source_file, _processed_dttm, year, month, _start_station_ride_num, is_valid_row]

(13) Project
Output [13]: [ride_id#171, rideable_type#172, cast(started_at#173 as timestamp) AS started_at#208, cast(ended_at#174 as timestamp) AS ended_at#209, start_station_name#175, start_station_id#176, end_station_name#177, end_station_id#178, cast(start_lat#179 as double) AS start_lat#203, cast(start_lng#180 as double) AS start_lng#204, cast(end_lat#181 as double) AS end_lat#205, cast(end_lng#182 as double) AS end_lng#206, member_casual#183]
Input [13]: [ride_id#171, rideable_type#172, started_at#173, ended_at#174, start_station_name#175, start_station_id#176, end_station_name#177, end_station_id#178, start_lat#179, start_lng#180, end_lat#181, end_lng#182, member_casual#183]

(14) Project
Output [19]: [ride_id#171, rideable_type#172, started_at#208, ended_at#209, start_station_name#175, start_station_id#176, end_station_name#177, end_station_id#178, start_lat#203, start_lng#204, end_lat#205, end_lng#206, member_casual#183, (isnotnull(ride_id#171) AND NOT (trim(ride_id#171, None) = )) AS flag_ride_id_valid#308, ((isnotnull(ended_at#209) AND isnotnull(started_at#208)) AND (ended_at#209 > started_at#208)) AS flag_time_valid#309, ((isnotnull(end_station_id#178) AND isnotnull(start_station_id#176)) AND NOT (end_station_id#178 = start_station_id#176)) AS flag_station_valid#310, input_file_name() AS _source_file#311, year(cast(started_at#208 as date)) AS year#313, month(cast(started_at#208 as date)) AS month#314]
Input [13]: [ride_id#171, rideable_type#172, started_at#208, ended_at#209, start_station_name#175, start_station_id#176, end_station_name#177, end_station_id#178, start_lat#203, start_lng#204, end_lat#205, end_lng#206, member_casual#183]

(15) Exchange
Input [19]: [ride_id#171, rideable_type#172, started_at#208, ended_at#209, start_station_name#175, start_station_id#176, end_station_name#177, end_station_id#178, start_lat#203, start_lng#204, end_lat#205, end_lng#206, member_casual#183, flag_ride_id_valid#308, flag_time_valid#309, flag_station_valid#310, _source_file#311, year#313, month#314]
Arguments: hashpartitioning(start_station_id#176, 200), ENSURE_REQUIREMENTS, [plan_id=377]

(16) Sort
Input [19]: [ride_id#171, rideable_type#172, started_at#208, ended_at#209, start_station_name#175, start_station_id#176, end_station_name#177, end_station_id#178, start_lat#203, start_lng#204, end_lat#205, end_lng#206, member_casual#183, flag_ride_id_valid#308, flag_time_valid#309, flag_station_valid#310, _source_file#311, year#313, month#314]
Arguments: [start_station_id#176 ASC NULLS FIRST, started_at#208 ASC NULLS FIRST, ride_id#171 ASC NULLS FIRST], false, 0

(17) Window
Input [19]: [ride_id#171, rideable_type#172, started_at#208, ended_at#209, start_station_name#175, start_station_id#176, end_station_name#177, end_station_id#178, start_lat#203, start_lng#204, end_lat#205, end_lng#206, member_casual#183, flag_ride_id_valid#308, flag_time_valid#309, flag_station_valid#310, _source_file#311, year#313, month#314]
Arguments: [row_number() windowspecdefinition(start_station_id#176, started_at#208 ASC NULLS FIRST, ride_id#171 ASC NULLS FIRST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS _start_station_ride_num#336], [start_station_id#176], [started_at#208 ASC NULLS FIRST, ride_id#171 ASC NULLS FIRST]

(18) Filter
Input [20]: [ride_id#171, rideable_type#172, started_at#208, ended_at#209, start_station_name#175, start_station_id#176, end_station_name#177, end_station_id#178, start_lat#203, start_lng#204, end_lat#205, end_lng#206, member_casual#183, flag_ride_id_valid#308, flag_time_valid#309, flag_station_valid#310, _source_file#311, year#313, month#314, _start_station_ride_num#336]
Condition : ((NOT flag_ride_id_valid#308 OR NOT flag_time_valid#309) OR NOT flag_station_valid#310)

(19) Project
Output [22]: [ride_id#171, rideable_type#172, started_at#208, ended_at#209, start_station_name#175, start_station_id#176, end_station_name#177, end_station_id#178, start_lat#203, start_lng#204, end_lat#205, end_lng#206, member_casual#183, flag_ride_id_valid#308, flag_time_valid#309, flag_station_valid#310, _source_file#311, 2026-11-04 19:02:24.909156 AS _processed_dttm#312, year#313, month#314, _start_station_ride_num#336, ((flag_ride_id_valid#308 AND flag_time_valid#309) AND flag_station_valid#310) AS is_valid_row#482]
Input [20]: [ride_id#171, rideable_type#172, started_at#208, ended_at#209, start_station_name#175, start_station_id#176, end_station_name#177, end_station_id#178, start_lat#203, start_lng#204, end_lat#205, end_lng#206, member_casual#183, flag_ride_id_valid#308, flag_time_valid#309, flag_station_valid#310, _source_file#311, year#313, month#314, _start_station_ride_num#336]

(20) WriteFiles
Input [22]: [ride_id#171, rideable_type#172, started_at#208, ended_at#209, start_station_name#175, start_station_id#176, end_station_name#177, end_station_id#178, start_lat#203, start_lng#204, end_lat#205, end_lng#206, member_casual#183, flag_ride_id_valid#308, flag_time_valid#309, flag_station_valid#310, _source_file#311, _processed_dttm#312, year#313, month#314, _start_station_ride_num#336, is_valid_row#482]

(21) Execute InsertIntoHadoopFsRelationCommand
Input: []
Arguments: s3a://rzvde-g11-terentyeva-kristina/raw_invalid/citibike_data/202510, false, CSV, [header=true, path=s3a://rzvde-g11-terentyeva-kristina/raw_invalid/citibike_data/202510/], Overwrite, [ride_id, rideable_type, started_at, ended_at, start_station_name, start_station_id, end_station_name, end_station_id, start_lat, start_lng, end_lat, end_lng, member_casual, flag_ride_id_valid, flag_time_valid, flag_station_valid, _source_file, _processed_dttm, year, month, _start_station_ride_num, is_valid_row]

(22) AdaptiveSparkPlan
Output: []
Arguments: isFinalPlan=true