g9_chernyshev_miron_s2 - Details for Query 6

Submitted Time: 2026/08/18 12:36:13
Duration: 14 s
Succeeded Jobs: 10 11

Show the Stage ID and Task ID that corresponds to the max metric

digraph G { 0 [labelType="html" label=" AdaptiveSparkPlan "]; subgraph cluster1 { isCluster="true"; label="WholeStageCodegen (2)\n \nduration: 1 ms"; 2 [labelType="html" label="HashAggregate time in aggregation build: 1 ms number of output rows: 1"]; } 3 [labelType="html" label="Exchange shuffle records written: 8 local merged chunks fetched: 0 shuffle write time total (min, med, max (stageId: taskId)) 5 ms (0 ms, 0 ms, 0 ms (stage 13.0: task 44)) remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 8 local bytes read: 456.0 B fetch wait time: 0 ms remote bytes read: 0.0 B merged fetch fallback count: 0 local blocks read: 8 remote merged chunks fetched: 0 remote blocks read: 0 data size total (min, med, max (stageId: taskId)) 448.0 B (0.0 B, 56.0 B, 56.0 B (stage 13.0: task 39)) local merged bytes read: 0.0 B number of partitions: 1 remote reqs duration: 0 ms remote bytes read to disk: 0.0 B shuffle bytes written total (min, med, max (stageId: taskId)) 456.0 B (0.0 B, 57.0 B, 57.0 B (stage 13.0: task 39))"]; subgraph cluster4 { isCluster="true"; label="WholeStageCodegen (1)\n \nduration: total (min, med, max (stageId: taskId))\n25.6 s (1.6 s, 3.4 s, 4.0 s (stage 13.0: task 40))"; 5 [labelType="html" label="HashAggregate time in aggregation build total (min, med, max (stageId: taskId)) 25.6 s (1.6 s, 3.4 s, 4.0 s (stage 13.0: task 40)) number of output rows: 8"]; 6 [labelType="html" label=" Project "]; } 7 [labelType="html" label="Scan csv number of output rows: 4,988,053 number of files read: 5 metadata time: 1 ms size of files read: 928.8 MiB"]; 2->0; 3->2; 5->3; 6->5; 7->6; }

AdaptiveSparkPlan isFinalPlan=true

HashAggregate(keys=[], functions=[sum(cast(isnan(start_station_id#115) as int)), sum(cast(isnan(end_station_id#117) as int)), sum(cast(isnan(start_lat#25) as int)), sum(cast(isnan(start_lng#26) as int)), sum(cast(isnan(end_lat#27) as int)), sum(cast(isnan(end_lng#28) as int))])

WholeStageCodegen (2)

Exchange SinglePartition, ENSURE_REQUIREMENTS, [plan_id=177]

HashAggregate(keys=[], functions=[partial_sum(cast(isnan(start_station_id#115) as int)), partial_sum(cast(isnan(end_station_id#117) as int)), partial_sum(cast(isnan(start_lat#25) as int)), partial_sum(cast(isnan(start_lng#26) as int)), partial_sum(cast(isnan(end_lat#27) as int)), partial_sum(cast(isnan(end_lng#28) as int))])

Project [cast(start_station_id#22 as float) AS start_station_id#115, cast(end_station_id#24 as float) AS end_station_id#117, start_lat#25, start_lng#26, end_lat#27, end_lng#28]

WholeStageCodegen (1)

FileScan csv [start_station_id#22,end_station_id#24,start_lat#25,start_lng#26,end_lat#27,end_lng#28] Batched: false, DataFilters: [], Format: CSV, Location: InMemoryFileIndex(5 paths)[s3a://rzvde-g9-chernyshev-miron/raw/citibike_data/202507/202507-citibi..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<start_station_id:string,end_station_id:string,start_lat:double,start_lng:double,end_lat:do...

Details

== Physical Plan ==
AdaptiveSparkPlan (11)
+- == Final Plan ==
   * HashAggregate (6)
   +- ShuffleQueryStage (5), Statistics(sizeInBytes=448.0 B, rowCount=8)
      +- Exchange (4)
         +- * HashAggregate (3)
            +- * Project (2)
               +- Scan csv  (1)
+- == Initial Plan ==
   HashAggregate (10)
   +- Exchange (9)
      +- HashAggregate (8)
         +- Project (7)
            +- Scan csv  (1)


(1) Scan csv 
Output [6]: [start_station_id#22, end_station_id#24, start_lat#25, start_lng#26, end_lat#27, end_lng#28]
Batched: false
Location: InMemoryFileIndex [s3a://rzvde-g9-chernyshev-miron/raw/citibike_data/202507/202507-citibike-tripdata-part00.csv, ... 4 entries]
ReadSchema: struct<start_station_id:string,end_station_id:string,start_lat:double,start_lng:double,end_lat:double,end_lng:double>

(2) Project [codegen id : 1]
Output [6]: [cast(start_station_id#22 as float) AS start_station_id#115, cast(end_station_id#24 as float) AS end_station_id#117, start_lat#25, start_lng#26, end_lat#27, end_lng#28]
Input [6]: [start_station_id#22, end_station_id#24, start_lat#25, start_lng#26, end_lat#27, end_lng#28]

(3) HashAggregate [codegen id : 1]
Input [6]: [start_station_id#115, end_station_id#117, start_lat#25, start_lng#26, end_lat#27, end_lng#28]
Keys: []
Functions [6]: [partial_sum(cast(isnan(start_station_id#115) as int)), partial_sum(cast(isnan(end_station_id#117) as int)), partial_sum(cast(isnan(start_lat#25) as int)), partial_sum(cast(isnan(start_lng#26) as int)), partial_sum(cast(isnan(end_lat#27) as int)), partial_sum(cast(isnan(end_lng#28) as int))]
Aggregate Attributes [6]: [sum#451L, sum#452L, sum#453L, sum#454L, sum#455L, sum#456L]
Results [6]: [sum#457L, sum#458L, sum#459L, sum#460L, sum#461L, sum#462L]

(4) Exchange
Input [6]: [sum#457L, sum#458L, sum#459L, sum#460L, sum#461L, sum#462L]
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=177]

(5) ShuffleQueryStage
Output [6]: [sum#457L, sum#458L, sum#459L, sum#460L, sum#461L, sum#462L]
Arguments: 0

(6) HashAggregate [codegen id : 2]
Input [6]: [sum#457L, sum#458L, sum#459L, sum#460L, sum#461L, sum#462L]
Keys: []
Functions [6]: [sum(cast(isnan(start_station_id#115) as int)), sum(cast(isnan(end_station_id#117) as int)), sum(cast(isnan(start_lat#25) as int)), sum(cast(isnan(start_lng#26) as int)), sum(cast(isnan(end_lat#27) as int)), sum(cast(isnan(end_lng#28) as int))]
Aggregate Attributes [6]: [sum(cast(isnan(start_station_id#115) as int))#410L, sum(cast(isnan(end_station_id#117) as int))#412L, sum(cast(isnan(start_lat#25) as int))#414L, sum(cast(isnan(start_lng#26) as int))#416L, sum(cast(isnan(end_lat#27) as int))#418L, sum(cast(isnan(end_lng#28) as int))#420L]
Results [6]: [toprettystring(sum(cast(isnan(start_station_id#115) as int))#410L, Some(Etc/UTC)) AS toprettystring(sum(CAST(isnan(start_station_id) AS INT) AS start_station_id))#439, toprettystring(sum(cast(isnan(end_station_id#117) as int))#412L, Some(Etc/UTC)) AS toprettystring(sum(CAST(isnan(end_station_id) AS INT) AS end_station_id))#440, toprettystring(sum(cast(isnan(start_lat#25) as int))#414L, Some(Etc/UTC)) AS toprettystring(sum(CAST(isnan(start_lat) AS INT) AS start_lat))#441, toprettystring(sum(cast(isnan(start_lng#26) as int))#416L, Some(Etc/UTC)) AS toprettystring(sum(CAST(isnan(start_lng) AS INT) AS start_lng))#442, toprettystring(sum(cast(isnan(end_lat#27) as int))#418L, Some(Etc/UTC)) AS toprettystring(sum(CAST(isnan(end_lat) AS INT) AS end_lat))#443, toprettystring(sum(cast(isnan(end_lng#28) as int))#420L, Some(Etc/UTC)) AS toprettystring(sum(CAST(isnan(end_lng) AS INT) AS end_lng))#444]

(7) Project
Output [6]: [cast(start_station_id#22 as float) AS start_station_id#115, cast(end_station_id#24 as float) AS end_station_id#117, start_lat#25, start_lng#26, end_lat#27, end_lng#28]
Input [6]: [start_station_id#22, end_station_id#24, start_lat#25, start_lng#26, end_lat#27, end_lng#28]

(8) HashAggregate
Input [6]: [start_station_id#115, end_station_id#117, start_lat#25, start_lng#26, end_lat#27, end_lng#28]
Keys: []
Functions [6]: [partial_sum(cast(isnan(start_station_id#115) as int)), partial_sum(cast(isnan(end_station_id#117) as int)), partial_sum(cast(isnan(start_lat#25) as int)), partial_sum(cast(isnan(start_lng#26) as int)), partial_sum(cast(isnan(end_lat#27) as int)), partial_sum(cast(isnan(end_lng#28) as int))]
Aggregate Attributes [6]: [sum#451L, sum#452L, sum#453L, sum#454L, sum#455L, sum#456L]
Results [6]: [sum#457L, sum#458L, sum#459L, sum#460L, sum#461L, sum#462L]

(9) Exchange
Input [6]: [sum#457L, sum#458L, sum#459L, sum#460L, sum#461L, sum#462L]
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=165]

(10) HashAggregate
Input [6]: [sum#457L, sum#458L, sum#459L, sum#460L, sum#461L, sum#462L]
Keys: []
Functions [6]: [sum(cast(isnan(start_station_id#115) as int)), sum(cast(isnan(end_station_id#117) as int)), sum(cast(isnan(start_lat#25) as int)), sum(cast(isnan(start_lng#26) as int)), sum(cast(isnan(end_lat#27) as int)), sum(cast(isnan(end_lng#28) as int))]
Aggregate Attributes [6]: [sum(cast(isnan(start_station_id#115) as int))#410L, sum(cast(isnan(end_station_id#117) as int))#412L, sum(cast(isnan(start_lat#25) as int))#414L, sum(cast(isnan(start_lng#26) as int))#416L, sum(cast(isnan(end_lat#27) as int))#418L, sum(cast(isnan(end_lng#28) as int))#420L]
Results [6]: [toprettystring(sum(cast(isnan(start_station_id#115) as int))#410L, Some(Etc/UTC)) AS toprettystring(sum(CAST(isnan(start_station_id) AS INT) AS start_station_id))#439, toprettystring(sum(cast(isnan(end_station_id#117) as int))#412L, Some(Etc/UTC)) AS toprettystring(sum(CAST(isnan(end_station_id) AS INT) AS end_station_id))#440, toprettystring(sum(cast(isnan(start_lat#25) as int))#414L, Some(Etc/UTC)) AS toprettystring(sum(CAST(isnan(start_lat) AS INT) AS start_lat))#441, toprettystring(sum(cast(isnan(start_lng#26) as int))#416L, Some(Etc/UTC)) AS toprettystring(sum(CAST(isnan(start_lng) AS INT) AS start_lng))#442, toprettystring(sum(cast(isnan(end_lat#27) as int))#418L, Some(Etc/UTC)) AS toprettystring(sum(CAST(isnan(end_lat) AS INT) AS end_lat))#443, toprettystring(sum(cast(isnan(end_lng#28) as int))#420L, Some(Etc/UTC)) AS toprettystring(sum(CAST(isnan(end_lng) AS INT) AS end_lng))#444]

(11) AdaptiveSparkPlan
Output [6]: [toprettystring(sum(CAST(isnan(start_station_id) AS INT) AS start_station_id))#439, toprettystring(sum(CAST(isnan(end_station_id) AS INT) AS end_station_id))#440, toprettystring(sum(CAST(isnan(start_lat) AS INT) AS start_lat))#441, toprettystring(sum(CAST(isnan(start_lng) AS INT) AS start_lng))#442, toprettystring(sum(CAST(isnan(end_lat) AS INT) AS end_lat))#443, toprettystring(sum(CAST(isnan(end_lng) AS INT) AS end_lng))#444]
Arguments: isFinalPlan=true