g9_chernyshev_miron_s2 - Details for Query 5

Submitted Time: 2026/08/18 12:35:52
Duration: 18 s
Succeeded Jobs: 8 9

Show the Stage ID and Task ID that corresponds to the max metric

digraph G { 0 [labelType="html" label=" AdaptiveSparkPlan "]; subgraph cluster1 { isCluster="true"; label="WholeStageCodegen (2)\n \nduration: 2 ms"; 2 [labelType="html" label="HashAggregate time in aggregation build: 1 ms number of output rows: 1"]; } 3 [labelType="html" label="Exchange shuffle records written: 8 local merged chunks fetched: 0 shuffle write time total (min, med, max (stageId: taskId)) 7 ms (0 ms, 0 ms, 1 ms (stage 10.0: task 34)) remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 8 local bytes read: 703.0 B fetch wait time: 0 ms remote bytes read: 0.0 B merged fetch fallback count: 0 local blocks read: 8 remote merged chunks fetched: 0 remote blocks read: 0 data size total (min, med, max (stageId: taskId)) 896.0 B (0.0 B, 112.0 B, 112.0 B (stage 10.0: task 29)) local merged bytes read: 0.0 B number of partitions: 1 remote reqs duration: 0 ms remote bytes read to disk: 0.0 B shuffle bytes written total (min, med, max (stageId: taskId)) 703.0 B (0.0 B, 87.0 B, 91.0 B (stage 10.0: task 35))"]; subgraph cluster4 { isCluster="true"; label="WholeStageCodegen (1)\n \nduration: total (min, med, max (stageId: taskId))\n32.5 s (2.2 s, 4.3 s, 4.7 s (stage 10.0: task 32))"; 5 [labelType="html" label="HashAggregate time in aggregation build total (min, med, max (stageId: taskId)) 32.5 s (2.2 s, 4.3 s, 4.7 s (stage 10.0: task 32)) number of output rows: 8"]; } 6 [labelType="html" label="Scan csv number of output rows: 4,988,053 number of files read: 5 metadata time: 0 ms size of files read: 928.8 MiB"]; 2->0; 3->2; 5->3; 6->5; }

AdaptiveSparkPlan isFinalPlan=true

HashAggregate(keys=[], functions=[sum(cast(isnull(ride_id#17) as int)), sum(cast(isnull(rideable_type#18) as int)), sum(cast(isnull(started_at#19) as int)), sum(cast(isnull(ended_at#20) as int)), sum(cast(isnull(start_station_name#21) as int)), sum(cast(isnull(start_station_id#22) as int)), sum(cast(isnull(end_station_name#23) as int)), sum(cast(isnull(end_station_id#24) as int)), sum(cast(isnull(start_lat#25) as int)), sum(cast(isnull(start_lng#26) as int)), sum(cast(isnull(end_lat#27) as int)), sum(cast(isnull(end_lng#28) as int)), sum(cast(isnull(member_casual#29) as int))])

WholeStageCodegen (2)

Exchange SinglePartition, ENSURE_REQUIREMENTS, [plan_id=140]

HashAggregate(keys=[], functions=[partial_sum(cast(isnull(ride_id#17) as int)), partial_sum(cast(isnull(rideable_type#18) as int)), partial_sum(cast(isnull(started_at#19) as int)), partial_sum(cast(isnull(ended_at#20) as int)), partial_sum(cast(isnull(start_station_name#21) as int)), partial_sum(cast(isnull(start_station_id#22) as int)), partial_sum(cast(isnull(end_station_name#23) as int)), partial_sum(cast(isnull(end_station_id#24) as int)), partial_sum(cast(isnull(start_lat#25) as int)), partial_sum(cast(isnull(start_lng#26) as int)), partial_sum(cast(isnull(end_lat#27) as int)), partial_sum(cast(isnull(end_lng#28) as int)), partial_sum(cast(isnull(member_casual#29) as int))])

WholeStageCodegen (1)

FileScan csv [ride_id#17,rideable_type#18,started_at#19,ended_at#20,start_station_name#21,start_station_id#22,end_station_name#23,end_station_id#24,start_lat#25,start_lng#26,end_lat#27,end_lng#28,member_casual#29] Batched: false, DataFilters: [], Format: CSV, Location: InMemoryFileIndex(5 paths)[s3a://rzvde-g9-chernyshev-miron/raw/citibike_data/202507/202507-citibi..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<ride_id:string,rideable_type:string,started_at:timestamp,ended_at:timestamp,start_station_...

Details

== Physical Plan ==
AdaptiveSparkPlan (9)
+- == Final Plan ==
   * HashAggregate (5)
   +- ShuffleQueryStage (4), Statistics(sizeInBytes=896.0 B, rowCount=8)
      +- Exchange (3)
         +- * HashAggregate (2)
            +- Scan csv  (1)
+- == Initial Plan ==
   HashAggregate (8)
   +- Exchange (7)
      +- HashAggregate (6)
         +- Scan csv  (1)


(1) Scan csv 
Output [13]: [ride_id#17, rideable_type#18, started_at#19, ended_at#20, start_station_name#21, start_station_id#22, end_station_name#23, end_station_id#24, start_lat#25, start_lng#26, end_lat#27, end_lng#28, member_casual#29]
Batched: false
Location: InMemoryFileIndex [s3a://rzvde-g9-chernyshev-miron/raw/citibike_data/202507/202507-citibike-tripdata-part00.csv, ... 4 entries]
ReadSchema: struct<ride_id:string,rideable_type:string,started_at:timestamp,ended_at:timestamp,start_station_name:string,start_station_id:string,end_station_name:string,end_station_id:string,start_lat:double,start_lng:double,end_lat:double,end_lng:double,member_casual:string>

(2) HashAggregate [codegen id : 1]
Input [13]: [ride_id#17, rideable_type#18, started_at#19, ended_at#20, start_station_name#21, start_station_id#22, end_station_name#23, end_station_id#24, start_lat#25, start_lng#26, end_lat#27, end_lng#28, member_casual#29]
Keys: []
Functions [13]: [partial_sum(cast(isnull(ride_id#17) as int)), partial_sum(cast(isnull(rideable_type#18) as int)), partial_sum(cast(isnull(started_at#19) as int)), partial_sum(cast(isnull(ended_at#20) as int)), partial_sum(cast(isnull(start_station_name#21) as int)), partial_sum(cast(isnull(start_station_id#22) as int)), partial_sum(cast(isnull(end_station_name#23) as int)), partial_sum(cast(isnull(end_station_id#24) as int)), partial_sum(cast(isnull(start_lat#25) as int)), partial_sum(cast(isnull(start_lng#26) as int)), partial_sum(cast(isnull(end_lat#27) as int)), partial_sum(cast(isnull(end_lng#28) as int)), partial_sum(cast(isnull(member_casual#29) as int))]
Aggregate Attributes [13]: [sum#330L, sum#331L, sum#332L, sum#333L, sum#334L, sum#335L, sum#336L, sum#337L, sum#338L, sum#339L, sum#340L, sum#341L, sum#342L]
Results [13]: [sum#343L, sum#344L, sum#345L, sum#346L, sum#347L, sum#348L, sum#349L, sum#350L, sum#351L, sum#352L, sum#353L, sum#354L, sum#355L]

(3) Exchange
Input [13]: [sum#343L, sum#344L, sum#345L, sum#346L, sum#347L, sum#348L, sum#349L, sum#350L, sum#351L, sum#352L, sum#353L, sum#354L, sum#355L]
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=140]

(4) ShuffleQueryStage
Output [13]: [sum#343L, sum#344L, sum#345L, sum#346L, sum#347L, sum#348L, sum#349L, sum#350L, sum#351L, sum#352L, sum#353L, sum#354L, sum#355L]
Arguments: 0

(5) HashAggregate [codegen id : 2]
Input [13]: [sum#343L, sum#344L, sum#345L, sum#346L, sum#347L, sum#348L, sum#349L, sum#350L, sum#351L, sum#352L, sum#353L, sum#354L, sum#355L]
Keys: []
Functions [13]: [sum(cast(isnull(ride_id#17) as int)), sum(cast(isnull(rideable_type#18) as int)), sum(cast(isnull(started_at#19) as int)), sum(cast(isnull(ended_at#20) as int)), sum(cast(isnull(start_station_name#21) as int)), sum(cast(isnull(start_station_id#22) as int)), sum(cast(isnull(end_station_name#23) as int)), sum(cast(isnull(end_station_id#24) as int)), sum(cast(isnull(start_lat#25) as int)), sum(cast(isnull(start_lng#26) as int)), sum(cast(isnull(end_lat#27) as int)), sum(cast(isnull(end_lng#28) as int)), sum(cast(isnull(member_casual#29) as int))]
Aggregate Attributes [13]: [sum(cast(isnull(ride_id#17) as int))#240L, sum(cast(isnull(rideable_type#18) as int))#242L, sum(cast(isnull(started_at#19) as int))#244L, sum(cast(isnull(ended_at#20) as int))#246L, sum(cast(isnull(start_station_name#21) as int))#248L, sum(cast(isnull(start_station_id#22) as int))#250L, sum(cast(isnull(end_station_name#23) as int))#252L, sum(cast(isnull(end_station_id#24) as int))#254L, sum(cast(isnull(start_lat#25) as int))#256L, sum(cast(isnull(start_lng#26) as int))#258L, sum(cast(isnull(end_lat#27) as int))#260L, sum(cast(isnull(end_lng#28) as int))#262L, sum(cast(isnull(member_casual#29) as int))#264L]
Results [13]: [toprettystring(sum(cast(isnull(ride_id#17) as int))#240L, Some(Etc/UTC)) AS toprettystring(sum(CAST((ride_id IS NULL) AS INT) AS ride_id))#304, toprettystring(sum(cast(isnull(rideable_type#18) as int))#242L, Some(Etc/UTC)) AS toprettystring(sum(CAST((rideable_type IS NULL) AS INT) AS rideable_type))#305, toprettystring(sum(cast(isnull(started_at#19) as int))#244L, Some(Etc/UTC)) AS toprettystring(sum(CAST((started_at IS NULL) AS INT) AS started_at))#306, toprettystring(sum(cast(isnull(ended_at#20) as int))#246L, Some(Etc/UTC)) AS toprettystring(sum(CAST((ended_at IS NULL) AS INT) AS ended_at))#307, toprettystring(sum(cast(isnull(start_station_name#21) as int))#248L, Some(Etc/UTC)) AS toprettystring(sum(CAST((start_station_name IS NULL) AS INT) AS start_station_name))#308, toprettystring(sum(cast(isnull(start_station_id#22) as int))#250L, Some(Etc/UTC)) AS toprettystring(sum(CAST((start_station_id IS NULL) AS INT) AS start_station_id))#309, toprettystring(sum(cast(isnull(end_station_name#23) as int))#252L, Some(Etc/UTC)) AS toprettystring(sum(CAST((end_station_name IS NULL) AS INT) AS end_station_name))#310, toprettystring(sum(cast(isnull(end_station_id#24) as int))#254L, Some(Etc/UTC)) AS toprettystring(sum(CAST((end_station_id IS NULL) AS INT) AS end_station_id))#311, toprettystring(sum(cast(isnull(start_lat#25) as int))#256L, Some(Etc/UTC)) AS toprettystring(sum(CAST((start_lat IS NULL) AS INT) AS start_lat))#312, toprettystring(sum(cast(isnull(start_lng#26) as int))#258L, Some(Etc/UTC)) AS toprettystring(sum(CAST((start_lng IS NULL) AS INT) AS start_lng))#313, toprettystring(sum(cast(isnull(end_lat#27) as int))#260L, Some(Etc/UTC)) AS toprettystring(sum(CAST((end_lat IS NULL) AS INT) AS end_lat))#314, toprettystring(sum(cast(isnull(end_lng#28) as int))#262L, Some(Etc/UTC)) AS toprettystring(sum(CAST((end_lng IS NULL) AS INT) AS end_lng))#315, toprettystring(sum(cast(isnull(member_casual#29) as int))#264L, Some(Etc/UTC)) AS toprettystring(sum(CAST((member_casual IS NULL) AS INT) AS member_casual))#316]

(6) HashAggregate
Input [13]: [ride_id#17, rideable_type#18, started_at#19, ended_at#20, start_station_name#21, start_station_id#22, end_station_name#23, end_station_id#24, start_lat#25, start_lng#26, end_lat#27, end_lng#28, member_casual#29]
Keys: []
Functions [13]: [partial_sum(cast(isnull(ride_id#17) as int)), partial_sum(cast(isnull(rideable_type#18) as int)), partial_sum(cast(isnull(started_at#19) as int)), partial_sum(cast(isnull(ended_at#20) as int)), partial_sum(cast(isnull(start_station_name#21) as int)), partial_sum(cast(isnull(start_station_id#22) as int)), partial_sum(cast(isnull(end_station_name#23) as int)), partial_sum(cast(isnull(end_station_id#24) as int)), partial_sum(cast(isnull(start_lat#25) as int)), partial_sum(cast(isnull(start_lng#26) as int)), partial_sum(cast(isnull(end_lat#27) as int)), partial_sum(cast(isnull(end_lng#28) as int)), partial_sum(cast(isnull(member_casual#29) as int))]
Aggregate Attributes [13]: [sum#330L, sum#331L, sum#332L, sum#333L, sum#334L, sum#335L, sum#336L, sum#337L, sum#338L, sum#339L, sum#340L, sum#341L, sum#342L]
Results [13]: [sum#343L, sum#344L, sum#345L, sum#346L, sum#347L, sum#348L, sum#349L, sum#350L, sum#351L, sum#352L, sum#353L, sum#354L, sum#355L]

(7) Exchange
Input [13]: [sum#343L, sum#344L, sum#345L, sum#346L, sum#347L, sum#348L, sum#349L, sum#350L, sum#351L, sum#352L, sum#353L, sum#354L, sum#355L]
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=131]

(8) HashAggregate
Input [13]: [sum#343L, sum#344L, sum#345L, sum#346L, sum#347L, sum#348L, sum#349L, sum#350L, sum#351L, sum#352L, sum#353L, sum#354L, sum#355L]
Keys: []
Functions [13]: [sum(cast(isnull(ride_id#17) as int)), sum(cast(isnull(rideable_type#18) as int)), sum(cast(isnull(started_at#19) as int)), sum(cast(isnull(ended_at#20) as int)), sum(cast(isnull(start_station_name#21) as int)), sum(cast(isnull(start_station_id#22) as int)), sum(cast(isnull(end_station_name#23) as int)), sum(cast(isnull(end_station_id#24) as int)), sum(cast(isnull(start_lat#25) as int)), sum(cast(isnull(start_lng#26) as int)), sum(cast(isnull(end_lat#27) as int)), sum(cast(isnull(end_lng#28) as int)), sum(cast(isnull(member_casual#29) as int))]
Aggregate Attributes [13]: [sum(cast(isnull(ride_id#17) as int))#240L, sum(cast(isnull(rideable_type#18) as int))#242L, sum(cast(isnull(started_at#19) as int))#244L, sum(cast(isnull(ended_at#20) as int))#246L, sum(cast(isnull(start_station_name#21) as int))#248L, sum(cast(isnull(start_station_id#22) as int))#250L, sum(cast(isnull(end_station_name#23) as int))#252L, sum(cast(isnull(end_station_id#24) as int))#254L, sum(cast(isnull(start_lat#25) as int))#256L, sum(cast(isnull(start_lng#26) as int))#258L, sum(cast(isnull(end_lat#27) as int))#260L, sum(cast(isnull(end_lng#28) as int))#262L, sum(cast(isnull(member_casual#29) as int))#264L]
Results [13]: [toprettystring(sum(cast(isnull(ride_id#17) as int))#240L, Some(Etc/UTC)) AS toprettystring(sum(CAST((ride_id IS NULL) AS INT) AS ride_id))#304, toprettystring(sum(cast(isnull(rideable_type#18) as int))#242L, Some(Etc/UTC)) AS toprettystring(sum(CAST((rideable_type IS NULL) AS INT) AS rideable_type))#305, toprettystring(sum(cast(isnull(started_at#19) as int))#244L, Some(Etc/UTC)) AS toprettystring(sum(CAST((started_at IS NULL) AS INT) AS started_at))#306, toprettystring(sum(cast(isnull(ended_at#20) as int))#246L, Some(Etc/UTC)) AS toprettystring(sum(CAST((ended_at IS NULL) AS INT) AS ended_at))#307, toprettystring(sum(cast(isnull(start_station_name#21) as int))#248L, Some(Etc/UTC)) AS toprettystring(sum(CAST((start_station_name IS NULL) AS INT) AS start_station_name))#308, toprettystring(sum(cast(isnull(start_station_id#22) as int))#250L, Some(Etc/UTC)) AS toprettystring(sum(CAST((start_station_id IS NULL) AS INT) AS start_station_id))#309, toprettystring(sum(cast(isnull(end_station_name#23) as int))#252L, Some(Etc/UTC)) AS toprettystring(sum(CAST((end_station_name IS NULL) AS INT) AS end_station_name))#310, toprettystring(sum(cast(isnull(end_station_id#24) as int))#254L, Some(Etc/UTC)) AS toprettystring(sum(CAST((end_station_id IS NULL) AS INT) AS end_station_id))#311, toprettystring(sum(cast(isnull(start_lat#25) as int))#256L, Some(Etc/UTC)) AS toprettystring(sum(CAST((start_lat IS NULL) AS INT) AS start_lat))#312, toprettystring(sum(cast(isnull(start_lng#26) as int))#258L, Some(Etc/UTC)) AS toprettystring(sum(CAST((start_lng IS NULL) AS INT) AS start_lng))#313, toprettystring(sum(cast(isnull(end_lat#27) as int))#260L, Some(Etc/UTC)) AS toprettystring(sum(CAST((end_lat IS NULL) AS INT) AS end_lat))#314, toprettystring(sum(cast(isnull(end_lng#28) as int))#262L, Some(Etc/UTC)) AS toprettystring(sum(CAST((end_lng IS NULL) AS INT) AS end_lng))#315, toprettystring(sum(cast(isnull(member_casual#29) as int))#264L, Some(Etc/UTC)) AS toprettystring(sum(CAST((member_casual IS NULL) AS INT) AS member_casual))#316]

(9) AdaptiveSparkPlan
Output [13]: [toprettystring(sum(CAST((ride_id IS NULL) AS INT) AS ride_id))#304, toprettystring(sum(CAST((rideable_type IS NULL) AS INT) AS rideable_type))#305, toprettystring(sum(CAST((started_at IS NULL) AS INT) AS started_at))#306, toprettystring(sum(CAST((ended_at IS NULL) AS INT) AS ended_at))#307, toprettystring(sum(CAST((start_station_name IS NULL) AS INT) AS start_station_name))#308, toprettystring(sum(CAST((start_station_id IS NULL) AS INT) AS start_station_id))#309, toprettystring(sum(CAST((end_station_name IS NULL) AS INT) AS end_station_name))#310, toprettystring(sum(CAST((end_station_id IS NULL) AS INT) AS end_station_id))#311, toprettystring(sum(CAST((start_lat IS NULL) AS INT) AS start_lat))#312, toprettystring(sum(CAST((start_lng IS NULL) AS INT) AS start_lng))#313, toprettystring(sum(CAST((end_lat IS NULL) AS INT) AS end_lat))#314, toprettystring(sum(CAST((end_lng IS NULL) AS INT) AS end_lng))#315, toprettystring(sum(CAST((member_casual IS NULL) AS INT) AS member_casual))#316]
Arguments: isFinalPlan=true