g10-chepusov-vladislav-stage2 - Details for Query 5

Submitted Time: 2026/08/18 07:27:34
Duration: 48 s
Succeeded Jobs: 14

Show the Stage ID and Task ID that corresponds to the max metric

digraph G { 0 [labelType="html" label=" AdaptiveSparkPlan "]; 1 [labelType="html" label="Execute InsertIntoHadoopFsRelationCommand task commit time total (min, med, max (stageId: taskId)) 12.0 s (1.4 s, 2.0 s, 3.4 s (stage 26.0: task 78)) number of written files: 12 job commit time: 17.4 s number of output rows: 4,982,002 number of dynamic part: 2 written output: 198.8 MiB"]; 2 [labelType="html" label=" WriteFiles "]; subgraph cluster3 { isCluster="true"; label="WholeStageCodegen (1)\n \nduration: total (min, med, max (stageId: taskId))\n37.6 s (3.7 s, 5.7 s, 8.8 s (stage 26.0: task 74))"; 4 [labelType="html" label="Sort sort time total (min, med, max (stageId: taskId)) 389 ms (38 ms, 54 ms, 111 ms (stage 26.0: task 75)) peak memory total (min, med, max (stageId: taskId)) 1456.0 MiB (176.0 MiB, 256.0 MiB, 256.0 MiB (stage 26.0: task 74)) spill size total (min, med, max (stageId: taskId)) 0.0 B (0.0 B, 0.0 B, 0.0 B (stage 26.0: task 74))"]; } 5 [labelType="html" label="InMemoryTableScan number of output rows: 4,982,002"]; 6 [labelType="html" label=" AdaptiveSparkPlan "]; subgraph cluster7 { isCluster="true"; label="WholeStageCodegen (3)\n \nduration: total (min, med, max (stageId: taskId))\n0 ms (0 ms, 0 ms, 0 ms (stage 26.0: task 74))"; 8 [labelType="html" label="Filter number of output rows: 0"]; } 9 [labelType="html" label="Window spill size total (min, med, max (stageId: taskId)) 0.0 B (0.0 B, 0.0 B, 0.0 B (stage 26.0: task 74))"]; subgraph cluster10 { isCluster="true"; label="WholeStageCodegen (2)\n \nduration: total (min, med, max (stageId: taskId))\n0 ms (0 ms, 0 ms, 0 ms (stage 26.0: task 74))"; 11 [labelType="html" label="Sort sort time total (min, med, max (stageId: taskId)) 0 ms (0 ms, 0 ms, 0 ms (stage 26.0: task 74)) peak memory total (min, med, max (stageId: taskId)) 0.0 B (0.0 B, 0.0 B, 0.0 B (stage 26.0: task 74)) spill size total (min, med, max (stageId: taskId)) 0.0 B (0.0 B, 0.0 B, 0.0 B (stage 26.0: task 74))"]; } 12 [labelType="html" label=" AQEShuffleRead "]; 13 [labelType="html" label="Exchange shuffle records written: 0 local merged chunks fetched: 0 shuffle write time total (min, med, max (stageId: taskId)) 0 ms (0 ms, 0 ms, 0 ms (stage 26.0: task 74)) remote merged bytes read total (min, med, max (stageId: taskId)) 0.0 B (0.0 B, 0.0 B, 0.0 B (stage 26.0: task 74)) local merged blocks fetched: 0 corrupt merged block chunks: 0 remote merged reqs duration total (min, med, max (stageId: taskId)) 0 ms (0 ms, 0 ms, 0 ms (stage 26.0: task 74)) remote merged blocks fetched: 0 records read: 0 local bytes read total (min, med, max (stageId: taskId)) 0.0 B (0.0 B, 0.0 B, 0.0 B (stage 26.0: task 74)) fetch wait time total (min, med, max (stageId: taskId)) 0 ms (0 ms, 0 ms, 0 ms (stage 26.0: task 74)) remote bytes read total (min, med, max (stageId: taskId)) 0.0 B (0.0 B, 0.0 B, 0.0 B (stage 26.0: task 74)) merged fetch fallback count: 0 local blocks read: 0 remote merged chunks fetched: 0 remote blocks read: 0 data size total (min, med, max (stageId: taskId)) 0.0 B (0.0 B, 0.0 B, 0.0 B (stage 26.0: task 74)) local merged bytes read total (min, med, max (stageId: taskId)) 0.0 B (0.0 B, 0.0 B, 0.0 B (stage 26.0: task 74)) number of partitions: 0 remote reqs duration total (min, med, max (stageId: taskId)) 0 ms (0 ms, 0 ms, 0 ms (stage 26.0: task 74)) remote bytes read to disk total (min, med, max (stageId: taskId)) 0.0 B (0.0 B, 0.0 B, 0.0 B (stage 26.0: task 74)) shuffle bytes written total (min, med, max (stageId: taskId)) 0.0 B (0.0 B, 0.0 B, 0.0 B (stage 26.0: task 74))"]; subgraph cluster14 { isCluster="true"; label="WholeStageCodegen (1)\n \nduration: total (min, med, max (stageId: taskId))\n0 ms (0 ms, 0 ms, 0 ms (stage 26.0: task 74))"; 15 [labelType="html" label=" Project "]; 16 [labelType="html" label=" Project "]; } 17 [labelType="html" label="InMemoryTableScan number of output rows: 0"]; 18 [labelType="html" label="Scan csv number of output rows: 0"]; 1->0; 2->1; 4->2; 5->4; 6->5; 8->6; 9->8; 11->9; 12->11; 13->12; 15->13; 16->15; 17->16; 18->17; }

AdaptiveSparkPlan isFinalPlan=true

Execute InsertIntoHadoopFsRelationCommand s3a://rzvde-g10-chepusov-vladislav/raw_valid/citibike_data, false, [year#513, month#514], Parquet, [partitionOverwriteMode=dynamic, __partition_columns=["year","month"], path=s3a://rzvde-g10-chepusov-vladislav/raw_valid/citibike_data/], Overwrite, [ride_id, rideable_type, started_at, ended_at, start_station_name, start_station_id, end_station_name, end_station_id, start_lat, start_lng, end_lat, end_lng, member_casual, _source_file, _processed_dttm, year, month, _start_station_ride_num]

WriteFiles

Sort [year#513 ASC NULLS FIRST, month#514 ASC NULLS FIRST], false, 0

WholeStageCodegen (1)

InMemoryTableScan [ride_id#26, rideable_type#27, started_at#28, ended_at#29, start_station_name#30, start_station_id#31, end_station_name#32, end_station_id#33, start_lat#34, start_lng#35, end_lat#36, end_lng#37, member_casual#38, _source_file#493, _processed_dttm#494, year#513, month#514, _start_station_ride_num#536]

AdaptiveSparkPlan isFinalPlan=true

Filter (isnotnull(is_ride_id_valid#474) AND ((is_ride_id_valid#474 AND is_duration_valid#475) AND is_station_valid#476))

WholeStageCodegen (3)

Window [row_number() windowspecdefinition(start_station_id#31, started_at#28 ASC NULLS FIRST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS _start_station_ride_num#536], [start_station_id#31], [started_at#28 ASC NULLS FIRST]

Sort [start_station_id#31 ASC NULLS FIRST, started_at#28 ASC NULLS FIRST], false, 0

WholeStageCodegen (2)

AQEShuffleRead coalesced

Exchange hashpartitioning(start_station_id#31, 200), ENSURE_REQUIREMENTS, [plan_id=207]

Project [ride_id#26, rideable_type#27, started_at#28, ended_at#29, start_station_name#30, start_station_id#31, end_station_name#32, end_station_id#33, start_lat#34, start_lng#35, end_lat#36, end_lng#37, member_casual#38, is_ride_id_valid#474, is_duration_valid#475, is_station_valid#476, _source_file#493, 2026-08-18 07:26:21.132535 AS _processed_dttm#494, year(cast(started_at#28 as date)) AS year#513, month(cast(started_at#28 as date)) AS month#514]

Project [ride_id#26, rideable_type#27, started_at#28, ended_at#29, start_station_name#30, start_station_id#31, end_station_name#32, end_station_id#33, start_lat#34, start_lng#35, end_lat#36, end_lng#37, member_casual#38, (isnotnull(ride_id#26) AND NOT (trim(ride_id#26, None) = )) AS is_ride_id_valid#474, coalesce((ended_at#29 > started_at#28), false) AS is_duration_valid#475, coalesce(NOT (end_station_id#33 = start_station_id#31), false) AS is_station_valid#476, input_file_name() AS _source_file#493]

WholeStageCodegen (1)

InMemoryTableScan [end_lat#36, end_lng#37, end_station_id#33, end_station_name#32, ended_at#29, member_casual#38, ride_id#26, rideable_type#27, start_lat#34, start_lng#35, start_station_id#31, start_station_name#30, started_at#28]

FileScan csv [ride_id#0,rideable_type#1,started_at#2,ended_at#3,start_station_name#4,start_station_id#5,end_station_name#6,end_station_id#7,start_lat#8,start_lng#9,end_lat#10,end_lng#11,member_casual#12] Batched: false, DataFilters: [], Format: CSV, Location: InMemoryFileIndex(6 paths)[s3a://rzvde-g10-chepusov-vladislav/raw/citibike_data/202508/202508-cit..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<ride_id:string,rideable_type:string,started_at:timestamp,ended_at:timestamp,start_station_...

Details

== Physical Plan ==
AdaptiveSparkPlan (20)
+- == Final Plan ==
   Execute InsertIntoHadoopFsRelationCommand (16)
   +- WriteFiles (15)
      +- * Sort (14)
         +- TableCacheQueryStage (13), Statistics(sizeInBytes=835.8 MiB, rowCount=4.98E+6)
            +- InMemoryTableScan (1)
                  +- InMemoryRelation (2)
                        +- AdaptiveSparkPlan (12)
                           +- Filter (11)
                              +- Window (10)
                                 +- Sort (9)
                                    +- Exchange (8)
                                       +- Project (7)
                                          +- Project (6)
                                             +- InMemoryTableScan (3)
                                                   +- InMemoryRelation (4)
                                                         +- Scan csv  (5)
+- == Initial Plan ==
   Execute InsertIntoHadoopFsRelationCommand (19)
   +- WriteFiles (18)
      +- Sort (17)
         +- InMemoryTableScan (1)
               +- InMemoryRelation (2)
                     +- AdaptiveSparkPlan (12)
                        +- Filter (11)
                           +- Window (10)
                              +- Sort (9)
                                 +- Exchange (8)
                                    +- Project (7)
                                       +- Project (6)
                                          +- InMemoryTableScan (3)
                                                +- InMemoryRelation (4)
                                                      +- Scan csv  (5)


(1) InMemoryTableScan
Output [18]: [ride_id#26, rideable_type#27, started_at#28, ended_at#29, start_station_name#30, start_station_id#31, end_station_name#32, end_station_id#33, start_lat#34, start_lng#35, end_lat#36, end_lng#37, member_casual#38, _source_file#493, _processed_dttm#494, year#513, month#514, _start_station_ride_num#536]
Arguments: [ride_id#26, rideable_type#27, started_at#28, ended_at#29, start_station_name#30, start_station_id#31, end_station_name#32, end_station_id#33, start_lat#34, start_lng#35, end_lat#36, end_lng#37, member_casual#38, _source_file#493, _processed_dttm#494, year#513, month#514, _start_station_ride_num#536]

(2) InMemoryRelation
Arguments: [ride_id#26, rideable_type#27, started_at#28, ended_at#29, start_station_name#30, start_station_id#31, end_station_name#32, end_station_id#33, start_lat#34, start_lng#35, end_lat#36, end_lng#37, member_casual#38, is_ride_id_valid#474, is_duration_valid#475, is_station_valid#476, _source_file#493, _processed_dttm#494, year#513, month#514, _start_station_ride_num#536], CachedRDDBuilder(org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer@3908c132,StorageLevel(disk, memory, deserialized, 1 replicas),AdaptiveSparkPlan isFinalPlan=true
+- == Final Plan ==
   *(3) Filter (isnotnull(is_ride_id_valid#474) AND ((is_ride_id_valid#474 AND is_duration_valid#475) AND is_station_valid#476))
   +- Window [row_number() windowspecdefinition(start_station_id#31, started_at#28 ASC NULLS FIRST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS _start_station_ride_num#536], [start_station_id#31], [started_at#28 ASC NULLS FIRST]
      +- *(2) Sort [start_station_id#31 ASC NULLS FIRST, started_at#28 ASC NULLS FIRST], false, 0
         +- AQEShuffleRead coalesced
            +- ShuffleQueryStage 1
               +- Exchange hashpartitioning(start_station_id#31, 200), ENSURE_REQUIREMENTS, [plan_id=207]
                  +- *(1) Project [ride_id#26, rideable_type#27, started_at#28, ended_at#29, start_station_name#30, start_station_id#31, end_station_name#32, end_station_id#33, start_lat#34, start_lng#35, end_lat#36, end_lng#37, member_casual#38, is_ride_id_valid#474, is_duration_valid#475, is_station_valid#476, _source_file#493, 2026-08-18 07:26:21.132535 AS _processed_dttm#494, year(cast(started_at#28 as date)) AS year#513, month(cast(started_at#28 as date)) AS month#514]
                     +- *(1) Project [ride_id#26, rideable_type#27, started_at#28, ended_at#29, start_station_name#30, start_station_id#31, end_station_name#32, end_station_id#33, start_lat#34, start_lng#35, end_lat#36, end_lng#37, member_casual#38, (isnotnull(ride_id#26) AND NOT (trim(ride_id#26, None) = )) AS is_ride_id_valid#474, coalesce((ended_at#29 > started_at#28), false) AS is_duration_valid#475, coalesce(NOT (end_station_id#33 = start_station_id#31), false) AS is_station_valid#476, input_file_name() AS _source_file#493]
                        +- TableCacheQueryStage 0
                           +- InMemoryTableScan [end_lat#36, end_lng#37, end_station_id#33, end_station_name#32, ended_at#29, member_casual#38, ride_id#26, rideable_type#27, start_lat#34, start_lng#35, start_station_id#31, start_station_name#30, started_at#28]
                                 +- InMemoryRelation [ride_id#26, rideable_type#27, started_at#28, ended_at#29, start_station_name#30, start_station_id#31, end_station_name#32, end_station_id#33, start_lat#34, start_lng#35, end_lat#36, end_lng#37, member_casual#38], StorageLevel(disk, memory, deserialized, 1 replicas)
                                       +- FileScan csv [ride_id#0,rideable_type#1,started_at#2,ended_at#3,start_station_name#4,start_station_id#5,end_station_name#6,end_station_id#7,start_lat#8,start_lng#9,end_lat#10,end_lng#11,member_casual#12] Batched: false, DataFilters: [], Format: CSV, Location: InMemoryFileIndex(6 paths)[s3a://rzvde-g10-chepusov-vladislav/raw/citibike_data/202508/202508-cit..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<ride_id:string,rideable_type:string,started_at:timestamp,ended_at:timestamp,start_station_...
+- == Initial Plan ==
   Filter (isnotnull(is_ride_id_valid#474) AND ((is_ride_id_valid#474 AND is_duration_valid#475) AND is_station_valid#476))
   +- Window [row_number() windowspecdefinition(start_station_id#31, started_at#28 ASC NULLS FIRST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS _start_station_ride_num#536], [start_station_id#31], [started_at#28 ASC NULLS FIRST]
      +- Sort [start_station_id#31 ASC NULLS FIRST, started_at#28 ASC NULLS FIRST], false, 0
         +- Exchange hashpartitioning(start_station_id#31, 200), ENSURE_REQUIREMENTS, [plan_id=68]
            +- Project [ride_id#26, rideable_type#27, started_at#28, ended_at#29, start_station_name#30, start_station_id#31, end_station_name#32, end_station_id#33, start_lat#34, start_lng#35, end_lat#36, end_lng#37, member_casual#38, is_ride_id_valid#474, is_duration_valid#475, is_station_valid#476, _source_file#493, 2026-08-18 07:26:21.132535 AS _processed_dttm#494, year(cast(started_at#28 as date)) AS year#513, month(cast(started_at#28 as date)) AS month#514]
               +- Project [ride_id#26, rideable_type#27, started_at#28, ended_at#29, start_station_name#30, start_station_id#31, end_station_name#32, end_station_id#33, start_lat#34, start_lng#35, end_lat#36, end_lng#37, member_casual#38, (isnotnull(ride_id#26) AND NOT (trim(ride_id#26, None) = )) AS is_ride_id_valid#474, coalesce((ended_at#29 > started_at#28), false) AS is_duration_valid#475, coalesce(NOT (end_station_id#33 = start_station_id#31), false) AS is_station_valid#476, input_file_name() AS _source_file#493]
                  +- InMemoryTableScan [end_lat#36, end_lng#37, end_station_id#33, end_station_name#32, ended_at#29, member_casual#38, ride_id#26, rideable_type#27, start_lat#34, start_lng#35, start_station_id#31, start_station_name#30, started_at#28]
                        +- InMemoryRelation [ride_id#26, rideable_type#27, started_at#28, ended_at#29, start_station_name#30, start_station_id#31, end_station_name#32, end_station_id#33, start_lat#34, start_lng#35, end_lat#36, end_lng#37, member_casual#38], StorageLevel(disk, memory, deserialized, 1 replicas)
                              +- FileScan csv [ride_id#0,rideable_type#1,started_at#2,ended_at#3,start_station_name#4,start_station_id#5,end_station_name#6,end_station_id#7,start_lat#8,start_lng#9,end_lat#10,end_lng#11,member_casual#12] Batched: false, DataFilters: [], Format: CSV, Location: InMemoryFileIndex(6 paths)[s3a://rzvde-g10-chepusov-vladislav/raw/citibike_data/202508/202508-cit..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<ride_id:string,rideable_type:string,started_at:timestamp,ended_at:timestamp,start_station_...
,None)

(3) InMemoryTableScan
Output [13]: [end_lat#36, end_lng#37, end_station_id#33, end_station_name#32, ended_at#29, member_casual#38, ride_id#26, rideable_type#27, start_lat#34, start_lng#35, start_station_id#31, start_station_name#30, started_at#28]
Arguments: [end_lat#36, end_lng#37, end_station_id#33, end_station_name#32, ended_at#29, member_casual#38, ride_id#26, rideable_type#27, start_lat#34, start_lng#35, start_station_id#31, start_station_name#30, started_at#28]

(4) InMemoryRelation
Arguments: [ride_id#26, rideable_type#27, started_at#28, ended_at#29, start_station_name#30, start_station_id#31, end_station_name#32, end_station_id#33, start_lat#34, start_lng#35, end_lat#36, end_lng#37, member_casual#38], CachedRDDBuilder(org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer@3908c132,StorageLevel(disk, memory, deserialized, 1 replicas),FileScan csv [ride_id#0,rideable_type#1,started_at#2,ended_at#3,start_station_name#4,start_station_id#5,end_station_name#6,end_station_id#7,start_lat#8,start_lng#9,end_lat#10,end_lng#11,member_casual#12] Batched: false, DataFilters: [], Format: CSV, Location: InMemoryFileIndex(6 paths)[s3a://rzvde-g10-chepusov-vladislav/raw/citibike_data/202508/202508-cit..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<ride_id:string,rideable_type:string,started_at:timestamp,ended_at:timestamp,start_station_...
,None)

(5) Scan csv 
Output [13]: [ride_id#0, rideable_type#1, started_at#2, ended_at#3, start_station_name#4, start_station_id#5, end_station_name#6, end_station_id#7, start_lat#8, start_lng#9, end_lat#10, end_lng#11, member_casual#12]
Batched: false
Location: InMemoryFileIndex [s3a://rzvde-g10-chepusov-vladislav/raw/citibike_data/202508/202508-citibike-tripdata_1.csv, ... 5 entries]
ReadSchema: struct<ride_id:string,rideable_type:string,started_at:timestamp,ended_at:timestamp,start_station_name:string,start_station_id:string,end_station_name:string,end_station_id:string,start_lat:float,start_lng:float,end_lat:float,end_lng:float,member_casual:string>

(6) Project
Output [17]: [ride_id#26, rideable_type#27, started_at#28, ended_at#29, start_station_name#30, start_station_id#31, end_station_name#32, end_station_id#33, start_lat#34, start_lng#35, end_lat#36, end_lng#37, member_casual#38, (isnotnull(ride_id#26) AND NOT (trim(ride_id#26, None) = )) AS is_ride_id_valid#474, coalesce((ended_at#29 > started_at#28), false) AS is_duration_valid#475, coalesce(NOT (end_station_id#33 = start_station_id#31), false) AS is_station_valid#476, input_file_name() AS _source_file#493]
Input [13]: [end_lat#36, end_lng#37, end_station_id#33, end_station_name#32, ended_at#29, member_casual#38, ride_id#26, rideable_type#27, start_lat#34, start_lng#35, start_station_id#31, start_station_name#30, started_at#28]

(7) Project
Output [20]: [ride_id#26, rideable_type#27, started_at#28, ended_at#29, start_station_name#30, start_station_id#31, end_station_name#32, end_station_id#33, start_lat#34, start_lng#35, end_lat#36, end_lng#37, member_casual#38, is_ride_id_valid#474, is_duration_valid#475, is_station_valid#476, _source_file#493, 2026-08-18 07:26:21.132535 AS _processed_dttm#494, year(cast(started_at#28 as date)) AS year#513, month(cast(started_at#28 as date)) AS month#514]
Input [17]: [ride_id#26, rideable_type#27, started_at#28, ended_at#29, start_station_name#30, start_station_id#31, end_station_name#32, end_station_id#33, start_lat#34, start_lng#35, end_lat#36, end_lng#37, member_casual#38, is_ride_id_valid#474, is_duration_valid#475, is_station_valid#476, _source_file#493]

(8) Exchange
Input [20]: [ride_id#26, rideable_type#27, started_at#28, ended_at#29, start_station_name#30, start_station_id#31, end_station_name#32, end_station_id#33, start_lat#34, start_lng#35, end_lat#36, end_lng#37, member_casual#38, is_ride_id_valid#474, is_duration_valid#475, is_station_valid#476, _source_file#493, _processed_dttm#494, year#513, month#514]
Arguments: hashpartitioning(start_station_id#31, 200), ENSURE_REQUIREMENTS, [plan_id=432]

(9) Sort
Input [20]: [ride_id#26, rideable_type#27, started_at#28, ended_at#29, start_station_name#30, start_station_id#31, end_station_name#32, end_station_id#33, start_lat#34, start_lng#35, end_lat#36, end_lng#37, member_casual#38, is_ride_id_valid#474, is_duration_valid#475, is_station_valid#476, _source_file#493, _processed_dttm#494, year#513, month#514]
Arguments: [start_station_id#31 ASC NULLS FIRST, started_at#28 ASC NULLS FIRST], false, 0

(10) Window
Input [20]: [ride_id#26, rideable_type#27, started_at#28, ended_at#29, start_station_name#30, start_station_id#31, end_station_name#32, end_station_id#33, start_lat#34, start_lng#35, end_lat#36, end_lng#37, member_casual#38, is_ride_id_valid#474, is_duration_valid#475, is_station_valid#476, _source_file#493, _processed_dttm#494, year#513, month#514]
Arguments: [row_number() windowspecdefinition(start_station_id#31, started_at#28 ASC NULLS FIRST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS _start_station_ride_num#536], [start_station_id#31], [started_at#28 ASC NULLS FIRST]

(11) Filter
Input [21]: [ride_id#26, rideable_type#27, started_at#28, ended_at#29, start_station_name#30, start_station_id#31, end_station_name#32, end_station_id#33, start_lat#34, start_lng#35, end_lat#36, end_lng#37, member_casual#38, is_ride_id_valid#474, is_duration_valid#475, is_station_valid#476, _source_file#493, _processed_dttm#494, year#513, month#514, _start_station_ride_num#536]
Condition : (isnotnull(is_ride_id_valid#474) AND ((is_ride_id_valid#474 AND is_duration_valid#475) AND is_station_valid#476))

(12) AdaptiveSparkPlan
Output [21]: [ride_id#26, rideable_type#27, started_at#28, ended_at#29, start_station_name#30, start_station_id#31, end_station_name#32, end_station_id#33, start_lat#34, start_lng#35, end_lat#36, end_lng#37, member_casual#38, is_ride_id_valid#474, is_duration_valid#475, is_station_valid#476, _source_file#493, _processed_dttm#494, year#513, month#514, _start_station_ride_num#536]
Arguments: isFinalPlan=false

(13) TableCacheQueryStage
Output [18]: [ride_id#26, rideable_type#27, started_at#28, ended_at#29, start_station_name#30, start_station_id#31, end_station_name#32, end_station_id#33, start_lat#34, start_lng#35, end_lat#36, end_lng#37, member_casual#38, _source_file#493, _processed_dttm#494, year#513, month#514, _start_station_ride_num#536]
Arguments: 0

(14) Sort [codegen id : 1]
Input [18]: [ride_id#26, rideable_type#27, started_at#28, ended_at#29, start_station_name#30, start_station_id#31, end_station_name#32, end_station_id#33, start_lat#34, start_lng#35, end_lat#36, end_lng#37, member_casual#38, _source_file#493, _processed_dttm#494, year#513, month#514, _start_station_ride_num#536]
Arguments: [year#513 ASC NULLS FIRST, month#514 ASC NULLS FIRST], false, 0

(15) WriteFiles
Input [18]: [ride_id#26, rideable_type#27, started_at#28, ended_at#29, start_station_name#30, start_station_id#31, end_station_name#32, end_station_id#33, start_lat#34, start_lng#35, end_lat#36, end_lng#37, member_casual#38, _source_file#493, _processed_dttm#494, year#513, month#514, _start_station_ride_num#536]

(16) Execute InsertIntoHadoopFsRelationCommand
Input: []
Arguments: s3a://rzvde-g10-chepusov-vladislav/raw_valid/citibike_data, false, [year#513, month#514], Parquet, [partitionOverwriteMode=dynamic, __partition_columns=["year","month"], path=s3a://rzvde-g10-chepusov-vladislav/raw_valid/citibike_data/], Overwrite, [ride_id, rideable_type, started_at, ended_at, start_station_name, start_station_id, end_station_name, end_station_id, start_lat, start_lng, end_lat, end_lng, member_casual, _source_file, _processed_dttm, year, month, _start_station_ride_num]

(17) Sort
Input [18]: [ride_id#26, rideable_type#27, started_at#28, ended_at#29, start_station_name#30, start_station_id#31, end_station_name#32, end_station_id#33, start_lat#34, start_lng#35, end_lat#36, end_lng#37, member_casual#38, _source_file#493, _processed_dttm#494, year#513, month#514, _start_station_ride_num#536]
Arguments: [year#513 ASC NULLS FIRST, month#514 ASC NULLS FIRST], false, 0

(18) WriteFiles
Input [18]: [ride_id#26, rideable_type#27, started_at#28, ended_at#29, start_station_name#30, start_station_id#31, end_station_name#32, end_station_id#33, start_lat#34, start_lng#35, end_lat#36, end_lng#37, member_casual#38, _source_file#493, _processed_dttm#494, year#513, month#514, _start_station_ride_num#536]

(19) Execute InsertIntoHadoopFsRelationCommand
Input: []
Arguments: s3a://rzvde-g10-chepusov-vladislav/raw_valid/citibike_data, false, [year#513, month#514], Parquet, [partitionOverwriteMode=dynamic, __partition_columns=["year","month"], path=s3a://rzvde-g10-chepusov-vladislav/raw_valid/citibike_data/], Overwrite, [ride_id, rideable_type, started_at, ended_at, start_station_name, start_station_id, end_station_name, end_station_id, start_lat, start_lng, end_lat, end_lng, member_casual, _source_file, _processed_dttm, year, month, _start_station_ride_num]

(20) AdaptiveSparkPlan
Output: []
Arguments: isFinalPlan=true