g10-chepusov-vladislav-stage2 - Details for Query 4

Submitted Time: 2026/08/18 07:27:24
Duration: 10 s
Succeeded Jobs: 13

Show the Stage ID and Task ID that corresponds to the max metric

digraph G { 0 [labelType="html" label=" AdaptiveSparkPlan "]; 1 [labelType="html" label="Execute InsertIntoHadoopFsRelationCommand task commit time total (min, med, max (stageId: taskId)) 3.9 s (493 ms, 645 ms, 842 ms (stage 24.0: task 70)) number of written files: 6 job commit time: 4.1 s number of output rows: 139,531 number of dynamic part: 0 written output: 31.6 MiB"]; 2 [labelType="html" label=" WriteFiles "]; 3 [labelType="html" label="InMemoryTableScan number of output rows: 139,531"]; 4 [labelType="html" label=" AdaptiveSparkPlan "]; subgraph cluster5 { isCluster="true"; label="WholeStageCodegen (3)\n \nduration: total (min, med, max (stageId: taskId))\n0 ms (0 ms, 0 ms, 0 ms (stage 24.0: task 69))"; 6 [labelType="html" label="Filter number of output rows: 0"]; } 7 [labelType="html" label="Window spill size total (min, med, max (stageId: taskId)) 0.0 B (0.0 B, 0.0 B, 0.0 B (stage 24.0: task 69))"]; subgraph cluster8 { isCluster="true"; label="WholeStageCodegen (2)\n \nduration: total (min, med, max (stageId: taskId))\n0 ms (0 ms, 0 ms, 0 ms (stage 24.0: task 69))"; 9 [labelType="html" label="Sort sort time total (min, med, max (stageId: taskId)) 0 ms (0 ms, 0 ms, 0 ms (stage 24.0: task 69)) peak memory total (min, med, max (stageId: taskId)) 0.0 B (0.0 B, 0.0 B, 0.0 B (stage 24.0: task 69)) spill size total (min, med, max (stageId: taskId)) 0.0 B (0.0 B, 0.0 B, 0.0 B (stage 24.0: task 69))"]; } 10 [labelType="html" label=" AQEShuffleRead "]; 11 [labelType="html" label="Exchange shuffle records written: 0 local merged chunks fetched: 0 shuffle write time total (min, med, max (stageId: taskId)) 0 ms (0 ms, 0 ms, 0 ms (stage 24.0: task 69)) remote merged bytes read total (min, med, max (stageId: taskId)) 0.0 B (0.0 B, 0.0 B, 0.0 B (stage 24.0: task 69)) local merged blocks fetched: 0 corrupt merged block chunks: 0 remote merged reqs duration total (min, med, max (stageId: taskId)) 0 ms (0 ms, 0 ms, 0 ms (stage 24.0: task 69)) remote merged blocks fetched: 0 records read: 0 local bytes read total (min, med, max (stageId: taskId)) 0.0 B (0.0 B, 0.0 B, 0.0 B (stage 24.0: task 69)) fetch wait time total (min, med, max (stageId: taskId)) 0 ms (0 ms, 0 ms, 0 ms (stage 24.0: task 69)) remote bytes read total (min, med, max (stageId: taskId)) 0.0 B (0.0 B, 0.0 B, 0.0 B (stage 24.0: task 69)) merged fetch fallback count: 0 local blocks read: 0 remote merged chunks fetched: 0 remote blocks read: 0 data size total (min, med, max (stageId: taskId)) 0.0 B (0.0 B, 0.0 B, 0.0 B (stage 24.0: task 69)) local merged bytes read total (min, med, max (stageId: taskId)) 0.0 B (0.0 B, 0.0 B, 0.0 B (stage 24.0: task 69)) number of partitions: 0 remote reqs duration total (min, med, max (stageId: taskId)) 0 ms (0 ms, 0 ms, 0 ms (stage 24.0: task 69)) remote bytes read to disk total (min, med, max (stageId: taskId)) 0.0 B (0.0 B, 0.0 B, 0.0 B (stage 24.0: task 69)) shuffle bytes written total (min, med, max (stageId: taskId)) 0.0 B (0.0 B, 0.0 B, 0.0 B (stage 24.0: task 69))"]; subgraph cluster12 { isCluster="true"; label="WholeStageCodegen (1)\n \nduration: total (min, med, max (stageId: taskId))\n0 ms (0 ms, 0 ms, 0 ms (stage 24.0: task 69))"; 13 [labelType="html" label=" Project "]; 14 [labelType="html" label=" Project "]; } 15 [labelType="html" label="InMemoryTableScan number of output rows: 0"]; 16 [labelType="html" label="Scan csv number of output rows: 0"]; 1->0; 2->1; 3->2; 4->3; 6->4; 7->6; 9->7; 10->9; 11->10; 13->11; 14->13; 15->14; 16->15; }

AdaptiveSparkPlan isFinalPlan=true

Execute InsertIntoHadoopFsRelationCommand s3a://rzvde-g10-chepusov-vladislav/raw_invalid/citibike_data/202508, false, CSV, [header=true, path=s3a://rzvde-g10-chepusov-vladislav/raw_invalid/citibike_data/202508], Overwrite, [ride_id, rideable_type, started_at, ended_at, start_station_name, start_station_id, end_station_name, end_station_id, start_lat, start_lng, end_lat, end_lng, member_casual, is_ride_id_valid, is_duration_valid, is_station_valid, _source_file, _processed_dttm, year, month, _start_station_ride_num]

WriteFiles

InMemoryTableScan [ride_id#26, rideable_type#27, started_at#28, ended_at#29, start_station_name#30, start_station_id#31, end_station_name#32, end_station_id#33, start_lat#34, start_lng#35, end_lat#36, end_lng#37, member_casual#38, is_ride_id_valid#474, is_duration_valid#475, is_station_valid#476, _source_file#493, _processed_dttm#494, year#513, month#514, _start_station_ride_num#536]

AdaptiveSparkPlan isFinalPlan=true

Filter ((NOT is_ride_id_valid#474 OR NOT is_duration_valid#475) OR NOT is_station_valid#476)

WholeStageCodegen (3)

Window [row_number() windowspecdefinition(start_station_id#31, started_at#28 ASC NULLS FIRST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS _start_station_ride_num#536], [start_station_id#31], [started_at#28 ASC NULLS FIRST]

Sort [start_station_id#31 ASC NULLS FIRST, started_at#28 ASC NULLS FIRST], false, 0

WholeStageCodegen (2)

AQEShuffleRead coalesced

Exchange hashpartitioning(start_station_id#31, 200), ENSURE_REQUIREMENTS, [plan_id=329]

Project [ride_id#26, rideable_type#27, started_at#28, ended_at#29, start_station_name#30, start_station_id#31, end_station_name#32, end_station_id#33, start_lat#34, start_lng#35, end_lat#36, end_lng#37, member_casual#38, is_ride_id_valid#474, is_duration_valid#475, is_station_valid#476, _source_file#493, 2026-08-18 07:26:21.350521 AS _processed_dttm#494, year(cast(started_at#28 as date)) AS year#513, month(cast(started_at#28 as date)) AS month#514]

Project [ride_id#26, rideable_type#27, started_at#28, ended_at#29, start_station_name#30, start_station_id#31, end_station_name#32, end_station_id#33, start_lat#34, start_lng#35, end_lat#36, end_lng#37, member_casual#38, (isnotnull(ride_id#26) AND NOT (trim(ride_id#26, None) = )) AS is_ride_id_valid#474, coalesce((ended_at#29 > started_at#28), false) AS is_duration_valid#475, coalesce(NOT (end_station_id#33 = start_station_id#31), false) AS is_station_valid#476, input_file_name() AS _source_file#493]

WholeStageCodegen (1)

InMemoryTableScan [end_lat#36, end_lng#37, end_station_id#33, end_station_name#32, ended_at#29, member_casual#38, ride_id#26, rideable_type#27, start_lat#34, start_lng#35, start_station_id#31, start_station_name#30, started_at#28]

FileScan csv [ride_id#0,rideable_type#1,started_at#2,ended_at#3,start_station_name#4,start_station_id#5,end_station_name#6,end_station_id#7,start_lat#8,start_lng#9,end_lat#10,end_lng#11,member_casual#12] Batched: false, DataFilters: [], Format: CSV, Location: InMemoryFileIndex(6 paths)[s3a://rzvde-g10-chepusov-vladislav/raw/citibike_data/202508/202508-cit..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<ride_id:string,rideable_type:string,started_at:timestamp,ended_at:timestamp,start_station_...

Details

== Physical Plan ==
AdaptiveSparkPlan (18)
+- == Final Plan ==
   Execute InsertIntoHadoopFsRelationCommand (15)
   +- WriteFiles (14)
      +- TableCacheQueryStage (13), Statistics(sizeInBytes=23.3 MiB, rowCount=1.40E+5)
         +- InMemoryTableScan (1)
               +- InMemoryRelation (2)
                     +- AdaptiveSparkPlan (12)
                        +- Filter (11)
                           +- Window (10)
                              +- Sort (9)
                                 +- Exchange (8)
                                    +- Project (7)
                                       +- Project (6)
                                          +- InMemoryTableScan (3)
                                                +- InMemoryRelation (4)
                                                      +- Scan csv  (5)
+- == Initial Plan ==
   Execute InsertIntoHadoopFsRelationCommand (17)
   +- WriteFiles (16)
      +- InMemoryTableScan (1)
            +- InMemoryRelation (2)
                  +- AdaptiveSparkPlan (12)
                     +- Filter (11)
                        +- Window (10)
                           +- Sort (9)
                              +- Exchange (8)
                                 +- Project (7)
                                    +- Project (6)
                                       +- InMemoryTableScan (3)
                                             +- InMemoryRelation (4)
                                                   +- Scan csv  (5)


(1) InMemoryTableScan
Output [21]: [ride_id#26, rideable_type#27, started_at#28, ended_at#29, start_station_name#30, start_station_id#31, end_station_name#32, end_station_id#33, start_lat#34, start_lng#35, end_lat#36, end_lng#37, member_casual#38, is_ride_id_valid#474, is_duration_valid#475, is_station_valid#476, _source_file#493, _processed_dttm#494, year#513, month#514, _start_station_ride_num#536]
Arguments: [ride_id#26, rideable_type#27, started_at#28, ended_at#29, start_station_name#30, start_station_id#31, end_station_name#32, end_station_id#33, start_lat#34, start_lng#35, end_lat#36, end_lng#37, member_casual#38, is_ride_id_valid#474, is_duration_valid#475, is_station_valid#476, _source_file#493, _processed_dttm#494, year#513, month#514, _start_station_ride_num#536]

(2) InMemoryRelation
Arguments: [ride_id#26, rideable_type#27, started_at#28, ended_at#29, start_station_name#30, start_station_id#31, end_station_name#32, end_station_id#33, start_lat#34, start_lng#35, end_lat#36, end_lng#37, member_casual#38, is_ride_id_valid#474, is_duration_valid#475, is_station_valid#476, _source_file#493, _processed_dttm#494, year#513, month#514, _start_station_ride_num#536], CachedRDDBuilder(org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer@3908c132,StorageLevel(disk, memory, deserialized, 1 replicas),AdaptiveSparkPlan isFinalPlan=true
+- == Final Plan ==
   *(3) Filter ((NOT is_ride_id_valid#474 OR NOT is_duration_valid#475) OR NOT is_station_valid#476)
   +- Window [row_number() windowspecdefinition(start_station_id#31, started_at#28 ASC NULLS FIRST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS _start_station_ride_num#536], [start_station_id#31], [started_at#28 ASC NULLS FIRST]
      +- *(2) Sort [start_station_id#31 ASC NULLS FIRST, started_at#28 ASC NULLS FIRST], false, 0
         +- AQEShuffleRead coalesced
            +- ShuffleQueryStage 1
               +- Exchange hashpartitioning(start_station_id#31, 200), ENSURE_REQUIREMENTS, [plan_id=329]
                  +- *(1) Project [ride_id#26, rideable_type#27, started_at#28, ended_at#29, start_station_name#30, start_station_id#31, end_station_name#32, end_station_id#33, start_lat#34, start_lng#35, end_lat#36, end_lng#37, member_casual#38, is_ride_id_valid#474, is_duration_valid#475, is_station_valid#476, _source_file#493, 2026-08-18 07:26:21.350521 AS _processed_dttm#494, year(cast(started_at#28 as date)) AS year#513, month(cast(started_at#28 as date)) AS month#514]
                     +- *(1) Project [ride_id#26, rideable_type#27, started_at#28, ended_at#29, start_station_name#30, start_station_id#31, end_station_name#32, end_station_id#33, start_lat#34, start_lng#35, end_lat#36, end_lng#37, member_casual#38, (isnotnull(ride_id#26) AND NOT (trim(ride_id#26, None) = )) AS is_ride_id_valid#474, coalesce((ended_at#29 > started_at#28), false) AS is_duration_valid#475, coalesce(NOT (end_station_id#33 = start_station_id#31), false) AS is_station_valid#476, input_file_name() AS _source_file#493]
                        +- TableCacheQueryStage 0
                           +- InMemoryTableScan [end_lat#36, end_lng#37, end_station_id#33, end_station_name#32, ended_at#29, member_casual#38, ride_id#26, rideable_type#27, start_lat#34, start_lng#35, start_station_id#31, start_station_name#30, started_at#28]
                                 +- InMemoryRelation [ride_id#26, rideable_type#27, started_at#28, ended_at#29, start_station_name#30, start_station_id#31, end_station_name#32, end_station_id#33, start_lat#34, start_lng#35, end_lat#36, end_lng#37, member_casual#38], StorageLevel(disk, memory, deserialized, 1 replicas)
                                       +- FileScan csv [ride_id#0,rideable_type#1,started_at#2,ended_at#3,start_station_name#4,start_station_id#5,end_station_name#6,end_station_id#7,start_lat#8,start_lng#9,end_lat#10,end_lng#11,member_casual#12] Batched: false, DataFilters: [], Format: CSV, Location: InMemoryFileIndex(6 paths)[s3a://rzvde-g10-chepusov-vladislav/raw/citibike_data/202508/202508-cit..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<ride_id:string,rideable_type:string,started_at:timestamp,ended_at:timestamp,start_station_...
+- == Initial Plan ==
   Filter ((NOT is_ride_id_valid#474 OR NOT is_duration_valid#475) OR NOT is_station_valid#476)
   +- Window [row_number() windowspecdefinition(start_station_id#31, started_at#28 ASC NULLS FIRST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS _start_station_ride_num#536], [start_station_id#31], [started_at#28 ASC NULLS FIRST]
      +- Sort [start_station_id#31 ASC NULLS FIRST, started_at#28 ASC NULLS FIRST], false, 0
         +- Exchange hashpartitioning(start_station_id#31, 200), ENSURE_REQUIREMENTS, [plan_id=91]
            +- Project [ride_id#26, rideable_type#27, started_at#28, ended_at#29, start_station_name#30, start_station_id#31, end_station_name#32, end_station_id#33, start_lat#34, start_lng#35, end_lat#36, end_lng#37, member_casual#38, is_ride_id_valid#474, is_duration_valid#475, is_station_valid#476, _source_file#493, 2026-08-18 07:26:21.350521 AS _processed_dttm#494, year(cast(started_at#28 as date)) AS year#513, month(cast(started_at#28 as date)) AS month#514]
               +- Project [ride_id#26, rideable_type#27, started_at#28, ended_at#29, start_station_name#30, start_station_id#31, end_station_name#32, end_station_id#33, start_lat#34, start_lng#35, end_lat#36, end_lng#37, member_casual#38, (isnotnull(ride_id#26) AND NOT (trim(ride_id#26, None) = )) AS is_ride_id_valid#474, coalesce((ended_at#29 > started_at#28), false) AS is_duration_valid#475, coalesce(NOT (end_station_id#33 = start_station_id#31), false) AS is_station_valid#476, input_file_name() AS _source_file#493]
                  +- InMemoryTableScan [end_lat#36, end_lng#37, end_station_id#33, end_station_name#32, ended_at#29, member_casual#38, ride_id#26, rideable_type#27, start_lat#34, start_lng#35, start_station_id#31, start_station_name#30, started_at#28]
                        +- InMemoryRelation [ride_id#26, rideable_type#27, started_at#28, ended_at#29, start_station_name#30, start_station_id#31, end_station_name#32, end_station_id#33, start_lat#34, start_lng#35, end_lat#36, end_lng#37, member_casual#38], StorageLevel(disk, memory, deserialized, 1 replicas)
                              +- FileScan csv [ride_id#0,rideable_type#1,started_at#2,ended_at#3,start_station_name#4,start_station_id#5,end_station_name#6,end_station_id#7,start_lat#8,start_lng#9,end_lat#10,end_lng#11,member_casual#12] Batched: false, DataFilters: [], Format: CSV, Location: InMemoryFileIndex(6 paths)[s3a://rzvde-g10-chepusov-vladislav/raw/citibike_data/202508/202508-cit..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<ride_id:string,rideable_type:string,started_at:timestamp,ended_at:timestamp,start_station_...
,None)

(3) InMemoryTableScan
Output [13]: [end_lat#36, end_lng#37, end_station_id#33, end_station_name#32, ended_at#29, member_casual#38, ride_id#26, rideable_type#27, start_lat#34, start_lng#35, start_station_id#31, start_station_name#30, started_at#28]
Arguments: [end_lat#36, end_lng#37, end_station_id#33, end_station_name#32, ended_at#29, member_casual#38, ride_id#26, rideable_type#27, start_lat#34, start_lng#35, start_station_id#31, start_station_name#30, started_at#28]

(4) InMemoryRelation
Arguments: [ride_id#26, rideable_type#27, started_at#28, ended_at#29, start_station_name#30, start_station_id#31, end_station_name#32, end_station_id#33, start_lat#34, start_lng#35, end_lat#36, end_lng#37, member_casual#38], CachedRDDBuilder(org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer@3908c132,StorageLevel(disk, memory, deserialized, 1 replicas),FileScan csv [ride_id#0,rideable_type#1,started_at#2,ended_at#3,start_station_name#4,start_station_id#5,end_station_name#6,end_station_id#7,start_lat#8,start_lng#9,end_lat#10,end_lng#11,member_casual#12] Batched: false, DataFilters: [], Format: CSV, Location: InMemoryFileIndex(6 paths)[s3a://rzvde-g10-chepusov-vladislav/raw/citibike_data/202508/202508-cit..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<ride_id:string,rideable_type:string,started_at:timestamp,ended_at:timestamp,start_station_...
,None)

(5) Scan csv 
Output [13]: [ride_id#0, rideable_type#1, started_at#2, ended_at#3, start_station_name#4, start_station_id#5, end_station_name#6, end_station_id#7, start_lat#8, start_lng#9, end_lat#10, end_lng#11, member_casual#12]
Batched: false
Location: InMemoryFileIndex [s3a://rzvde-g10-chepusov-vladislav/raw/citibike_data/202508/202508-citibike-tripdata_1.csv, ... 5 entries]
ReadSchema: struct<ride_id:string,rideable_type:string,started_at:timestamp,ended_at:timestamp,start_station_name:string,start_station_id:string,end_station_name:string,end_station_id:string,start_lat:float,start_lng:float,end_lat:float,end_lng:float,member_casual:string>

(6) Project
Output [17]: [ride_id#26, rideable_type#27, started_at#28, ended_at#29, start_station_name#30, start_station_id#31, end_station_name#32, end_station_id#33, start_lat#34, start_lng#35, end_lat#36, end_lng#37, member_casual#38, (isnotnull(ride_id#26) AND NOT (trim(ride_id#26, None) = )) AS is_ride_id_valid#474, coalesce((ended_at#29 > started_at#28), false) AS is_duration_valid#475, coalesce(NOT (end_station_id#33 = start_station_id#31), false) AS is_station_valid#476, input_file_name() AS _source_file#493]
Input [13]: [end_lat#36, end_lng#37, end_station_id#33, end_station_name#32, ended_at#29, member_casual#38, ride_id#26, rideable_type#27, start_lat#34, start_lng#35, start_station_id#31, start_station_name#30, started_at#28]

(7) Project
Output [20]: [ride_id#26, rideable_type#27, started_at#28, ended_at#29, start_station_name#30, start_station_id#31, end_station_name#32, end_station_id#33, start_lat#34, start_lng#35, end_lat#36, end_lng#37, member_casual#38, is_ride_id_valid#474, is_duration_valid#475, is_station_valid#476, _source_file#493, 2026-08-18 07:26:21.350521 AS _processed_dttm#494, year(cast(started_at#28 as date)) AS year#513, month(cast(started_at#28 as date)) AS month#514]
Input [17]: [ride_id#26, rideable_type#27, started_at#28, ended_at#29, start_station_name#30, start_station_id#31, end_station_name#32, end_station_id#33, start_lat#34, start_lng#35, end_lat#36, end_lng#37, member_casual#38, is_ride_id_valid#474, is_duration_valid#475, is_station_valid#476, _source_file#493]

(8) Exchange
Input [20]: [ride_id#26, rideable_type#27, started_at#28, ended_at#29, start_station_name#30, start_station_id#31, end_station_name#32, end_station_id#33, start_lat#34, start_lng#35, end_lat#36, end_lng#37, member_casual#38, is_ride_id_valid#474, is_duration_valid#475, is_station_valid#476, _source_file#493, _processed_dttm#494, year#513, month#514]
Arguments: hashpartitioning(start_station_id#31, 200), ENSURE_REQUIREMENTS, [plan_id=400]

(9) Sort
Input [20]: [ride_id#26, rideable_type#27, started_at#28, ended_at#29, start_station_name#30, start_station_id#31, end_station_name#32, end_station_id#33, start_lat#34, start_lng#35, end_lat#36, end_lng#37, member_casual#38, is_ride_id_valid#474, is_duration_valid#475, is_station_valid#476, _source_file#493, _processed_dttm#494, year#513, month#514]
Arguments: [start_station_id#31 ASC NULLS FIRST, started_at#28 ASC NULLS FIRST], false, 0

(10) Window
Input [20]: [ride_id#26, rideable_type#27, started_at#28, ended_at#29, start_station_name#30, start_station_id#31, end_station_name#32, end_station_id#33, start_lat#34, start_lng#35, end_lat#36, end_lng#37, member_casual#38, is_ride_id_valid#474, is_duration_valid#475, is_station_valid#476, _source_file#493, _processed_dttm#494, year#513, month#514]
Arguments: [row_number() windowspecdefinition(start_station_id#31, started_at#28 ASC NULLS FIRST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS _start_station_ride_num#536], [start_station_id#31], [started_at#28 ASC NULLS FIRST]

(11) Filter
Input [21]: [ride_id#26, rideable_type#27, started_at#28, ended_at#29, start_station_name#30, start_station_id#31, end_station_name#32, end_station_id#33, start_lat#34, start_lng#35, end_lat#36, end_lng#37, member_casual#38, is_ride_id_valid#474, is_duration_valid#475, is_station_valid#476, _source_file#493, _processed_dttm#494, year#513, month#514, _start_station_ride_num#536]
Condition : ((NOT is_ride_id_valid#474 OR NOT is_duration_valid#475) OR NOT is_station_valid#476)

(12) AdaptiveSparkPlan
Output [21]: [ride_id#26, rideable_type#27, started_at#28, ended_at#29, start_station_name#30, start_station_id#31, end_station_name#32, end_station_id#33, start_lat#34, start_lng#35, end_lat#36, end_lng#37, member_casual#38, is_ride_id_valid#474, is_duration_valid#475, is_station_valid#476, _source_file#493, _processed_dttm#494, year#513, month#514, _start_station_ride_num#536]
Arguments: isFinalPlan=false

(13) TableCacheQueryStage
Output [21]: [ride_id#26, rideable_type#27, started_at#28, ended_at#29, start_station_name#30, start_station_id#31, end_station_name#32, end_station_id#33, start_lat#34, start_lng#35, end_lat#36, end_lng#37, member_casual#38, is_ride_id_valid#474, is_duration_valid#475, is_station_valid#476, _source_file#493, _processed_dttm#494, year#513, month#514, _start_station_ride_num#536]
Arguments: 0

(14) WriteFiles
Input [21]: [ride_id#26, rideable_type#27, started_at#28, ended_at#29, start_station_name#30, start_station_id#31, end_station_name#32, end_station_id#33, start_lat#34, start_lng#35, end_lat#36, end_lng#37, member_casual#38, is_ride_id_valid#474, is_duration_valid#475, is_station_valid#476, _source_file#493, _processed_dttm#494, year#513, month#514, _start_station_ride_num#536]

(15) Execute InsertIntoHadoopFsRelationCommand
Input: []
Arguments: s3a://rzvde-g10-chepusov-vladislav/raw_invalid/citibike_data/202508, false, CSV, [header=true, path=s3a://rzvde-g10-chepusov-vladislav/raw_invalid/citibike_data/202508], Overwrite, [ride_id, rideable_type, started_at, ended_at, start_station_name, start_station_id, end_station_name, end_station_id, start_lat, start_lng, end_lat, end_lng, member_casual, is_ride_id_valid, is_duration_valid, is_station_valid, _source_file, _processed_dttm, year, month, _start_station_ride_num]

(16) WriteFiles
Input [21]: [ride_id#26, rideable_type#27, started_at#28, ended_at#29, start_station_name#30, start_station_id#31, end_station_name#32, end_station_id#33, start_lat#34, start_lng#35, end_lat#36, end_lng#37, member_casual#38, is_ride_id_valid#474, is_duration_valid#475, is_station_valid#476, _source_file#493, _processed_dttm#494, year#513, month#514, _start_station_ride_num#536]

(17) Execute InsertIntoHadoopFsRelationCommand
Input: []
Arguments: s3a://rzvde-g10-chepusov-vladislav/raw_invalid/citibike_data/202508, false, CSV, [header=true, path=s3a://rzvde-g10-chepusov-vladislav/raw_invalid/citibike_data/202508], Overwrite, [ride_id, rideable_type, started_at, ended_at, start_station_name, start_station_id, end_station_name, end_station_id, start_lat, start_lng, end_lat, end_lng, member_casual, is_ride_id_valid, is_duration_valid, is_station_valid, _source_file, _processed_dttm, year, month, _start_station_ride_num]

(18) AdaptiveSparkPlan
Output: []
Arguments: isFinalPlan=true