SP1 - Details for Query 2

Submitted Time: 2026/08/26 13:58:31
Duration: 17 s
Succeeded Jobs: 2 3

Show the Stage ID and Task ID that corresponds to the max metric

digraph G { 0 [labelType="html" label=" AdaptiveSparkPlan "]; 1 [labelType="html" label="Execute InsertIntoHadoopFsRelationCommand task commit time: 14 ms number of written files: 1 job commit time: 21 ms number of output rows: 2,401,930 number of dynamic part: 0 written output: 91.4 MiB"]; 2 [labelType="html" label=" WriteFiles "]; 3 [labelType="html" label="Window spill size: 0.0 B"]; subgraph cluster4 { isCluster="true"; label="WholeStageCodegen (3)\n \nduration: 8.0 s"; 5 [labelType="html" label=" Project "]; 6 [labelType="html" label="Filter number of output rows: 2,401,930"]; } 7 [labelType="html" label="Window spill size: 0.0 B"]; subgraph cluster8 { isCluster="true"; label="WholeStageCodegen (2)\n \nduration: 3.2 s"; 9 [labelType="html" label="Sort sort time: 208 ms peak memory: 576.0 MiB spill size: 0.0 B"]; } 10 [labelType="html" label="AQEShuffleRead number of partitions: 1 partition data size: 89.4 MiB number of coalesced partitions: 1"]; 11 [labelType="html" label="Exchange shuffle records written: 2,401,931 local merged chunks fetched: 0 shuffle write time total (min, med, max (stageId: taskId)) 271 ms (22 ms, 27 ms, 31 ms (stage 2.0: task 17)) remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 2,401,931 local bytes read: 86.0 MiB fetch wait time: 0 ms remote bytes read: 0.0 B merged fetch fallback count: 0 local blocks read: 10 remote merged chunks fetched: 0 remote blocks read: 0 data size total (min, med, max (stageId: taskId)) 500.2 MiB (45.5 MiB, 50.5 MiB, 50.6 MiB (stage 2.0: task 11)) local merged bytes read: 0.0 B number of partitions: 10 remote reqs duration: 0 ms remote bytes read to disk: 0.0 B shuffle bytes written total (min, med, max (stageId: taskId)) 86.0 MiB (7.6 MiB, 9.0 MiB, 9.1 MiB (stage 2.0: task 19))"]; subgraph cluster12 { isCluster="true"; label="WholeStageCodegen (1)\n \nduration: total (min, med, max (stageId: taskId))\n13.9 s (1.2 s, 1.4 s, 1.5 s (stage 2.0: task 13))"; 13 [labelType="html" label=" Project "]; } 14 [labelType="html" label="Scan csv number of output rows: 2,401,931 number of files read: 1 metadata time: 0 ms size of files read: 417.5 MiB"]; 1->0; 2->1; 3->2; 5->3; 6->5; 7->6; 9->7; 10->9; 11->10; 13->11; 14->13; }

AdaptiveSparkPlan isFinalPlan=true

Execute InsertIntoHadoopFsRelationCommand file:/data/output/cache/parquet/datablock/01_A_PRE_DECOUP_425007, false, Parquet, [path=file:///data/output/cache/parquet/datablock/01_A_PRE_DECOUP_425007], Overwrite, [COL_30863531_1D7E_428B_B583_E41A16C72ABB, COL_68EAEFC1_2326_4E45_A372_A7E0B2A1C23F, COL_17B43B85_6EF2_4511_9F9E_84EDB43F71E9, COL_EA2F05A2_A6F1_4291_910C_47CB1DD38763, COL_1C2F8BFA_DEC1_42A6_B904_489BF5DA6E09]

WriteFiles

Window [last(COL_EA2F05A2_A6F1_4291_910C_47CB1DD38763#133, true) windowspecdefinition(1, 1 DESC NULLS LAST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS COL_1C2F8BFA_DEC1_42A6_B904_489BF5DA6E09#155], [1], [1 DESC NULLS LAST]

Project [COL_30863531_1D7E_428B_B583_E41A16C72ABB#47, COL_68EAEFC1_2326_4E45_A372_A7E0B2A1C23F#57, COL_17B43B85_6EF2_4511_9F9E_84EDB43F71E9#49, str_if_with_rule((COL_17B43B85_6EF2_4511_9F9E_84EDB43F71E9#49 <=> A), str_join(array(str_if_with_rule((COL_17B43B85_6EF2_4511_9F9E_84EDB43F71E9#49 <=> A), str_complete(bigint_to_str(COL_68EAEFC1_2326_4E45_A372_A7E0B2A1C23F#57), 0, 5, LEFT), ), str_if_with_rule((COL_17B43B85_6EF2_4511_9F9E_84EDB43F71E9#49 <=> A), str_extract_middle(COL_30863531_1D7E_428B_B583_E41A16C72ABB#47, 40, 9), )), -), null) AS COL_EA2F05A2_A6F1_4291_910C_47CB1DD38763#133]

Filter (COL_68EAEFC1_2326_4E45_A372_A7E0B2A1C23F#57 > 1)

WholeStageCodegen (3)

Window [row_number() windowspecdefinition(1, 1 DESC NULLS LAST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS COL_68EAEFC1_2326_4E45_A372_A7E0B2A1C23F#57], [1], [1 DESC NULLS LAST]

Sort [1 ASC NULLS FIRST, 1 DESC NULLS LAST], false, 0

WholeStageCodegen (2)

AQEShuffleRead coalesced

Exchange hashpartitioning(1, 10), ENSURE_REQUIREMENTS, [plan_id=159]

Project [_c0#30 AS COL_30863531_1D7E_428B_B583_E41A16C72ABB#47, str_extract(_c0#30, LEFT, 1) AS COL_17B43B85_6EF2_4511_9F9E_84EDB43F71E9#49]

WholeStageCodegen (1)

FileScan csv [_c0#30] Batched: false, DataFilters: [], Format: CSV, Location: InMemoryFileIndex(1 paths)[file:/data/input/depot/csv/execution/RSF_SOURCE_LOCA_424973/current/la..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<_c0:string>

Details

== Physical Plan ==
AdaptiveSparkPlan (22)
+- == Final Plan ==
   Execute InsertIntoHadoopFsRelationCommand (12)
   +- WriteFiles (11)
      +- Window (10)
         +- * Project (9)
            +- * Filter (8)
               +- Window (7)
                  +- * Sort (6)
                     +- AQEShuffleRead (5)
                        +- ShuffleQueryStage (4), Statistics(sizeInBytes=500.2 MiB, rowCount=2.40E+6)
                           +- Exchange (3)
                              +- * Project (2)
                                 +- Scan csv  (1)
+- == Initial Plan ==
   Execute InsertIntoHadoopFsRelationCommand (21)
   +- WriteFiles (20)
      +- Window (19)
         +- Project (18)
            +- Filter (17)
               +- Window (16)
                  +- Sort (15)
                     +- Exchange (14)
                        +- Project (13)
                           +- Scan csv  (1)


(1) Scan csv 
Output [1]: [_c0#30]
Batched: false
Location: InMemoryFileIndex [file:/data/input/depot/csv/execution/RSF_SOURCE_LOCA_424973/current/lamrsf_corrige_Ayming_MCO_2026-07-23_011.txt]
ReadSchema: struct<_c0:string>

(2) Project [codegen id : 1]
Output [2]: [_c0#30 AS COL_30863531_1D7E_428B_B583_E41A16C72ABB#47, str_extract(_c0#30, LEFT, 1) AS COL_17B43B85_6EF2_4511_9F9E_84EDB43F71E9#49]
Input [1]: [_c0#30]

(3) Exchange
Input [2]: [COL_30863531_1D7E_428B_B583_E41A16C72ABB#47, COL_17B43B85_6EF2_4511_9F9E_84EDB43F71E9#49]
Arguments: hashpartitioning(1, 10), ENSURE_REQUIREMENTS, [plan_id=159]

(4) ShuffleQueryStage
Output [2]: [COL_30863531_1D7E_428B_B583_E41A16C72ABB#47, COL_17B43B85_6EF2_4511_9F9E_84EDB43F71E9#49]
Arguments: 0

(5) AQEShuffleRead
Input [2]: [COL_30863531_1D7E_428B_B583_E41A16C72ABB#47, COL_17B43B85_6EF2_4511_9F9E_84EDB43F71E9#49]
Arguments: coalesced

(6) Sort [codegen id : 2]
Input [2]: [COL_30863531_1D7E_428B_B583_E41A16C72ABB#47, COL_17B43B85_6EF2_4511_9F9E_84EDB43F71E9#49]
Arguments: [1 ASC NULLS FIRST, 1 DESC NULLS LAST], false, 0

(7) Window
Input [2]: [COL_30863531_1D7E_428B_B583_E41A16C72ABB#47, COL_17B43B85_6EF2_4511_9F9E_84EDB43F71E9#49]
Arguments: [row_number() windowspecdefinition(1, 1 DESC NULLS LAST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS COL_68EAEFC1_2326_4E45_A372_A7E0B2A1C23F#57], [1], [1 DESC NULLS LAST]

(8) Filter [codegen id : 3]
Input [3]: [COL_30863531_1D7E_428B_B583_E41A16C72ABB#47, COL_17B43B85_6EF2_4511_9F9E_84EDB43F71E9#49, COL_68EAEFC1_2326_4E45_A372_A7E0B2A1C23F#57]
Condition : (COL_68EAEFC1_2326_4E45_A372_A7E0B2A1C23F#57 > 1)

(9) Project [codegen id : 3]
Output [4]: [COL_30863531_1D7E_428B_B583_E41A16C72ABB#47, COL_68EAEFC1_2326_4E45_A372_A7E0B2A1C23F#57, COL_17B43B85_6EF2_4511_9F9E_84EDB43F71E9#49, str_if_with_rule((COL_17B43B85_6EF2_4511_9F9E_84EDB43F71E9#49 <=> A), str_join(array(str_if_with_rule((COL_17B43B85_6EF2_4511_9F9E_84EDB43F71E9#49 <=> A), str_complete(bigint_to_str(COL_68EAEFC1_2326_4E45_A372_A7E0B2A1C23F#57), 0, 5, LEFT), ), str_if_with_rule((COL_17B43B85_6EF2_4511_9F9E_84EDB43F71E9#49 <=> A), str_extract_middle(COL_30863531_1D7E_428B_B583_E41A16C72ABB#47, 40, 9), )), -), null) AS COL_EA2F05A2_A6F1_4291_910C_47CB1DD38763#133]
Input [3]: [COL_30863531_1D7E_428B_B583_E41A16C72ABB#47, COL_17B43B85_6EF2_4511_9F9E_84EDB43F71E9#49, COL_68EAEFC1_2326_4E45_A372_A7E0B2A1C23F#57]

(10) Window
Input [4]: [COL_30863531_1D7E_428B_B583_E41A16C72ABB#47, COL_68EAEFC1_2326_4E45_A372_A7E0B2A1C23F#57, COL_17B43B85_6EF2_4511_9F9E_84EDB43F71E9#49, COL_EA2F05A2_A6F1_4291_910C_47CB1DD38763#133]
Arguments: [last(COL_EA2F05A2_A6F1_4291_910C_47CB1DD38763#133, true) windowspecdefinition(1, 1 DESC NULLS LAST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS COL_1C2F8BFA_DEC1_42A6_B904_489BF5DA6E09#155], [1], [1 DESC NULLS LAST]

(11) WriteFiles
Input [5]: [COL_30863531_1D7E_428B_B583_E41A16C72ABB#47, COL_68EAEFC1_2326_4E45_A372_A7E0B2A1C23F#57, COL_17B43B85_6EF2_4511_9F9E_84EDB43F71E9#49, COL_EA2F05A2_A6F1_4291_910C_47CB1DD38763#133, COL_1C2F8BFA_DEC1_42A6_B904_489BF5DA6E09#155]

(12) Execute InsertIntoHadoopFsRelationCommand
Input: []
Arguments: file:/data/output/cache/parquet/datablock/01_A_PRE_DECOUP_425007, false, Parquet, [path=file:///data/output/cache/parquet/datablock/01_A_PRE_DECOUP_425007], Overwrite, [COL_30863531_1D7E_428B_B583_E41A16C72ABB, COL_68EAEFC1_2326_4E45_A372_A7E0B2A1C23F, COL_17B43B85_6EF2_4511_9F9E_84EDB43F71E9, COL_EA2F05A2_A6F1_4291_910C_47CB1DD38763, COL_1C2F8BFA_DEC1_42A6_B904_489BF5DA6E09]

(13) Project
Output [2]: [_c0#30 AS COL_30863531_1D7E_428B_B583_E41A16C72ABB#47, str_extract(_c0#30, LEFT, 1) AS COL_17B43B85_6EF2_4511_9F9E_84EDB43F71E9#49]
Input [1]: [_c0#30]

(14) Exchange
Input [2]: [COL_30863531_1D7E_428B_B583_E41A16C72ABB#47, COL_17B43B85_6EF2_4511_9F9E_84EDB43F71E9#49]
Arguments: hashpartitioning(1, 10), ENSURE_REQUIREMENTS, [plan_id=144]

(15) Sort
Input [2]: [COL_30863531_1D7E_428B_B583_E41A16C72ABB#47, COL_17B43B85_6EF2_4511_9F9E_84EDB43F71E9#49]
Arguments: [1 ASC NULLS FIRST, 1 DESC NULLS LAST], false, 0

(16) Window
Input [2]: [COL_30863531_1D7E_428B_B583_E41A16C72ABB#47, COL_17B43B85_6EF2_4511_9F9E_84EDB43F71E9#49]
Arguments: [row_number() windowspecdefinition(1, 1 DESC NULLS LAST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS COL_68EAEFC1_2326_4E45_A372_A7E0B2A1C23F#57], [1], [1 DESC NULLS LAST]

(17) Filter
Input [3]: [COL_30863531_1D7E_428B_B583_E41A16C72ABB#47, COL_17B43B85_6EF2_4511_9F9E_84EDB43F71E9#49, COL_68EAEFC1_2326_4E45_A372_A7E0B2A1C23F#57]
Condition : (COL_68EAEFC1_2326_4E45_A372_A7E0B2A1C23F#57 > 1)

(18) Project
Output [4]: [COL_30863531_1D7E_428B_B583_E41A16C72ABB#47, COL_68EAEFC1_2326_4E45_A372_A7E0B2A1C23F#57, COL_17B43B85_6EF2_4511_9F9E_84EDB43F71E9#49, str_if_with_rule((COL_17B43B85_6EF2_4511_9F9E_84EDB43F71E9#49 <=> A), str_join(array(str_if_with_rule((COL_17B43B85_6EF2_4511_9F9E_84EDB43F71E9#49 <=> A), str_complete(bigint_to_str(COL_68EAEFC1_2326_4E45_A372_A7E0B2A1C23F#57), 0, 5, LEFT), ), str_if_with_rule((COL_17B43B85_6EF2_4511_9F9E_84EDB43F71E9#49 <=> A), str_extract_middle(COL_30863531_1D7E_428B_B583_E41A16C72ABB#47, 40, 9), )), -), null) AS COL_EA2F05A2_A6F1_4291_910C_47CB1DD38763#133]
Input [3]: [COL_30863531_1D7E_428B_B583_E41A16C72ABB#47, COL_17B43B85_6EF2_4511_9F9E_84EDB43F71E9#49, COL_68EAEFC1_2326_4E45_A372_A7E0B2A1C23F#57]

(19) Window
Input [4]: [COL_30863531_1D7E_428B_B583_E41A16C72ABB#47, COL_68EAEFC1_2326_4E45_A372_A7E0B2A1C23F#57, COL_17B43B85_6EF2_4511_9F9E_84EDB43F71E9#49, COL_EA2F05A2_A6F1_4291_910C_47CB1DD38763#133]
Arguments: [last(COL_EA2F05A2_A6F1_4291_910C_47CB1DD38763#133, true) windowspecdefinition(1, 1 DESC NULLS LAST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS COL_1C2F8BFA_DEC1_42A6_B904_489BF5DA6E09#155], [1], [1 DESC NULLS LAST]

(20) WriteFiles
Input [5]: [COL_30863531_1D7E_428B_B583_E41A16C72ABB#47, COL_68EAEFC1_2326_4E45_A372_A7E0B2A1C23F#57, COL_17B43B85_6EF2_4511_9F9E_84EDB43F71E9#49, COL_EA2F05A2_A6F1_4291_910C_47CB1DD38763#133, COL_1C2F8BFA_DEC1_42A6_B904_489BF5DA6E09#155]

(21) Execute InsertIntoHadoopFsRelationCommand
Input: []
Arguments: file:/data/output/cache/parquet/datablock/01_A_PRE_DECOUP_425007, false, Parquet, [path=file:///data/output/cache/parquet/datablock/01_A_PRE_DECOUP_425007], Overwrite, [COL_30863531_1D7E_428B_B583_E41A16C72ABB, COL_68EAEFC1_2326_4E45_A372_A7E0B2A1C23F, COL_17B43B85_6EF2_4511_9F9E_84EDB43F71E9, COL_EA2F05A2_A6F1_4291_910C_47CB1DD38763, COL_1C2F8BFA_DEC1_42A6_B904_489BF5DA6E09]

(22) AdaptiveSparkPlan
Output: []
Arguments: isFinalPlan=true

SQL / DataFrame Properties

Name	Value
spark.sql.optimizer.nestedPredicatePushdown.supportedFileSources	parquet,orc,geoparquet