SP1 - Details for Query 190

Submitted Time: 2026/02/20 08:08:03
Duration: 0.6 s
Succeeded Jobs: 289 300

Show the Stage ID and Task ID that corresponds to the max metric

digraph G { 0 [labelType="html" label=" AdaptiveSparkPlan "]; 1 [labelType="html" label=" TakeOrderedAndProject "]; subgraph cluster2 { isCluster="true"; label="WholeStageCodegen (2)\n \nduration: 16 ms"; 3 [labelType="html" label="HashAggregate spill size: 0.0 B time in aggregation build: 0 ms peak memory: 16.3 MiB number of output rows: 6 number of sort fallback tasks: 0 avg hash probes per key: 1"]; } 4 [labelType="html" label="AQEShuffleRead number of partitions: 1 partition data size: 661.0 B number of coalesced partitions: 1"]; 5 [labelType="html" label="Exchange shuffle records written: 6 local merged chunks fetched: 0 shuffle write time total (min, med, max (stageId: taskId)) 9 ms (1 ms, 1 ms, 6 ms (stage 418.0: task 543)) remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 6 local bytes read: 635.0 B fetch wait time: 0 ms remote bytes read: 0.0 B merged fetch fallback count: 0 local blocks read: 3 remote merged chunks fetched: 0 remote blocks read: 0 data size total (min, med, max (stageId: taskId)) 408.0 B (120.0 B, 136.0 B, 152.0 B (stage 418.0: task 541)) local merged bytes read: 0.0 B number of partitions: 10 remote reqs duration: 0 ms remote bytes read to disk: 0.0 B shuffle bytes written total (min, med, max (stageId: taskId)) 635.0 B (197.0 B, 214.0 B, 224.0 B (stage 418.0: task 541))"]; subgraph cluster6 { isCluster="true"; label="WholeStageCodegen (1)\n \nduration: total (min, med, max (stageId: taskId))\n292 ms (96 ms, 97 ms, 99 ms (stage 418.0: task 541))"; 7 [labelType="html" label="HashAggregate spill size: 0.0 B time in aggregation build total (min, med, max (stageId: taskId)) 25 ms (8 ms, 8 ms, 9 ms (stage 418.0: task 541)) peak memory total (min, med, max (stageId: taskId)) 768.0 KiB (256.0 KiB, 256.0 KiB, 256.0 KiB (stage 418.0: task 541)) number of output rows: 6 number of sort fallback tasks: 0 avg hash probes per key: 0"]; 8 [labelType="html" label=" Project "]; 9 [labelType="html" label="ColumnarToRow number of output rows: 72 number of input batches: 3"]; } 10 [labelType="html" label="Scan parquet number of files read: 3 scan time total (min, med, max (stageId: taskId)) 18 ms (4 ms, 6 ms, 8 ms (stage 418.0: task 541)) metadata time: 0 ms size of files read: 4.5 MiB number of output rows: 72"]; 1->0; 3->1; 4->3; 5->4; 7->5; 8->7; 9->8; 10->9; }

AdaptiveSparkPlan isFinalPlan=true

TakeOrderedAndProject(limit=500000, orderBy=[COL_41AE8892_46A7_4ECE_B588_CB425C1B351C#61221 ASC NULLS FIRST,COL_8639E8DD_E645_438D_8220_3B334C61C513#61222 ASC NULLS FIRST], output=[COL_41AE8892_46A7_4ECE_B588_CB425C1B351C#61221,COL_8639E8DD_E645_438D_8220_3B334C61C513#61222,COL_479BB2B2_C798_4B3F_A3D2_AC43C8AFB16C#61233])

HashAggregate(keys=[COL_A94EBD1B_2CF2_470B_96BE_30B3BE4FD332#61195, COL_1733FA1B_DAAD_4190_86A3_F964472ADC81#61194], functions=[last(COL_D62D1D3F_A2D2_4CF3_82C7_154802876D54#61201, false)])

WholeStageCodegen (2)

AQEShuffleRead coalesced

Exchange hashpartitioning(COL_A94EBD1B_2CF2_470B_96BE_30B3BE4FD332#61195, COL_1733FA1B_DAAD_4190_86A3_F964472ADC81#61194, 10), ENSURE_REQUIREMENTS, [plan_id=6946]

HashAggregate(keys=[COL_A94EBD1B_2CF2_470B_96BE_30B3BE4FD332#61195, COL_1733FA1B_DAAD_4190_86A3_F964472ADC81#61194], functions=[partial_last(COL_D62D1D3F_A2D2_4CF3_82C7_154802876D54#61201, false)])

Project [COL_0EFAFD75_A8DE_42EF_8A2B_87C19F240FF0#61038 AS COL_1733FA1B_DAAD_4190_86A3_F964472ADC81#61194, COL_BD34C176_D5D8_497C_AC6E_1ECE0068FFA3#61039 AS COL_A94EBD1B_2CF2_470B_96BE_30B3BE4FD332#61195, COL_208CED73_0168_41C9_A2D0_B5ED3BB68CC4#61045 AS COL_D62D1D3F_A2D2_4CF3_82C7_154802876D54#61201]

ColumnarToRow

WholeStageCodegen (1)

FileScan parquet [COL_0EFAFD75_A8DE_42EF_8A2B_87C19F240FF0#61038,COL_BD34C176_D5D8_497C_AC6E_1ECE0068FFA3#61039,COL_208CED73_0168_41C9_A2D0_B5ED3BB68CC4#61045] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/data/output/cache/parquet/datablock/REPARTITION_PRO_14440], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<COL_0EFAFD75_A8DE_42EF_8A2B_87C19F240FF0:string,COL_BD34C176_D5D8_497C_AC6E_1ECE0068FFA3:s...

Details

== Physical Plan ==
AdaptiveSparkPlan (15)
+- == Final Plan ==
   TakeOrderedAndProject (9)
   +- * HashAggregate (8)
      +- AQEShuffleRead (7)
         +- ShuffleQueryStage (6), Statistics(sizeInBytes=408.0 B, rowCount=6)
            +- Exchange (5)
               +- * HashAggregate (4)
                  +- * Project (3)
                     +- * ColumnarToRow (2)
                        +- Scan parquet  (1)
+- == Initial Plan ==
   TakeOrderedAndProject (14)
   +- HashAggregate (13)
      +- Exchange (12)
         +- HashAggregate (11)
            +- Project (10)
               +- Scan parquet  (1)


(1) Scan parquet 
Output [3]: [COL_0EFAFD75_A8DE_42EF_8A2B_87C19F240FF0#61038, COL_BD34C176_D5D8_497C_AC6E_1ECE0068FFA3#61039, COL_208CED73_0168_41C9_A2D0_B5ED3BB68CC4#61045]
Batched: true
Location: InMemoryFileIndex [file:/data/output/cache/parquet/datablock/REPARTITION_PRO_14440]
ReadSchema: struct<COL_0EFAFD75_A8DE_42EF_8A2B_87C19F240FF0:string,COL_BD34C176_D5D8_497C_AC6E_1ECE0068FFA3:string,COL_208CED73_0168_41C9_A2D0_B5ED3BB68CC4:double>

(2) ColumnarToRow [codegen id : 1]
Input [3]: [COL_0EFAFD75_A8DE_42EF_8A2B_87C19F240FF0#61038, COL_BD34C176_D5D8_497C_AC6E_1ECE0068FFA3#61039, COL_208CED73_0168_41C9_A2D0_B5ED3BB68CC4#61045]

(3) Project [codegen id : 1]
Output [3]: [COL_0EFAFD75_A8DE_42EF_8A2B_87C19F240FF0#61038 AS COL_1733FA1B_DAAD_4190_86A3_F964472ADC81#61194, COL_BD34C176_D5D8_497C_AC6E_1ECE0068FFA3#61039 AS COL_A94EBD1B_2CF2_470B_96BE_30B3BE4FD332#61195, COL_208CED73_0168_41C9_A2D0_B5ED3BB68CC4#61045 AS COL_D62D1D3F_A2D2_4CF3_82C7_154802876D54#61201]
Input [3]: [COL_0EFAFD75_A8DE_42EF_8A2B_87C19F240FF0#61038, COL_BD34C176_D5D8_497C_AC6E_1ECE0068FFA3#61039, COL_208CED73_0168_41C9_A2D0_B5ED3BB68CC4#61045]

(4) HashAggregate [codegen id : 1]
Input [3]: [COL_1733FA1B_DAAD_4190_86A3_F964472ADC81#61194, COL_A94EBD1B_2CF2_470B_96BE_30B3BE4FD332#61195, COL_D62D1D3F_A2D2_4CF3_82C7_154802876D54#61201]
Keys [2]: [COL_A94EBD1B_2CF2_470B_96BE_30B3BE4FD332#61195, COL_1733FA1B_DAAD_4190_86A3_F964472ADC81#61194]
Functions [1]: [partial_last(COL_D62D1D3F_A2D2_4CF3_82C7_154802876D54#61201, false)]
Aggregate Attributes [2]: [last#61424, valueSet#61425]
Results [4]: [COL_A94EBD1B_2CF2_470B_96BE_30B3BE4FD332#61195, COL_1733FA1B_DAAD_4190_86A3_F964472ADC81#61194, last#61426, valueSet#61427]

(5) Exchange
Input [4]: [COL_A94EBD1B_2CF2_470B_96BE_30B3BE4FD332#61195, COL_1733FA1B_DAAD_4190_86A3_F964472ADC81#61194, last#61426, valueSet#61427]
Arguments: hashpartitioning(COL_A94EBD1B_2CF2_470B_96BE_30B3BE4FD332#61195, COL_1733FA1B_DAAD_4190_86A3_F964472ADC81#61194, 10), ENSURE_REQUIREMENTS, [plan_id=6946]

(6) ShuffleQueryStage
Output [4]: [COL_A94EBD1B_2CF2_470B_96BE_30B3BE4FD332#61195, COL_1733FA1B_DAAD_4190_86A3_F964472ADC81#61194, last#61426, valueSet#61427]
Arguments: 0

(7) AQEShuffleRead
Input [4]: [COL_A94EBD1B_2CF2_470B_96BE_30B3BE4FD332#61195, COL_1733FA1B_DAAD_4190_86A3_F964472ADC81#61194, last#61426, valueSet#61427]
Arguments: coalesced

(8) HashAggregate [codegen id : 2]
Input [4]: [COL_A94EBD1B_2CF2_470B_96BE_30B3BE4FD332#61195, COL_1733FA1B_DAAD_4190_86A3_F964472ADC81#61194, last#61426, valueSet#61427]
Keys [2]: [COL_A94EBD1B_2CF2_470B_96BE_30B3BE4FD332#61195, COL_1733FA1B_DAAD_4190_86A3_F964472ADC81#61194]
Functions [1]: [last(COL_D62D1D3F_A2D2_4CF3_82C7_154802876D54#61201, false)]
Aggregate Attributes [1]: [last(COL_D62D1D3F_A2D2_4CF3_82C7_154802876D54#61201)()#61232]
Results [3]: [COL_A94EBD1B_2CF2_470B_96BE_30B3BE4FD332#61195 AS COL_41AE8892_46A7_4ECE_B588_CB425C1B351C#61221, COL_1733FA1B_DAAD_4190_86A3_F964472ADC81#61194 AS COL_8639E8DD_E645_438D_8220_3B334C61C513#61222, last(COL_D62D1D3F_A2D2_4CF3_82C7_154802876D54#61201)()#61232 AS COL_479BB2B2_C798_4B3F_A3D2_AC43C8AFB16C#61233]

(9) TakeOrderedAndProject
Input [3]: [COL_41AE8892_46A7_4ECE_B588_CB425C1B351C#61221, COL_8639E8DD_E645_438D_8220_3B334C61C513#61222, COL_479BB2B2_C798_4B3F_A3D2_AC43C8AFB16C#61233]
Arguments: 500000, [COL_41AE8892_46A7_4ECE_B588_CB425C1B351C#61221 ASC NULLS FIRST, COL_8639E8DD_E645_438D_8220_3B334C61C513#61222 ASC NULLS FIRST], [COL_41AE8892_46A7_4ECE_B588_CB425C1B351C#61221, COL_8639E8DD_E645_438D_8220_3B334C61C513#61222, COL_479BB2B2_C798_4B3F_A3D2_AC43C8AFB16C#61233]

(10) Project
Output [3]: [COL_0EFAFD75_A8DE_42EF_8A2B_87C19F240FF0#61038 AS COL_1733FA1B_DAAD_4190_86A3_F964472ADC81#61194, COL_BD34C176_D5D8_497C_AC6E_1ECE0068FFA3#61039 AS COL_A94EBD1B_2CF2_470B_96BE_30B3BE4FD332#61195, COL_208CED73_0168_41C9_A2D0_B5ED3BB68CC4#61045 AS COL_D62D1D3F_A2D2_4CF3_82C7_154802876D54#61201]
Input [3]: [COL_0EFAFD75_A8DE_42EF_8A2B_87C19F240FF0#61038, COL_BD34C176_D5D8_497C_AC6E_1ECE0068FFA3#61039, COL_208CED73_0168_41C9_A2D0_B5ED3BB68CC4#61045]

(11) HashAggregate
Input [3]: [COL_1733FA1B_DAAD_4190_86A3_F964472ADC81#61194, COL_A94EBD1B_2CF2_470B_96BE_30B3BE4FD332#61195, COL_D62D1D3F_A2D2_4CF3_82C7_154802876D54#61201]
Keys [2]: [COL_A94EBD1B_2CF2_470B_96BE_30B3BE4FD332#61195, COL_1733FA1B_DAAD_4190_86A3_F964472ADC81#61194]
Functions [1]: [partial_last(COL_D62D1D3F_A2D2_4CF3_82C7_154802876D54#61201, false)]
Aggregate Attributes [2]: [last#61424, valueSet#61425]
Results [4]: [COL_A94EBD1B_2CF2_470B_96BE_30B3BE4FD332#61195, COL_1733FA1B_DAAD_4190_86A3_F964472ADC81#61194, last#61426, valueSet#61427]

(12) Exchange
Input [4]: [COL_A94EBD1B_2CF2_470B_96BE_30B3BE4FD332#61195, COL_1733FA1B_DAAD_4190_86A3_F964472ADC81#61194, last#61426, valueSet#61427]
Arguments: hashpartitioning(COL_A94EBD1B_2CF2_470B_96BE_30B3BE4FD332#61195, COL_1733FA1B_DAAD_4190_86A3_F964472ADC81#61194, 10), ENSURE_REQUIREMENTS, [plan_id=6921]

(13) HashAggregate
Input [4]: [COL_A94EBD1B_2CF2_470B_96BE_30B3BE4FD332#61195, COL_1733FA1B_DAAD_4190_86A3_F964472ADC81#61194, last#61426, valueSet#61427]
Keys [2]: [COL_A94EBD1B_2CF2_470B_96BE_30B3BE4FD332#61195, COL_1733FA1B_DAAD_4190_86A3_F964472ADC81#61194]
Functions [1]: [last(COL_D62D1D3F_A2D2_4CF3_82C7_154802876D54#61201, false)]
Aggregate Attributes [1]: [last(COL_D62D1D3F_A2D2_4CF3_82C7_154802876D54#61201)()#61232]
Results [3]: [COL_A94EBD1B_2CF2_470B_96BE_30B3BE4FD332#61195 AS COL_41AE8892_46A7_4ECE_B588_CB425C1B351C#61221, COL_1733FA1B_DAAD_4190_86A3_F964472ADC81#61194 AS COL_8639E8DD_E645_438D_8220_3B334C61C513#61222, last(COL_D62D1D3F_A2D2_4CF3_82C7_154802876D54#61201)()#61232 AS COL_479BB2B2_C798_4B3F_A3D2_AC43C8AFB16C#61233]

(14) TakeOrderedAndProject
Input [3]: [COL_41AE8892_46A7_4ECE_B588_CB425C1B351C#61221, COL_8639E8DD_E645_438D_8220_3B334C61C513#61222, COL_479BB2B2_C798_4B3F_A3D2_AC43C8AFB16C#61233]
Arguments: 500000, [COL_41AE8892_46A7_4ECE_B588_CB425C1B351C#61221 ASC NULLS FIRST, COL_8639E8DD_E645_438D_8220_3B334C61C513#61222 ASC NULLS FIRST], [COL_41AE8892_46A7_4ECE_B588_CB425C1B351C#61221, COL_8639E8DD_E645_438D_8220_3B334C61C513#61222, COL_479BB2B2_C798_4B3F_A3D2_AC43C8AFB16C#61233]

(15) AdaptiveSparkPlan
Output [3]: [COL_41AE8892_46A7_4ECE_B588_CB425C1B351C#61221, COL_8639E8DD_E645_438D_8220_3B334C61C513#61222, COL_479BB2B2_C798_4B3F_A3D2_AC43C8AFB16C#61233]
Arguments: isFinalPlan=true

SQL / DataFrame Properties

Name	Value
spark.sql.optimizer.nestedPredicatePushdown.supportedFileSources	parquet,orc,geoparquet