hdh-spark-job - Details for Query 1526

Submitted Time: 2026/10/23 14:48:17
Duration: 91 ms
Succeeded Jobs: 1543 1544

Show the Stage ID and Task ID that corresponds to the max metric

digraph G { 0 [labelType="html" label=" AdaptiveSparkPlan "]; subgraph cluster1 { isCluster="true"; label="WholeStageCodegen (2)\n \nduration: 0 ms"; 2 [labelType="html" label=" Project "]; 3 [labelType="html" label=" Project "]; 4 [labelType="html" label="HashAggregate time in aggregation build: 0 ms number of output rows: 1"]; } 5 [labelType="html" label="Exchange shuffle records written: 4 local merged chunks fetched: 0 shuffle write time total (min, med, max (stageId: taskId)) 1 ms (0 ms, 0 ms, 0 ms (stage 2250.0: task 1964)) remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 4 local bytes read: 234.0 B fetch wait time: 0 ms remote bytes read: 0.0 B merged fetch fallback count: 0 local blocks read: 4 remote merged chunks fetched: 0 remote blocks read: 0 data size total (min, med, max (stageId: taskId)) 96.0 B (24.0 B, 24.0 B, 24.0 B (stage 2250.0: task 1963)) local merged bytes read: 0.0 B number of partitions: 1 remote reqs duration: 0 ms remote bytes read to disk: 0.0 B shuffle bytes written total (min, med, max (stageId: taskId)) 234.0 B (57.0 B, 57.0 B, 63.0 B (stage 2250.0: task 1963))"]; subgraph cluster6 { isCluster="true"; label="WholeStageCodegen (1)\n \nduration: total (min, med, max (stageId: taskId))\n10 ms (2 ms, 2 ms, 4 ms (stage 2250.0: task 1963))"; 7 [labelType="html" label="HashAggregate time in aggregation build total (min, med, max (stageId: taskId)) 9 ms (2 ms, 2 ms, 3 ms (stage 2250.0: task 1963)) number of output rows: 4"]; 8 [labelType="html" label=" Project "]; 9 [labelType="html" label="ColumnarToRow number of output rows: 4,446 number of input batches: 2"]; } 10 [labelType="html" label="Scan parquet number of files read: 1 scan time total (min, med, max (stageId: taskId)) 9 ms (2 ms, 2 ms, 3 ms (stage 2250.0: task 1963)) metadata time: 0 ms size of files read: 12.7 MiB number of output rows: 4,446"]; 2->0; 3->2; 4->3; 5->4; 7->5; 8->7; 9->8; 10->9; }

AdaptiveSparkPlan isFinalPlan=true

Project [COL_80266E70_59B6_43AE_9AB2_9445811B5208#792671L, COL_208CC240_35A6_4318_B860_DB29C73D30B7#792672L, COL_0A846EF9_B70F_4C70_81D6_AB723504BCFB#792675, str_concat(array(dec_to_str(dec_round(dec_mult(array(dec_div(array(cast(COL_208CC240_35A6_4318_B860_DB29C73D30B7#792672L as double), COL_0A846EF9_B70F_4C70_81D6_AB723504BCFB#792675)), 100.0)), 2)), %)) AS COL_2325CB07_6A41_4B8D_BED4_3A87EA74321A#792679]

Project [COL_80266E70_59B6_43AE_9AB2_9445811B5208#792671L, COL_208CC240_35A6_4318_B860_DB29C73D30B7#792672L, dec_sum(array(COL_80266E70_59B6_43AE_9AB2_9445811B5208#792671L, COL_208CC240_35A6_4318_B860_DB29C73D30B7#792672L)) AS COL_0A846EF9_B70F_4C70_81D6_AB723504BCFB#792675]

HashAggregate(keys=[], functions=[sum(COL_249B43DD_69EF_40A7_B06E_C5681056E946#792641), sum(COL_4501100D_E2AA_48CB_9965_82A8E0C6886A#792642)])

WholeStageCodegen (2)

Exchange SinglePartition, ENSURE_REQUIREMENTS, [plan_id=42250]

HashAggregate(keys=[], functions=[partial_sum(COL_249B43DD_69EF_40A7_B06E_C5681056E946#792641), partial_sum(COL_4501100D_E2AA_48CB_9965_82A8E0C6886A#792642)])

Project [COL_D358A492_8954_41E5_99F5_A6ECB000829D#792489 AS COL_249B43DD_69EF_40A7_B06E_C5681056E946#792641, COL_F5F9154B_2BF5_41DC_95DC_85AD1B0D0ADF#792488 AS COL_4501100D_E2AA_48CB_9965_82A8E0C6886A#792642]

ColumnarToRow

WholeStageCodegen (1)

FileScan parquet [COL_F5F9154B_2BF5_41DC_95DC_85AD1B0D0ADF#792488,COL_D358A492_8954_41E5_99F5_A6ECB000829D#792489] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/data/output/cache/parquet/datablock/DATABLOCK_INFOR_326796], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<COL_F5F9154B_2BF5_41DC_95DC_85AD1B0D0ADF:int,COL_D358A492_8954_41E5_99F5_A6ECB000829D:int>

Details

== Physical Plan ==
AdaptiveSparkPlan (16)
+- == Final Plan ==
   * Project (9)
   +- * Project (8)
      +- * HashAggregate (7)
         +- ShuffleQueryStage (6), Statistics(sizeInBytes=96.0 B, rowCount=4)
            +- Exchange (5)
               +- * HashAggregate (4)
                  +- * Project (3)
                     +- * ColumnarToRow (2)
                        +- Scan parquet  (1)
+- == Initial Plan ==
   Project (15)
   +- Project (14)
      +- HashAggregate (13)
         +- Exchange (12)
            +- HashAggregate (11)
               +- Project (10)
                  +- Scan parquet  (1)


(1) Scan parquet 
Output [2]: [COL_F5F9154B_2BF5_41DC_95DC_85AD1B0D0ADF#792488, COL_D358A492_8954_41E5_99F5_A6ECB000829D#792489]
Batched: true
Location: InMemoryFileIndex [file:/data/output/cache/parquet/datablock/DATABLOCK_INFOR_326796]
ReadSchema: struct<COL_F5F9154B_2BF5_41DC_95DC_85AD1B0D0ADF:int,COL_D358A492_8954_41E5_99F5_A6ECB000829D:int>

(2) ColumnarToRow [codegen id : 1]
Input [2]: [COL_F5F9154B_2BF5_41DC_95DC_85AD1B0D0ADF#792488, COL_D358A492_8954_41E5_99F5_A6ECB000829D#792489]

(3) Project [codegen id : 1]
Output [2]: [COL_D358A492_8954_41E5_99F5_A6ECB000829D#792489 AS COL_249B43DD_69EF_40A7_B06E_C5681056E946#792641, COL_F5F9154B_2BF5_41DC_95DC_85AD1B0D0ADF#792488 AS COL_4501100D_E2AA_48CB_9965_82A8E0C6886A#792642]
Input [2]: [COL_F5F9154B_2BF5_41DC_95DC_85AD1B0D0ADF#792488, COL_D358A492_8954_41E5_99F5_A6ECB000829D#792489]

(4) HashAggregate [codegen id : 1]
Input [2]: [COL_249B43DD_69EF_40A7_B06E_C5681056E946#792641, COL_4501100D_E2AA_48CB_9965_82A8E0C6886A#792642]
Keys: []
Functions [2]: [partial_sum(COL_249B43DD_69EF_40A7_B06E_C5681056E946#792641), partial_sum(COL_4501100D_E2AA_48CB_9965_82A8E0C6886A#792642)]
Aggregate Attributes [2]: [sum#792708L, sum#792709L]
Results [2]: [sum#792710L, sum#792711L]

(5) Exchange
Input [2]: [sum#792710L, sum#792711L]
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=42250]

(6) ShuffleQueryStage
Output [2]: [sum#792710L, sum#792711L]
Arguments: 0

(7) HashAggregate [codegen id : 2]
Input [2]: [sum#792710L, sum#792711L]
Keys: []
Functions [2]: [sum(COL_249B43DD_69EF_40A7_B06E_C5681056E946#792641), sum(COL_4501100D_E2AA_48CB_9965_82A8E0C6886A#792642)]
Aggregate Attributes [2]: [sum(COL_249B43DD_69EF_40A7_B06E_C5681056E946#792641)#792657L, sum(COL_4501100D_E2AA_48CB_9965_82A8E0C6886A#792642)#792659L]
Results [2]: [sum(COL_249B43DD_69EF_40A7_B06E_C5681056E946#792641)#792657L AS COL_80266E70_59B6_43AE_9AB2_9445811B5208#792671L, sum(COL_4501100D_E2AA_48CB_9965_82A8E0C6886A#792642)#792659L AS COL_208CC240_35A6_4318_B860_DB29C73D30B7#792672L]

(8) Project [codegen id : 2]
Output [3]: [COL_80266E70_59B6_43AE_9AB2_9445811B5208#792671L, COL_208CC240_35A6_4318_B860_DB29C73D30B7#792672L, dec_sum(array(COL_80266E70_59B6_43AE_9AB2_9445811B5208#792671L, COL_208CC240_35A6_4318_B860_DB29C73D30B7#792672L)) AS COL_0A846EF9_B70F_4C70_81D6_AB723504BCFB#792675]
Input [2]: [COL_80266E70_59B6_43AE_9AB2_9445811B5208#792671L, COL_208CC240_35A6_4318_B860_DB29C73D30B7#792672L]

(9) Project [codegen id : 2]
Output [4]: [COL_80266E70_59B6_43AE_9AB2_9445811B5208#792671L, COL_208CC240_35A6_4318_B860_DB29C73D30B7#792672L, COL_0A846EF9_B70F_4C70_81D6_AB723504BCFB#792675, str_concat(array(dec_to_str(dec_round(dec_mult(array(dec_div(array(cast(COL_208CC240_35A6_4318_B860_DB29C73D30B7#792672L as double), COL_0A846EF9_B70F_4C70_81D6_AB723504BCFB#792675)), 100.0)), 2)),  %)) AS COL_2325CB07_6A41_4B8D_BED4_3A87EA74321A#792679]
Input [3]: [COL_80266E70_59B6_43AE_9AB2_9445811B5208#792671L, COL_208CC240_35A6_4318_B860_DB29C73D30B7#792672L, COL_0A846EF9_B70F_4C70_81D6_AB723504BCFB#792675]

(10) Project
Output [2]: [COL_D358A492_8954_41E5_99F5_A6ECB000829D#792489 AS COL_249B43DD_69EF_40A7_B06E_C5681056E946#792641, COL_F5F9154B_2BF5_41DC_95DC_85AD1B0D0ADF#792488 AS COL_4501100D_E2AA_48CB_9965_82A8E0C6886A#792642]
Input [2]: [COL_F5F9154B_2BF5_41DC_95DC_85AD1B0D0ADF#792488, COL_D358A492_8954_41E5_99F5_A6ECB000829D#792489]

(11) HashAggregate
Input [2]: [COL_249B43DD_69EF_40A7_B06E_C5681056E946#792641, COL_4501100D_E2AA_48CB_9965_82A8E0C6886A#792642]
Keys: []
Functions [2]: [partial_sum(COL_249B43DD_69EF_40A7_B06E_C5681056E946#792641), partial_sum(COL_4501100D_E2AA_48CB_9965_82A8E0C6886A#792642)]
Aggregate Attributes [2]: [sum#792708L, sum#792709L]
Results [2]: [sum#792710L, sum#792711L]

(12) Exchange
Input [2]: [sum#792710L, sum#792711L]
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=42231]

(13) HashAggregate
Input [2]: [sum#792710L, sum#792711L]
Keys: []
Functions [2]: [sum(COL_249B43DD_69EF_40A7_B06E_C5681056E946#792641), sum(COL_4501100D_E2AA_48CB_9965_82A8E0C6886A#792642)]
Aggregate Attributes [2]: [sum(COL_249B43DD_69EF_40A7_B06E_C5681056E946#792641)#792657L, sum(COL_4501100D_E2AA_48CB_9965_82A8E0C6886A#792642)#792659L]
Results [2]: [sum(COL_249B43DD_69EF_40A7_B06E_C5681056E946#792641)#792657L AS COL_80266E70_59B6_43AE_9AB2_9445811B5208#792671L, sum(COL_4501100D_E2AA_48CB_9965_82A8E0C6886A#792642)#792659L AS COL_208CC240_35A6_4318_B860_DB29C73D30B7#792672L]

(14) Project
Output [3]: [COL_80266E70_59B6_43AE_9AB2_9445811B5208#792671L, COL_208CC240_35A6_4318_B860_DB29C73D30B7#792672L, dec_sum(array(COL_80266E70_59B6_43AE_9AB2_9445811B5208#792671L, COL_208CC240_35A6_4318_B860_DB29C73D30B7#792672L)) AS COL_0A846EF9_B70F_4C70_81D6_AB723504BCFB#792675]
Input [2]: [COL_80266E70_59B6_43AE_9AB2_9445811B5208#792671L, COL_208CC240_35A6_4318_B860_DB29C73D30B7#792672L]

(15) Project
Output [4]: [COL_80266E70_59B6_43AE_9AB2_9445811B5208#792671L, COL_208CC240_35A6_4318_B860_DB29C73D30B7#792672L, COL_0A846EF9_B70F_4C70_81D6_AB723504BCFB#792675, str_concat(array(dec_to_str(dec_round(dec_mult(array(dec_div(array(cast(COL_208CC240_35A6_4318_B860_DB29C73D30B7#792672L as double), COL_0A846EF9_B70F_4C70_81D6_AB723504BCFB#792675)), 100.0)), 2)),  %)) AS COL_2325CB07_6A41_4B8D_BED4_3A87EA74321A#792679]
Input [3]: [COL_80266E70_59B6_43AE_9AB2_9445811B5208#792671L, COL_208CC240_35A6_4318_B860_DB29C73D30B7#792672L, COL_0A846EF9_B70F_4C70_81D6_AB723504BCFB#792675]

(16) AdaptiveSparkPlan
Output [4]: [COL_80266E70_59B6_43AE_9AB2_9445811B5208#792671L, COL_208CC240_35A6_4318_B860_DB29C73D30B7#792672L, COL_0A846EF9_B70F_4C70_81D6_AB723504BCFB#792675, COL_2325CB07_6A41_4B8D_BED4_3A87EA74321A#792679]
Arguments: isFinalPlan=true

SQL / DataFrame Properties

Name	Value
spark.sql.optimizer.nestedPredicatePushdown.supportedFileSources	parquet,orc,geoparquet