hdh-spark-job - Details for Query 1510

Submitted Time: 2026/10/23 14:48:14
Duration: 0.2 s
Succeeded Jobs: 1517 1518 1519

Show the Stage ID and Task ID that corresponds to the max metric

digraph G { 0 [labelType="html" label=" AdaptiveSparkPlan "]; subgraph cluster1 { isCluster="true"; label="WholeStageCodegen (4)\n \nduration: 1 ms"; 2 [labelType="html" label=" Project "]; 3 [labelType="html" label="HashAggregate spill size: 0.0 B time in aggregation build: 0 ms peak memory: 64.2 MiB number of output rows: 1 number of sort fallback tasks: 0 avg hash probes per key: 1"]; } 4 [labelType="html" label="AQEShuffleRead number of partitions: 1 partition data size: 66.0 B number of coalesced partitions: 1"]; 5 [labelType="html" label="Exchange shuffle records written: 1 local merged chunks fetched: 0 shuffle write time: 0 ms remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 1 local bytes read: 64.0 B fetch wait time: 0 ms remote bytes read: 0.0 B merged fetch fallback count: 0 local blocks read: 1 remote merged chunks fetched: 0 remote blocks read: 0 data size: 32.0 B local merged bytes read: 0.0 B number of partitions: 10 remote reqs duration: 0 ms remote bytes read to disk: 0.0 B shuffle bytes written: 64.0 B"]; subgraph cluster6 { isCluster="true"; label="WholeStageCodegen (3)\n \nduration: 12 ms"; 7 [labelType="html" label="HashAggregate spill size: 0.0 B time in aggregation build: 2 ms peak memory: 256.0 KiB number of output rows: 1 number of sort fallback tasks: 0 avg hash probes per key: 0"]; 8 [labelType="html" label="Filter number of output rows: 505"]; } 9 [labelType="html" label="Window spill size: 0.0 B"]; subgraph cluster10 { isCluster="true"; label="WholeStageCodegen (2)\n \nduration: 29 ms"; 11 [labelType="html" label="Sort sort time: 0 ms peak memory: 64.1 MiB spill size: 0.0 B"]; } 12 [labelType="html" label="AQEShuffleRead number of partitions: 1 partition data size: 1692.0 B number of coalesced partitions: 1"]; 13 [labelType="html" label="Exchange shuffle records written: 2,873 local merged chunks fetched: 0 shuffle write time: 0 ms remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 2,873 local bytes read: 1588.0 B fetch wait time: 0 ms remote bytes read: 0.0 B merged fetch fallback count: 0 local blocks read: 1 remote merged chunks fetched: 0 remote blocks read: 0 data size: 44.9 KiB local merged bytes read: 0.0 B number of partitions: 10 remote reqs duration: 0 ms remote bytes read to disk: 0.0 B shuffle bytes written: 1588.0 B"]; subgraph cluster14 { isCluster="true"; label="WholeStageCodegen (1)\n \nduration: 26 ms"; 15 [labelType="html" label=" Project "]; 16 [labelType="html" label="ColumnarToRow number of output rows: 2,873 number of input batches: 1"]; } 17 [labelType="html" label="Scan parquet number of files read: 1 scan time: 23 ms metadata time: 0 ms size of files read: 446.0 KiB number of output rows: 2,873"]; 2->0; 3->2; 4->3; 5->4; 7->5; 8->7; 9->8; 11->9; 12->11; 13->12; 15->13; 16->15; 17->16; }

AdaptiveSparkPlan isFinalPlan=true

Project [COL_60C1A3D1_598B_49D4_B5CE_396F67C29CF5#790593, COL_19464531_C3A0_405B_89D5_2760216560E0#790594L, COL_FA3F0A75_771C_495A_8837_873B81F6CAFA#790595L, str_concat(array(bigint_to_str(dec_fix(dec_operator(PRODUIT, array(dec_operator(DIVISION, array(COL_FA3F0A75_771C_495A_8837_873B81F6CAFA#790595L, COL_19464531_C3A0_405B_89D5_2760216560E0#790594L)), 100.0)))), , %)) AS COL_E01D9084_14AD_4D8D_BA8F_A6AFBF9379D1#790607]

HashAggregate(keys=[COL_79AB3626_4DF1_4A00_8B9D_4C226BD6554C#790301, COL_E9807AF4_1F11_4218_865F_3A9CA35E8F25#790390L], functions=[count(1)])

WholeStageCodegen (4)

AQEShuffleRead coalesced

Exchange hashpartitioning(COL_79AB3626_4DF1_4A00_8B9D_4C226BD6554C#790301, COL_E9807AF4_1F11_4218_865F_3A9CA35E8F25#790390L, 10), ENSURE_REQUIREMENTS, [plan_id=41655]

HashAggregate(keys=[COL_79AB3626_4DF1_4A00_8B9D_4C226BD6554C#790301, COL_E9807AF4_1F11_4218_865F_3A9CA35E8F25#790390L], functions=[partial_count(1)])

Filter (COL_79AB3626_4DF1_4A00_8B9D_4C226BD6554C#790301 <=> 0)

WholeStageCodegen (3)

Window [count(1) windowspecdefinition(1, 1 DESC NULLS LAST, specifiedwindowframe(RangeFrame, unboundedpreceding$(), currentrow$())) AS COL_E9807AF4_1F11_4218_865F_3A9CA35E8F25#790390L], [1], [1 DESC NULLS LAST]

Sort [1 ASC NULLS FIRST, 1 DESC NULLS LAST], false, 0

WholeStageCodegen (2)

AQEShuffleRead coalesced

Exchange hashpartitioning(1, 10), ENSURE_REQUIREMENTS, [plan_id=41593]

Project [COL_EBA13257_701E_4FA9_A41F_8C295EADB9DF#789987 AS COL_79AB3626_4DF1_4A00_8B9D_4C226BD6554C#790301]

ColumnarToRow

WholeStageCodegen (1)

FileScan parquet [COL_EBA13257_701E_4FA9_A41F_8C295EADB9DF#789987] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/data/output/cache/parquet/datablock/LES_DEPOTS_INFO_326686], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<COL_EBA13257_701E_4FA9_A41F_8C295EADB9DF:int>

Details

== Physical Plan ==
AdaptiveSparkPlan (26)
+- == Final Plan ==
   * Project (15)
   +- * HashAggregate (14)
      +- AQEShuffleRead (13)
         +- ShuffleQueryStage (12), Statistics(sizeInBytes=32.0 B, rowCount=1)
            +- Exchange (11)
               +- * HashAggregate (10)
                  +- * Filter (9)
                     +- Window (8)
                        +- * Sort (7)
                           +- AQEShuffleRead (6)
                              +- ShuffleQueryStage (5), Statistics(sizeInBytes=44.9 KiB, rowCount=2.87E+3)
                                 +- Exchange (4)
                                    +- * Project (3)
                                       +- * ColumnarToRow (2)
                                          +- Scan parquet  (1)
+- == Initial Plan ==
   CollectLimit (25)
   +- Project (24)
      +- HashAggregate (23)
         +- Exchange (22)
            +- HashAggregate (21)
               +- Filter (20)
                  +- Window (19)
                     +- Sort (18)
                        +- Exchange (17)
                           +- Project (16)
                              +- Scan parquet  (1)


(1) Scan parquet 
Output [1]: [COL_EBA13257_701E_4FA9_A41F_8C295EADB9DF#789987]
Batched: true
Location: InMemoryFileIndex [file:/data/output/cache/parquet/datablock/LES_DEPOTS_INFO_326686]
ReadSchema: struct<COL_EBA13257_701E_4FA9_A41F_8C295EADB9DF:int>

(2) ColumnarToRow [codegen id : 1]
Input [1]: [COL_EBA13257_701E_4FA9_A41F_8C295EADB9DF#789987]

(3) Project [codegen id : 1]
Output [1]: [COL_EBA13257_701E_4FA9_A41F_8C295EADB9DF#789987 AS COL_79AB3626_4DF1_4A00_8B9D_4C226BD6554C#790301]
Input [1]: [COL_EBA13257_701E_4FA9_A41F_8C295EADB9DF#789987]

(4) Exchange
Input [1]: [COL_79AB3626_4DF1_4A00_8B9D_4C226BD6554C#790301]
Arguments: hashpartitioning(1, 10), ENSURE_REQUIREMENTS, [plan_id=41593]

(5) ShuffleQueryStage
Output [1]: [COL_79AB3626_4DF1_4A00_8B9D_4C226BD6554C#790301]
Arguments: 0

(6) AQEShuffleRead
Input [1]: [COL_79AB3626_4DF1_4A00_8B9D_4C226BD6554C#790301]
Arguments: coalesced

(7) Sort [codegen id : 2]
Input [1]: [COL_79AB3626_4DF1_4A00_8B9D_4C226BD6554C#790301]
Arguments: [1 ASC NULLS FIRST, 1 DESC NULLS LAST], false, 0

(8) Window
Input [1]: [COL_79AB3626_4DF1_4A00_8B9D_4C226BD6554C#790301]
Arguments: [count(1) windowspecdefinition(1, 1 DESC NULLS LAST, specifiedwindowframe(RangeFrame, unboundedpreceding$(), currentrow$())) AS COL_E9807AF4_1F11_4218_865F_3A9CA35E8F25#790390L], [1], [1 DESC NULLS LAST]

(9) Filter [codegen id : 3]
Input [2]: [COL_79AB3626_4DF1_4A00_8B9D_4C226BD6554C#790301, COL_E9807AF4_1F11_4218_865F_3A9CA35E8F25#790390L]
Condition : (COL_79AB3626_4DF1_4A00_8B9D_4C226BD6554C#790301 <=> 0)

(10) HashAggregate [codegen id : 3]
Input [2]: [COL_79AB3626_4DF1_4A00_8B9D_4C226BD6554C#790301, COL_E9807AF4_1F11_4218_865F_3A9CA35E8F25#790390L]
Keys [2]: [COL_79AB3626_4DF1_4A00_8B9D_4C226BD6554C#790301, COL_E9807AF4_1F11_4218_865F_3A9CA35E8F25#790390L]
Functions [1]: [partial_count(1)]
Aggregate Attributes [1]: [count#790612L]
Results [3]: [COL_79AB3626_4DF1_4A00_8B9D_4C226BD6554C#790301, COL_E9807AF4_1F11_4218_865F_3A9CA35E8F25#790390L, count#790613L]

(11) Exchange
Input [3]: [COL_79AB3626_4DF1_4A00_8B9D_4C226BD6554C#790301, COL_E9807AF4_1F11_4218_865F_3A9CA35E8F25#790390L, count#790613L]
Arguments: hashpartitioning(COL_79AB3626_4DF1_4A00_8B9D_4C226BD6554C#790301, COL_E9807AF4_1F11_4218_865F_3A9CA35E8F25#790390L, 10), ENSURE_REQUIREMENTS, [plan_id=41655]

(12) ShuffleQueryStage
Output [3]: [COL_79AB3626_4DF1_4A00_8B9D_4C226BD6554C#790301, COL_E9807AF4_1F11_4218_865F_3A9CA35E8F25#790390L, count#790613L]
Arguments: 1

(13) AQEShuffleRead
Input [3]: [COL_79AB3626_4DF1_4A00_8B9D_4C226BD6554C#790301, COL_E9807AF4_1F11_4218_865F_3A9CA35E8F25#790390L, count#790613L]
Arguments: coalesced

(14) HashAggregate [codegen id : 4]
Input [3]: [COL_79AB3626_4DF1_4A00_8B9D_4C226BD6554C#790301, COL_E9807AF4_1F11_4218_865F_3A9CA35E8F25#790390L, count#790613L]
Keys [2]: [COL_79AB3626_4DF1_4A00_8B9D_4C226BD6554C#790301, COL_E9807AF4_1F11_4218_865F_3A9CA35E8F25#790390L]
Functions [1]: [count(1)]
Aggregate Attributes [1]: [count(1)#790576L]
Results [3]: [COL_79AB3626_4DF1_4A00_8B9D_4C226BD6554C#790301 AS COL_60C1A3D1_598B_49D4_B5CE_396F67C29CF5#790593, COL_E9807AF4_1F11_4218_865F_3A9CA35E8F25#790390L AS COL_19464531_C3A0_405B_89D5_2760216560E0#790594L, count(1)#790576L AS COL_FA3F0A75_771C_495A_8837_873B81F6CAFA#790595L]

(15) Project [codegen id : 4]
Output [4]: [COL_60C1A3D1_598B_49D4_B5CE_396F67C29CF5#790593, COL_19464531_C3A0_405B_89D5_2760216560E0#790594L, COL_FA3F0A75_771C_495A_8837_873B81F6CAFA#790595L, str_concat(array(bigint_to_str(dec_fix(dec_operator(PRODUIT, array(dec_operator(DIVISION, array(COL_FA3F0A75_771C_495A_8837_873B81F6CAFA#790595L, COL_19464531_C3A0_405B_89D5_2760216560E0#790594L)), 100.0)))),  , %)) AS COL_E01D9084_14AD_4D8D_BA8F_A6AFBF9379D1#790607]
Input [3]: [COL_60C1A3D1_598B_49D4_B5CE_396F67C29CF5#790593, COL_19464531_C3A0_405B_89D5_2760216560E0#790594L, COL_FA3F0A75_771C_495A_8837_873B81F6CAFA#790595L]

(16) Project
Output [1]: [COL_EBA13257_701E_4FA9_A41F_8C295EADB9DF#789987 AS COL_79AB3626_4DF1_4A00_8B9D_4C226BD6554C#790301]
Input [1]: [COL_EBA13257_701E_4FA9_A41F_8C295EADB9DF#789987]

(17) Exchange
Input [1]: [COL_79AB3626_4DF1_4A00_8B9D_4C226BD6554C#790301]
Arguments: hashpartitioning(1, 10), ENSURE_REQUIREMENTS, [plan_id=41563]

(18) Sort
Input [1]: [COL_79AB3626_4DF1_4A00_8B9D_4C226BD6554C#790301]
Arguments: [1 ASC NULLS FIRST, 1 DESC NULLS LAST], false, 0

(19) Window
Input [1]: [COL_79AB3626_4DF1_4A00_8B9D_4C226BD6554C#790301]
Arguments: [count(1) windowspecdefinition(1, 1 DESC NULLS LAST, specifiedwindowframe(RangeFrame, unboundedpreceding$(), currentrow$())) AS COL_E9807AF4_1F11_4218_865F_3A9CA35E8F25#790390L], [1], [1 DESC NULLS LAST]

(20) Filter
Input [2]: [COL_79AB3626_4DF1_4A00_8B9D_4C226BD6554C#790301, COL_E9807AF4_1F11_4218_865F_3A9CA35E8F25#790390L]
Condition : (COL_79AB3626_4DF1_4A00_8B9D_4C226BD6554C#790301 <=> 0)

(21) HashAggregate
Input [2]: [COL_79AB3626_4DF1_4A00_8B9D_4C226BD6554C#790301, COL_E9807AF4_1F11_4218_865F_3A9CA35E8F25#790390L]
Keys [2]: [COL_79AB3626_4DF1_4A00_8B9D_4C226BD6554C#790301, COL_E9807AF4_1F11_4218_865F_3A9CA35E8F25#790390L]
Functions [1]: [partial_count(1)]
Aggregate Attributes [1]: [count#790612L]
Results [3]: [COL_79AB3626_4DF1_4A00_8B9D_4C226BD6554C#790301, COL_E9807AF4_1F11_4218_865F_3A9CA35E8F25#790390L, count#790613L]

(22) Exchange
Input [3]: [COL_79AB3626_4DF1_4A00_8B9D_4C226BD6554C#790301, COL_E9807AF4_1F11_4218_865F_3A9CA35E8F25#790390L, count#790613L]
Arguments: hashpartitioning(COL_79AB3626_4DF1_4A00_8B9D_4C226BD6554C#790301, COL_E9807AF4_1F11_4218_865F_3A9CA35E8F25#790390L, 10), ENSURE_REQUIREMENTS, [plan_id=41569]

(23) HashAggregate
Input [3]: [COL_79AB3626_4DF1_4A00_8B9D_4C226BD6554C#790301, COL_E9807AF4_1F11_4218_865F_3A9CA35E8F25#790390L, count#790613L]
Keys [2]: [COL_79AB3626_4DF1_4A00_8B9D_4C226BD6554C#790301, COL_E9807AF4_1F11_4218_865F_3A9CA35E8F25#790390L]
Functions [1]: [count(1)]
Aggregate Attributes [1]: [count(1)#790576L]
Results [3]: [COL_79AB3626_4DF1_4A00_8B9D_4C226BD6554C#790301 AS COL_60C1A3D1_598B_49D4_B5CE_396F67C29CF5#790593, COL_E9807AF4_1F11_4218_865F_3A9CA35E8F25#790390L AS COL_19464531_C3A0_405B_89D5_2760216560E0#790594L, count(1)#790576L AS COL_FA3F0A75_771C_495A_8837_873B81F6CAFA#790595L]

(24) Project
Output [4]: [COL_60C1A3D1_598B_49D4_B5CE_396F67C29CF5#790593, COL_19464531_C3A0_405B_89D5_2760216560E0#790594L, COL_FA3F0A75_771C_495A_8837_873B81F6CAFA#790595L, str_concat(array(bigint_to_str(dec_fix(dec_operator(PRODUIT, array(dec_operator(DIVISION, array(COL_FA3F0A75_771C_495A_8837_873B81F6CAFA#790595L, COL_19464531_C3A0_405B_89D5_2760216560E0#790594L)), 100.0)))),  , %)) AS COL_E01D9084_14AD_4D8D_BA8F_A6AFBF9379D1#790607]
Input [3]: [COL_60C1A3D1_598B_49D4_B5CE_396F67C29CF5#790593, COL_19464531_C3A0_405B_89D5_2760216560E0#790594L, COL_FA3F0A75_771C_495A_8837_873B81F6CAFA#790595L]

(25) CollectLimit
Input [4]: [COL_60C1A3D1_598B_49D4_B5CE_396F67C29CF5#790593, COL_19464531_C3A0_405B_89D5_2760216560E0#790594L, COL_FA3F0A75_771C_495A_8837_873B81F6CAFA#790595L, COL_E01D9084_14AD_4D8D_BA8F_A6AFBF9379D1#790607]
Arguments: 500000

(26) AdaptiveSparkPlan
Output [4]: [COL_60C1A3D1_598B_49D4_B5CE_396F67C29CF5#790593, COL_19464531_C3A0_405B_89D5_2760216560E0#790594L, COL_FA3F0A75_771C_495A_8837_873B81F6CAFA#790595L, COL_E01D9084_14AD_4D8D_BA8F_A6AFBF9379D1#790607]
Arguments: isFinalPlan=true

SQL / DataFrame Properties

Name	Value
spark.sql.optimizer.nestedPredicatePushdown.supportedFileSources	parquet,orc,geoparquet