hdh-spark-job - Details for Query 1503

Submitted Time: 2026/10/23 14:48:14
Duration: 0.6 s
Succeeded Jobs: 1503 1510 1514

Show the Stage ID and Task ID that corresponds to the max metric

digraph G { 0 [labelType="html" label=" AdaptiveSparkPlan "]; subgraph cluster1 { isCluster="true"; label="WholeStageCodegen (4)\n \nduration: 2 ms"; 2 [labelType="html" label=" Project "]; 3 [labelType="html" label="HashAggregate spill size: 0.0 B time in aggregation build: 0 ms peak memory: 64.2 MiB number of output rows: 1 number of sort fallback tasks: 0 avg hash probes per key: 1"]; } 4 [labelType="html" label="AQEShuffleRead number of partitions: 1 partition data size: 97.0 B number of coalesced partitions: 1"]; 5 [labelType="html" label="Exchange shuffle records written: 1 local merged chunks fetched: 0 shuffle write time: 0 ms remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 1 local bytes read: 94.0 B fetch wait time: 0 ms remote bytes read: 0.0 B merged fetch fallback count: 0 local blocks read: 1 remote merged chunks fetched: 0 remote blocks read: 0 data size: 56.0 B local merged bytes read: 0.0 B number of partitions: 10 remote reqs duration: 0 ms remote bytes read to disk: 0.0 B shuffle bytes written: 94.0 B"]; subgraph cluster6 { isCluster="true"; label="WholeStageCodegen (3)\n \nduration: 20 ms"; 7 [labelType="html" label="HashAggregate spill size: 0.0 B time in aggregation build: 5 ms peak memory: 256.0 KiB number of output rows: 1 number of sort fallback tasks: 0 avg hash probes per key: 0"]; 8 [labelType="html" label="Filter number of output rows: 80"]; } 9 [labelType="html" label="Window spill size: 0.0 B"]; subgraph cluster10 { isCluster="true"; label="WholeStageCodegen (2)\n \nduration: 64 ms"; 11 [labelType="html" label="Sort sort time: 0 ms peak memory: 64.1 MiB spill size: 0.0 B"]; } 12 [labelType="html" label="AQEShuffleRead number of partitions: 1 partition data size: 1156.0 B number of coalesced partitions: 1"]; 13 [labelType="html" label="Exchange shuffle records written: 2,175 local merged chunks fetched: 0 shuffle write time: 0 ms remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 2,175 local bytes read: 1069.0 B fetch wait time: 0 ms remote bytes read: 0.0 B merged fetch fallback count: 0 local blocks read: 1 remote merged chunks fetched: 0 remote blocks read: 0 data size: 85.0 KiB local merged bytes read: 0.0 B number of partitions: 10 remote reqs duration: 0 ms remote bytes read to disk: 0.0 B shuffle bytes written: 1069.0 B"]; subgraph cluster14 { isCluster="true"; label="WholeStageCodegen (1)\n \nduration: 33 ms"; 15 [labelType="html" label=" Project "]; 16 [labelType="html" label="ColumnarToRow number of output rows: 2,175 number of input batches: 1"]; } 17 [labelType="html" label="Scan parquet number of files read: 1 scan time: 21 ms metadata time: 0 ms size of files read: 74.7 KiB number of output rows: 2,175"]; 2->0; 3->2; 4->3; 5->4; 7->5; 8->7; 9->8; 11->9; 12->11; 13->12; 15->13; 16->15; 17->16; }

AdaptiveSparkPlan isFinalPlan=true

Project [COL_B48461D6_A3BC_45F3_AB8F_03BF01D41C93#789383, COL_4584C49D_AB64_44A6_9441_F5A5A85410A8#789384L, COL_30DDCE22_5C87_4039_9224_AD4B5B64084B#789385L, str_concat(array(str_concat(array(bigint_to_str(dec_fix(dec_operator(PRODUIT, array(dec_operator(DIVISION, array(COL_30DDCE22_5C87_4039_9224_AD4B5B64084B#789385L, COL_4584C49D_AB64_44A6_9441_F5A5A85410A8#789384L)), 100.0)))), , )), , %)) AS COL_3C8D7FB5_B59E_41C3_9D27_870F51F07F2E#789489]

HashAggregate(keys=[COL_6A7222C5_E1C1_4F61_8065_319A3FBE09A1#788566, COL_E34D9E0C_E565_42B9_AFA1_A780AB15F0DF#788724L], functions=[count(1)])

WholeStageCodegen (4)

AQEShuffleRead coalesced

Exchange hashpartitioning(COL_6A7222C5_E1C1_4F61_8065_319A3FBE09A1#788566, COL_E34D9E0C_E565_42B9_AFA1_A780AB15F0DF#788724L, 10), ENSURE_REQUIREMENTS, [plan_id=41367]

HashAggregate(keys=[COL_6A7222C5_E1C1_4F61_8065_319A3FBE09A1#788566, COL_E34D9E0C_E565_42B9_AFA1_A780AB15F0DF#788724L], functions=[partial_count(1)])

Filter CASE WHEN isnull(Contains(COL_6A7222C5_E1C1_4F61_8065_319A3FBE09A1#788566, Absence)) THEN false ELSE Contains(COL_6A7222C5_E1C1_4F61_8065_319A3FBE09A1#788566, Absence) END

WholeStageCodegen (3)

Window [count(1) windowspecdefinition(1, 1 DESC NULLS LAST, specifiedwindowframe(RangeFrame, unboundedpreceding$(), currentrow$())) AS COL_E34D9E0C_E565_42B9_AFA1_A780AB15F0DF#788724L], [1], [1 DESC NULLS LAST]

Sort [1 ASC NULLS FIRST, 1 DESC NULLS LAST], false, 0

WholeStageCodegen (2)

AQEShuffleRead coalesced

Exchange hashpartitioning(1, 10), ENSURE_REQUIREMENTS, [plan_id=41270]

Project [str_if_with_rule((COL_986311C7_DD53_48E4_873B_CF92B4323DE8#788154 <=> 0), Absence de Dépôt, Présence de dépôts) AS COL_6A7222C5_E1C1_4F61_8065_319A3FBE09A1#788566]

ColumnarToRow

WholeStageCodegen (1)

FileScan parquet [COL_986311C7_DD53_48E4_873B_CF92B4323DE8#788154] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/data/output/cache/parquet/datablock/ENTITE_METIER_I_326771], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<COL_986311C7_DD53_48E4_873B_CF92B4323DE8:int>

Details

== Physical Plan ==
AdaptiveSparkPlan (26)
+- == Final Plan ==
   * Project (15)
   +- * HashAggregate (14)
      +- AQEShuffleRead (13)
         +- ShuffleQueryStage (12), Statistics(sizeInBytes=56.0 B, rowCount=1)
            +- Exchange (11)
               +- * HashAggregate (10)
                  +- * Filter (9)
                     +- Window (8)
                        +- * Sort (7)
                           +- AQEShuffleRead (6)
                              +- ShuffleQueryStage (5), Statistics(sizeInBytes=85.0 KiB, rowCount=2.18E+3)
                                 +- Exchange (4)
                                    +- * Project (3)
                                       +- * ColumnarToRow (2)
                                          +- Scan parquet  (1)
+- == Initial Plan ==
   CollectLimit (25)
   +- Project (24)
      +- HashAggregate (23)
         +- Exchange (22)
            +- HashAggregate (21)
               +- Filter (20)
                  +- Window (19)
                     +- Sort (18)
                        +- Exchange (17)
                           +- Project (16)
                              +- Scan parquet  (1)


(1) Scan parquet 
Output [1]: [COL_986311C7_DD53_48E4_873B_CF92B4323DE8#788154]
Batched: true
Location: InMemoryFileIndex [file:/data/output/cache/parquet/datablock/ENTITE_METIER_I_326771]
ReadSchema: struct<COL_986311C7_DD53_48E4_873B_CF92B4323DE8:int>

(2) ColumnarToRow [codegen id : 1]
Input [1]: [COL_986311C7_DD53_48E4_873B_CF92B4323DE8#788154]

(3) Project [codegen id : 1]
Output [1]: [str_if_with_rule((COL_986311C7_DD53_48E4_873B_CF92B4323DE8#788154 <=> 0), Absence de Dépôt, Présence de dépôts) AS COL_6A7222C5_E1C1_4F61_8065_319A3FBE09A1#788566]
Input [1]: [COL_986311C7_DD53_48E4_873B_CF92B4323DE8#788154]

(4) Exchange
Input [1]: [COL_6A7222C5_E1C1_4F61_8065_319A3FBE09A1#788566]
Arguments: hashpartitioning(1, 10), ENSURE_REQUIREMENTS, [plan_id=41270]

(5) ShuffleQueryStage
Output [1]: [COL_6A7222C5_E1C1_4F61_8065_319A3FBE09A1#788566]
Arguments: 0

(6) AQEShuffleRead
Input [1]: [COL_6A7222C5_E1C1_4F61_8065_319A3FBE09A1#788566]
Arguments: coalesced

(7) Sort [codegen id : 2]
Input [1]: [COL_6A7222C5_E1C1_4F61_8065_319A3FBE09A1#788566]
Arguments: [1 ASC NULLS FIRST, 1 DESC NULLS LAST], false, 0

(8) Window
Input [1]: [COL_6A7222C5_E1C1_4F61_8065_319A3FBE09A1#788566]
Arguments: [count(1) windowspecdefinition(1, 1 DESC NULLS LAST, specifiedwindowframe(RangeFrame, unboundedpreceding$(), currentrow$())) AS COL_E34D9E0C_E565_42B9_AFA1_A780AB15F0DF#788724L], [1], [1 DESC NULLS LAST]

(9) Filter [codegen id : 3]
Input [2]: [COL_6A7222C5_E1C1_4F61_8065_319A3FBE09A1#788566, COL_E34D9E0C_E565_42B9_AFA1_A780AB15F0DF#788724L]
Condition : CASE WHEN isnull(Contains(COL_6A7222C5_E1C1_4F61_8065_319A3FBE09A1#788566, Absence)) THEN false ELSE Contains(COL_6A7222C5_E1C1_4F61_8065_319A3FBE09A1#788566, Absence) END

(10) HashAggregate [codegen id : 3]
Input [2]: [COL_6A7222C5_E1C1_4F61_8065_319A3FBE09A1#788566, COL_E34D9E0C_E565_42B9_AFA1_A780AB15F0DF#788724L]
Keys [2]: [COL_6A7222C5_E1C1_4F61_8065_319A3FBE09A1#788566, COL_E34D9E0C_E565_42B9_AFA1_A780AB15F0DF#788724L]
Functions [1]: [partial_count(1)]
Aggregate Attributes [1]: [count#789526L]
Results [3]: [COL_6A7222C5_E1C1_4F61_8065_319A3FBE09A1#788566, COL_E34D9E0C_E565_42B9_AFA1_A780AB15F0DF#788724L, count#789527L]

(11) Exchange
Input [3]: [COL_6A7222C5_E1C1_4F61_8065_319A3FBE09A1#788566, COL_E34D9E0C_E565_42B9_AFA1_A780AB15F0DF#788724L, count#789527L]
Arguments: hashpartitioning(COL_6A7222C5_E1C1_4F61_8065_319A3FBE09A1#788566, COL_E34D9E0C_E565_42B9_AFA1_A780AB15F0DF#788724L, 10), ENSURE_REQUIREMENTS, [plan_id=41367]

(12) ShuffleQueryStage
Output [3]: [COL_6A7222C5_E1C1_4F61_8065_319A3FBE09A1#788566, COL_E34D9E0C_E565_42B9_AFA1_A780AB15F0DF#788724L, count#789527L]
Arguments: 1

(13) AQEShuffleRead
Input [3]: [COL_6A7222C5_E1C1_4F61_8065_319A3FBE09A1#788566, COL_E34D9E0C_E565_42B9_AFA1_A780AB15F0DF#788724L, count#789527L]
Arguments: coalesced

(14) HashAggregate [codegen id : 4]
Input [3]: [COL_6A7222C5_E1C1_4F61_8065_319A3FBE09A1#788566, COL_E34D9E0C_E565_42B9_AFA1_A780AB15F0DF#788724L, count#789527L]
Keys [2]: [COL_6A7222C5_E1C1_4F61_8065_319A3FBE09A1#788566, COL_E34D9E0C_E565_42B9_AFA1_A780AB15F0DF#788724L]
Functions [1]: [count(1)]
Aggregate Attributes [1]: [count(1)#789148L]
Results [3]: [COL_6A7222C5_E1C1_4F61_8065_319A3FBE09A1#788566 AS COL_B48461D6_A3BC_45F3_AB8F_03BF01D41C93#789383, COL_E34D9E0C_E565_42B9_AFA1_A780AB15F0DF#788724L AS COL_4584C49D_AB64_44A6_9441_F5A5A85410A8#789384L, count(1)#789148L AS COL_30DDCE22_5C87_4039_9224_AD4B5B64084B#789385L]

(15) Project [codegen id : 4]
Output [4]: [COL_B48461D6_A3BC_45F3_AB8F_03BF01D41C93#789383, COL_4584C49D_AB64_44A6_9441_F5A5A85410A8#789384L, COL_30DDCE22_5C87_4039_9224_AD4B5B64084B#789385L, str_concat(array(str_concat(array(bigint_to_str(dec_fix(dec_operator(PRODUIT, array(dec_operator(DIVISION, array(COL_30DDCE22_5C87_4039_9224_AD4B5B64084B#789385L, COL_4584C49D_AB64_44A6_9441_F5A5A85410A8#789384L)), 100.0)))),  , )),  , %)) AS COL_3C8D7FB5_B59E_41C3_9D27_870F51F07F2E#789489]
Input [3]: [COL_B48461D6_A3BC_45F3_AB8F_03BF01D41C93#789383, COL_4584C49D_AB64_44A6_9441_F5A5A85410A8#789384L, COL_30DDCE22_5C87_4039_9224_AD4B5B64084B#789385L]

(16) Project
Output [1]: [str_if_with_rule((COL_986311C7_DD53_48E4_873B_CF92B4323DE8#788154 <=> 0), Absence de Dépôt, Présence de dépôts) AS COL_6A7222C5_E1C1_4F61_8065_319A3FBE09A1#788566]
Input [1]: [COL_986311C7_DD53_48E4_873B_CF92B4323DE8#788154]

(17) Exchange
Input [1]: [COL_6A7222C5_E1C1_4F61_8065_319A3FBE09A1#788566]
Arguments: hashpartitioning(1, 10), ENSURE_REQUIREMENTS, [plan_id=41240]

(18) Sort
Input [1]: [COL_6A7222C5_E1C1_4F61_8065_319A3FBE09A1#788566]
Arguments: [1 ASC NULLS FIRST, 1 DESC NULLS LAST], false, 0

(19) Window
Input [1]: [COL_6A7222C5_E1C1_4F61_8065_319A3FBE09A1#788566]
Arguments: [count(1) windowspecdefinition(1, 1 DESC NULLS LAST, specifiedwindowframe(RangeFrame, unboundedpreceding$(), currentrow$())) AS COL_E34D9E0C_E565_42B9_AFA1_A780AB15F0DF#788724L], [1], [1 DESC NULLS LAST]

(20) Filter
Input [2]: [COL_6A7222C5_E1C1_4F61_8065_319A3FBE09A1#788566, COL_E34D9E0C_E565_42B9_AFA1_A780AB15F0DF#788724L]
Condition : CASE WHEN isnull(Contains(COL_6A7222C5_E1C1_4F61_8065_319A3FBE09A1#788566, Absence)) THEN false ELSE Contains(COL_6A7222C5_E1C1_4F61_8065_319A3FBE09A1#788566, Absence) END

(21) HashAggregate
Input [2]: [COL_6A7222C5_E1C1_4F61_8065_319A3FBE09A1#788566, COL_E34D9E0C_E565_42B9_AFA1_A780AB15F0DF#788724L]
Keys [2]: [COL_6A7222C5_E1C1_4F61_8065_319A3FBE09A1#788566, COL_E34D9E0C_E565_42B9_AFA1_A780AB15F0DF#788724L]
Functions [1]: [partial_count(1)]
Aggregate Attributes [1]: [count#789526L]
Results [3]: [COL_6A7222C5_E1C1_4F61_8065_319A3FBE09A1#788566, COL_E34D9E0C_E565_42B9_AFA1_A780AB15F0DF#788724L, count#789527L]

(22) Exchange
Input [3]: [COL_6A7222C5_E1C1_4F61_8065_319A3FBE09A1#788566, COL_E34D9E0C_E565_42B9_AFA1_A780AB15F0DF#788724L, count#789527L]
Arguments: hashpartitioning(COL_6A7222C5_E1C1_4F61_8065_319A3FBE09A1#788566, COL_E34D9E0C_E565_42B9_AFA1_A780AB15F0DF#788724L, 10), ENSURE_REQUIREMENTS, [plan_id=41246]

(23) HashAggregate
Input [3]: [COL_6A7222C5_E1C1_4F61_8065_319A3FBE09A1#788566, COL_E34D9E0C_E565_42B9_AFA1_A780AB15F0DF#788724L, count#789527L]
Keys [2]: [COL_6A7222C5_E1C1_4F61_8065_319A3FBE09A1#788566, COL_E34D9E0C_E565_42B9_AFA1_A780AB15F0DF#788724L]
Functions [1]: [count(1)]
Aggregate Attributes [1]: [count(1)#789148L]
Results [3]: [COL_6A7222C5_E1C1_4F61_8065_319A3FBE09A1#788566 AS COL_B48461D6_A3BC_45F3_AB8F_03BF01D41C93#789383, COL_E34D9E0C_E565_42B9_AFA1_A780AB15F0DF#788724L AS COL_4584C49D_AB64_44A6_9441_F5A5A85410A8#789384L, count(1)#789148L AS COL_30DDCE22_5C87_4039_9224_AD4B5B64084B#789385L]

(24) Project
Output [4]: [COL_B48461D6_A3BC_45F3_AB8F_03BF01D41C93#789383, COL_4584C49D_AB64_44A6_9441_F5A5A85410A8#789384L, COL_30DDCE22_5C87_4039_9224_AD4B5B64084B#789385L, str_concat(array(str_concat(array(bigint_to_str(dec_fix(dec_operator(PRODUIT, array(dec_operator(DIVISION, array(COL_30DDCE22_5C87_4039_9224_AD4B5B64084B#789385L, COL_4584C49D_AB64_44A6_9441_F5A5A85410A8#789384L)), 100.0)))),  , )),  , %)) AS COL_3C8D7FB5_B59E_41C3_9D27_870F51F07F2E#789489]
Input [3]: [COL_B48461D6_A3BC_45F3_AB8F_03BF01D41C93#789383, COL_4584C49D_AB64_44A6_9441_F5A5A85410A8#789384L, COL_30DDCE22_5C87_4039_9224_AD4B5B64084B#789385L]

(25) CollectLimit
Input [4]: [COL_B48461D6_A3BC_45F3_AB8F_03BF01D41C93#789383, COL_4584C49D_AB64_44A6_9441_F5A5A85410A8#789384L, COL_30DDCE22_5C87_4039_9224_AD4B5B64084B#789385L, COL_3C8D7FB5_B59E_41C3_9D27_870F51F07F2E#789489]
Arguments: 500000

(26) AdaptiveSparkPlan
Output [4]: [COL_B48461D6_A3BC_45F3_AB8F_03BF01D41C93#789383, COL_4584C49D_AB64_44A6_9441_F5A5A85410A8#789384L, COL_30DDCE22_5C87_4039_9224_AD4B5B64084B#789385L, COL_3C8D7FB5_B59E_41C3_9D27_870F51F07F2E#789489]
Arguments: isFinalPlan=true

SQL / DataFrame Properties

Name	Value
spark.sql.optimizer.nestedPredicatePushdown.supportedFileSources	parquet,orc,geoparquet