hdh-spark-job - Details for Query 1710

Submitted Time: 2026/09/03 11:15:12
Duration: 0.1 s
Succeeded Jobs: 2163 2164

Show the Stage ID and Task ID that corresponds to the max metric

digraph G { 0 [labelType="html" label=" AdaptiveSparkPlan "]; 1 [labelType="html" label="SortAggregate number of output rows: 1"]; 2 [labelType="html" label="Exchange shuffle records written: 2 local merged chunks fetched: 0 shuffle write time total (min, med, max (stageId: taskId)) 0 ms (0 ms, 0 ms, 0 ms (stage 3166.0: task 25854)) remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 2 local bytes read: 196.0 B fetch wait time: 0 ms remote bytes read: 0.0 B merged fetch fallback count: 0 local blocks read: 2 remote merged chunks fetched: 0 remote blocks read: 0 data size total (min, med, max (stageId: taskId)) 128.0 B (64.0 B, 64.0 B, 64.0 B (stage 3166.0: task 25854)) local merged bytes read: 0.0 B number of partitions: 1 remote reqs duration: 0 ms remote bytes read to disk: 0.0 B shuffle bytes written total (min, med, max (stageId: taskId)) 196.0 B (97.0 B, 99.0 B, 99.0 B (stage 3166.0: task 25855))"]; 3 [labelType="html" label="SortAggregate number of output rows: 2"]; subgraph cluster4 { isCluster="true"; label="WholeStageCodegen (1)\n \nduration: total (min, med, max (stageId: taskId))\n46 ms (23 ms, 23 ms, 23 ms (stage 3166.0: task 25854))"; 5 [labelType="html" label=" Project "]; 6 [labelType="html" label="ColumnarToRow number of output rows: 18,958 number of input batches: 6"]; } 7 [labelType="html" label="Scan parquet number of files read: 2 scan time total (min, med, max (stageId: taskId)) 4 ms (2 ms, 2 ms, 2 ms (stage 3166.0: task 25854)) metadata time: 0 ms size of files read: 213.4 KiB number of output rows: 18,958"]; 1->0; 2->1; 3->2; 5->3; 6->5; 7->6; }

AdaptiveSparkPlan isFinalPlan=true

SortAggregate(key=[], functions=[count(1), count(COL_A3FEC8E3_0B77_4108_9EE7_3A921E109834#445599), min(COL_A3FEC8E3_0B77_4108_9EE7_3A921E109834#445599), max(COL_A3FEC8E3_0B77_4108_9EE7_3A921E109834#445599)])

Exchange SinglePartition, ENSURE_REQUIREMENTS, [plan_id=106356]

SortAggregate(key=[], functions=[partial_count(1), partial_count(COL_A3FEC8E3_0B77_4108_9EE7_3A921E109834#445599), partial_min(COL_A3FEC8E3_0B77_4108_9EE7_3A921E109834#445599), partial_max(COL_A3FEC8E3_0B77_4108_9EE7_3A921E109834#445599)])

Project [Mode+de+transport#445403 AS COL_A3FEC8E3_0B77_4108_9EE7_3A921E109834#445599]

ColumnarToRow

WholeStageCodegen (1)

FileScan parquet [Mode+de+transport#445403] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/data/input/depot/excel/cache/A66DACD4_1DD0_4911_8D5F_C28868EB12B..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<Mode+de+transport:string>

Details

== Physical Plan ==
AdaptiveSparkPlan (12)
+- == Final Plan ==
   SortAggregate (7)
   +- ShuffleQueryStage (6), Statistics(sizeInBytes=128.0 B, rowCount=2)
      +- Exchange (5)
         +- SortAggregate (4)
            +- * Project (3)
               +- * ColumnarToRow (2)
                  +- Scan parquet  (1)
+- == Initial Plan ==
   SortAggregate (11)
   +- Exchange (10)
      +- SortAggregate (9)
         +- Project (8)
            +- Scan parquet  (1)


(1) Scan parquet 
Output [1]: [Mode+de+transport#445403]
Batched: true
Location: InMemoryFileIndex [file:/data/input/depot/excel/cache/A66DACD4_1DD0_4911_8D5F_C28868EB12BC_2d39697772a592ca7c5825321f52ae5d/Passages_urgences_Hexagone.xls_b5cf56a30efce2c88d81adbe37b36dcf]
ReadSchema: struct<Mode+de+transport:string>

(2) ColumnarToRow [codegen id : 1]
Input [1]: [Mode+de+transport#445403]

(3) Project [codegen id : 1]
Output [1]: [Mode+de+transport#445403 AS COL_A3FEC8E3_0B77_4108_9EE7_3A921E109834#445599]
Input [1]: [Mode+de+transport#445403]

(4) SortAggregate
Input [1]: [COL_A3FEC8E3_0B77_4108_9EE7_3A921E109834#445599]
Keys: []
Functions [4]: [partial_count(1), partial_count(COL_A3FEC8E3_0B77_4108_9EE7_3A921E109834#445599), partial_min(COL_A3FEC8E3_0B77_4108_9EE7_3A921E109834#445599), partial_max(COL_A3FEC8E3_0B77_4108_9EE7_3A921E109834#445599)]
Aggregate Attributes [4]: [count#445639L, count#445640L, min#445641, max#445642]
Results [4]: [count#445643L, count#445644L, min#445645, max#445646]

(5) Exchange
Input [4]: [count#445643L, count#445644L, min#445645, max#445646]
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=106356]

(6) ShuffleQueryStage
Output [4]: [count#445643L, count#445644L, min#445645, max#445646]
Arguments: 0

(7) SortAggregate
Input [4]: [count#445643L, count#445644L, min#445645, max#445646]
Keys: []
Functions [4]: [count(1), count(COL_A3FEC8E3_0B77_4108_9EE7_3A921E109834#445599), min(COL_A3FEC8E3_0B77_4108_9EE7_3A921E109834#445599), max(COL_A3FEC8E3_0B77_4108_9EE7_3A921E109834#445599)]
Aggregate Attributes [4]: [count(1)#445611L, count(COL_A3FEC8E3_0B77_4108_9EE7_3A921E109834#445599)#445614L, min(COL_A3FEC8E3_0B77_4108_9EE7_3A921E109834#445599)#445617, max(COL_A3FEC8E3_0B77_4108_9EE7_3A921E109834#445599)#445619]
Results [4]: [count(1)#445611L AS COL_A3FEC8E3_0B77_4108_9EE7_3A921E1098341#445613L, count(COL_A3FEC8E3_0B77_4108_9EE7_3A921E109834#445599)#445614L AS COL_A3FEC8E3_0B77_4108_9EE7_3A921E1098342#445616L, min(COL_A3FEC8E3_0B77_4108_9EE7_3A921E109834#445599)#445617 AS COL_A3FEC8E3_0B77_4108_9EE7_3A921E1098344#445618, max(COL_A3FEC8E3_0B77_4108_9EE7_3A921E109834#445599)#445619 AS COL_A3FEC8E3_0B77_4108_9EE7_3A921E1098345#445620]

(8) Project
Output [1]: [Mode+de+transport#445403 AS COL_A3FEC8E3_0B77_4108_9EE7_3A921E109834#445599]
Input [1]: [Mode+de+transport#445403]

(9) SortAggregate
Input [1]: [COL_A3FEC8E3_0B77_4108_9EE7_3A921E109834#445599]
Keys: []
Functions [4]: [partial_count(1), partial_count(COL_A3FEC8E3_0B77_4108_9EE7_3A921E109834#445599), partial_min(COL_A3FEC8E3_0B77_4108_9EE7_3A921E109834#445599), partial_max(COL_A3FEC8E3_0B77_4108_9EE7_3A921E109834#445599)]
Aggregate Attributes [4]: [count#445639L, count#445640L, min#445641, max#445642]
Results [4]: [count#445643L, count#445644L, min#445645, max#445646]

(10) Exchange
Input [4]: [count#445643L, count#445644L, min#445645, max#445646]
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=106339]

(11) SortAggregate
Input [4]: [count#445643L, count#445644L, min#445645, max#445646]
Keys: []
Functions [4]: [count(1), count(COL_A3FEC8E3_0B77_4108_9EE7_3A921E109834#445599), min(COL_A3FEC8E3_0B77_4108_9EE7_3A921E109834#445599), max(COL_A3FEC8E3_0B77_4108_9EE7_3A921E109834#445599)]
Aggregate Attributes [4]: [count(1)#445611L, count(COL_A3FEC8E3_0B77_4108_9EE7_3A921E109834#445599)#445614L, min(COL_A3FEC8E3_0B77_4108_9EE7_3A921E109834#445599)#445617, max(COL_A3FEC8E3_0B77_4108_9EE7_3A921E109834#445599)#445619]
Results [4]: [count(1)#445611L AS COL_A3FEC8E3_0B77_4108_9EE7_3A921E1098341#445613L, count(COL_A3FEC8E3_0B77_4108_9EE7_3A921E109834#445599)#445614L AS COL_A3FEC8E3_0B77_4108_9EE7_3A921E1098342#445616L, min(COL_A3FEC8E3_0B77_4108_9EE7_3A921E109834#445599)#445617 AS COL_A3FEC8E3_0B77_4108_9EE7_3A921E1098344#445618, max(COL_A3FEC8E3_0B77_4108_9EE7_3A921E109834#445599)#445619 AS COL_A3FEC8E3_0B77_4108_9EE7_3A921E1098345#445620]

(12) AdaptiveSparkPlan
Output [4]: [COL_A3FEC8E3_0B77_4108_9EE7_3A921E1098341#445613L, COL_A3FEC8E3_0B77_4108_9EE7_3A921E1098342#445616L, COL_A3FEC8E3_0B77_4108_9EE7_3A921E1098344#445618, COL_A3FEC8E3_0B77_4108_9EE7_3A921E1098345#445620]
Arguments: isFinalPlan=true

SQL / DataFrame Properties

Name	Value
spark.sql.optimizer.nestedPredicatePushdown.supportedFileSources	parquet,orc,geoparquet