SP1 - Details for Query 295

Submitted Time: 2026/02/20 09:26:12
Duration: 3 s
Succeeded Jobs: 426 435

Show the Stage ID and Task ID that corresponds to the max metric

digraph G { 0 [labelType="html" label=" AdaptiveSparkPlan "]; subgraph cluster1 { isCluster="true"; label="WholeStageCodegen (2)\n \nduration: 0 ms"; 2 [labelType="html" label="HashAggregate time in aggregation build: 0 ms number of output rows: 1"]; } 3 [labelType="html" label="Exchange shuffle records written: 1 local merged chunks fetched: 0 shuffle write time: 0 ms remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 1 local bytes read: 59.0 B fetch wait time: 0 ms remote bytes read: 0.0 B merged fetch fallback count: 0 local blocks read: 1 remote merged chunks fetched: 0 remote blocks read: 0 data size: 16.0 B local merged bytes read: 0.0 B number of partitions: 1 remote reqs duration: 0 ms remote bytes read to disk: 0.0 B shuffle bytes written: 59.0 B"]; subgraph cluster4 { isCluster="true"; label="WholeStageCodegen (1)\n \nduration: 1.7 s"; 5 [labelType="html" label="HashAggregate time in aggregation build: 1.7 s number of output rows: 1"]; 6 [labelType="html" label=" Project "]; 7 [labelType="html" label="Filter number of output rows: 2,479"]; 8 [labelType="html" label="ColumnarToRow number of output rows: 39,201 number of input batches: 10"]; } 9 [labelType="html" label="Scan parquet number of files read: 1 scan time: 70 ms metadata time: 0 ms size of files read: 2.7 MiB number of output rows: 39,201"]; 2->0; 3->2; 5->3; 6->5; 7->6; 8->7; 9->8; }

AdaptiveSparkPlan isFinalPlan=true

HashAggregate(keys=[], functions=[count(1)])

WholeStageCodegen (2)

Exchange SinglePartition, ENSURE_REQUIREMENTS, [plan_id=8624]

HashAggregate(keys=[], functions=[partial_count(1)])

Project

Filter (((((((((CASE WHEN isnull(Contains(str_remove_accent(upper(CODE_COMMUNE_INSEE_16#118025)), ME)) THEN false ELSE Contains(str_remove_accent(upper(CODE_COMMUNE_INSEE_16#118025)), ME) END OR CASE WHEN isnull(Contains(str_remove_accent(upper(NOM_COMMUNE_17#118031)), ME)) THEN false ELSE Contains(str_remove_accent(upper(NOM_COMMUNE_17#118031)), ME) END) OR CASE WHEN isnull(Contains(str_remove_accent(upper(CODE_POSTAL_18#118026)), ME)) THEN false ELSE Contains(str_remove_accent(upper(CODE_POSTAL_18#118026)), ME) END) OR CASE WHEN isnull(Contains(str_remove_accent(upper(LIBELLE_ACHEMINEMENT_19#118029)), ME)) THEN false ELSE Contains(str_remove_accent(upper(LIBELLE_ACHEMINEMENT_19#118029)), ME) END) OR CASE WHEN isnull(Contains(str_remove_accent(upper(LIGNE_5_20#118030)), ME)) THEN false ELSE Contains(str_remove_accent(upper(LIGNE_5_20#118030)), ME) END) OR CASE WHEN isnull(Contains(str_remove_accent(upper(COORDONNEES_GPS_21#118027)), ME)) THEN false ELSE Contains(str_remove_accent(upper(COORDONNEES_GPS_21#118027)), ME) END) OR CASE WHEN isnull(Contains(str_remove_accent(upper(DETAILS_DEPOT_22#118028)), ME)) THEN false ELSE Contains(str_remove_accent(upper(DETAILS_DEPOT_22#118028)), ME) END) OR CASE WHEN isnull(Contains(str_remove_accent(upper(CODE_DEPARTEMENT_23#118032)), ME)) THEN false ELSE Contains(str_remove_accent(upper(CODE_DEPARTEMENT_23#118032)), ME) END) OR CASE WHEN isnull(Contains(str_remove_accent(upper(LATITUDE_24#118033)), ME)) THEN false ELSE Contains(str_remove_accent(upper(LATITUDE_24#118033)), ME) END) OR CASE WHEN isnull(Contains(str_remove_accent(upper(LONGITUDE_25#118034)), ME)) THEN false ELSE Contains(str_remove_accent(upper(LONGITUDE_25#118034)), ME) END)

ColumnarToRow

WholeStageCodegen (1)

FileScan parquet [CODE_COMMUNE_INSEE_16#118025,CODE_POSTAL_18#118026,COORDONNEES_GPS_21#118027,DETAILS_DEPOT_22#118028,LIBELLE_ACHEMINEMENT_19#118029,LIGNE_5_20#118030,NOM_COMMUNE_17#118031,CODE_DEPARTEMENT_23#118032,LATITUDE_24#118033,LONGITUDE_25#118034] Batched: true, DataFilters: [(((((((((CASE WHEN isnull(Contains(str_remove_accent(upper(CODE_COMMUNE_INSEE_16#118025)), ME)) ..., Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/data/output/cache/parquet/uet/COMMUNE_FRANCE_REFER_15], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<CODE_COMMUNE_INSEE_16:string,CODE_POSTAL_18:string,COORDONNEES_GPS_21:string,DETAILS_DEPOT...

Details

== Physical Plan ==
AdaptiveSparkPlan (14)
+- == Final Plan ==
   * HashAggregate (8)
   +- ShuffleQueryStage (7), Statistics(sizeInBytes=16.0 B, rowCount=1)
      +- Exchange (6)
         +- * HashAggregate (5)
            +- * Project (4)
               +- * Filter (3)
                  +- * ColumnarToRow (2)
                     +- Scan parquet  (1)
+- == Initial Plan ==
   HashAggregate (13)
   +- Exchange (12)
      +- HashAggregate (11)
         +- Project (10)
            +- Filter (9)
               +- Scan parquet  (1)


(1) Scan parquet 
Output [10]: [CODE_COMMUNE_INSEE_16#118025, CODE_POSTAL_18#118026, COORDONNEES_GPS_21#118027, DETAILS_DEPOT_22#118028, LIBELLE_ACHEMINEMENT_19#118029, LIGNE_5_20#118030, NOM_COMMUNE_17#118031, CODE_DEPARTEMENT_23#118032, LATITUDE_24#118033, LONGITUDE_25#118034]
Batched: true
Location: InMemoryFileIndex [file:/data/output/cache/parquet/uet/COMMUNE_FRANCE_REFER_15]
ReadSchema: struct<CODE_COMMUNE_INSEE_16:string,CODE_POSTAL_18:string,COORDONNEES_GPS_21:string,DETAILS_DEPOT_22:string,LIBELLE_ACHEMINEMENT_19:string,LIGNE_5_20:string,NOM_COMMUNE_17:string,CODE_DEPARTEMENT_23:string,LATITUDE_24:string,LONGITUDE_25:string>

(2) ColumnarToRow [codegen id : 1]
Input [10]: [CODE_COMMUNE_INSEE_16#118025, CODE_POSTAL_18#118026, COORDONNEES_GPS_21#118027, DETAILS_DEPOT_22#118028, LIBELLE_ACHEMINEMENT_19#118029, LIGNE_5_20#118030, NOM_COMMUNE_17#118031, CODE_DEPARTEMENT_23#118032, LATITUDE_24#118033, LONGITUDE_25#118034]

(3) Filter [codegen id : 1]
Input [10]: [CODE_COMMUNE_INSEE_16#118025, CODE_POSTAL_18#118026, COORDONNEES_GPS_21#118027, DETAILS_DEPOT_22#118028, LIBELLE_ACHEMINEMENT_19#118029, LIGNE_5_20#118030, NOM_COMMUNE_17#118031, CODE_DEPARTEMENT_23#118032, LATITUDE_24#118033, LONGITUDE_25#118034]
Condition : (((((((((CASE WHEN isnull(Contains(str_remove_accent(upper(CODE_COMMUNE_INSEE_16#118025)), ME)) THEN false ELSE Contains(str_remove_accent(upper(CODE_COMMUNE_INSEE_16#118025)), ME) END OR CASE WHEN isnull(Contains(str_remove_accent(upper(NOM_COMMUNE_17#118031)), ME)) THEN false ELSE Contains(str_remove_accent(upper(NOM_COMMUNE_17#118031)), ME) END) OR CASE WHEN isnull(Contains(str_remove_accent(upper(CODE_POSTAL_18#118026)), ME)) THEN false ELSE Contains(str_remove_accent(upper(CODE_POSTAL_18#118026)), ME) END) OR CASE WHEN isnull(Contains(str_remove_accent(upper(LIBELLE_ACHEMINEMENT_19#118029)), ME)) THEN false ELSE Contains(str_remove_accent(upper(LIBELLE_ACHEMINEMENT_19#118029)), ME) END) OR CASE WHEN isnull(Contains(str_remove_accent(upper(LIGNE_5_20#118030)), ME)) THEN false ELSE Contains(str_remove_accent(upper(LIGNE_5_20#118030)), ME) END) OR CASE WHEN isnull(Contains(str_remove_accent(upper(COORDONNEES_GPS_21#118027)), ME)) THEN false ELSE Contains(str_remove_accent(upper(COORDONNEES_GPS_21#118027)), ME) END) OR CASE WHEN isnull(Contains(str_remove_accent(upper(DETAILS_DEPOT_22#118028)), ME)) THEN false ELSE Contains(str_remove_accent(upper(DETAILS_DEPOT_22#118028)), ME) END) OR CASE WHEN isnull(Contains(str_remove_accent(upper(CODE_DEPARTEMENT_23#118032)), ME)) THEN false ELSE Contains(str_remove_accent(upper(CODE_DEPARTEMENT_23#118032)), ME) END) OR CASE WHEN isnull(Contains(str_remove_accent(upper(LATITUDE_24#118033)), ME)) THEN false ELSE Contains(str_remove_accent(upper(LATITUDE_24#118033)), ME) END) OR CASE WHEN isnull(Contains(str_remove_accent(upper(LONGITUDE_25#118034)), ME)) THEN false ELSE Contains(str_remove_accent(upper(LONGITUDE_25#118034)), ME) END)

(4) Project [codegen id : 1]
Output: []
Input [10]: [CODE_COMMUNE_INSEE_16#118025, CODE_POSTAL_18#118026, COORDONNEES_GPS_21#118027, DETAILS_DEPOT_22#118028, LIBELLE_ACHEMINEMENT_19#118029, LIGNE_5_20#118030, NOM_COMMUNE_17#118031, CODE_DEPARTEMENT_23#118032, LATITUDE_24#118033, LONGITUDE_25#118034]

(5) HashAggregate [codegen id : 1]
Input: []
Keys: []
Functions [1]: [partial_count(1)]
Aggregate Attributes [1]: [count#118567L]
Results [1]: [count#118568L]

(6) Exchange
Input [1]: [count#118568L]
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=8624]

(7) ShuffleQueryStage
Output [1]: [count#118568L]
Arguments: 0

(8) HashAggregate [codegen id : 2]
Input [1]: [count#118568L]
Keys: []
Functions [1]: [count(1)]
Aggregate Attributes [1]: [count(1)#118137L]
Results [1]: [count(1)#118137L AS COL_93fa355d-255b-4dc5-a07a-2796caea35bc#118139L]

(9) Filter
Input [10]: [CODE_COMMUNE_INSEE_16#118025, CODE_POSTAL_18#118026, COORDONNEES_GPS_21#118027, DETAILS_DEPOT_22#118028, LIBELLE_ACHEMINEMENT_19#118029, LIGNE_5_20#118030, NOM_COMMUNE_17#118031, CODE_DEPARTEMENT_23#118032, LATITUDE_24#118033, LONGITUDE_25#118034]
Condition : (((((((((CASE WHEN isnull(Contains(str_remove_accent(upper(CODE_COMMUNE_INSEE_16#118025)), ME)) THEN false ELSE Contains(str_remove_accent(upper(CODE_COMMUNE_INSEE_16#118025)), ME) END OR CASE WHEN isnull(Contains(str_remove_accent(upper(NOM_COMMUNE_17#118031)), ME)) THEN false ELSE Contains(str_remove_accent(upper(NOM_COMMUNE_17#118031)), ME) END) OR CASE WHEN isnull(Contains(str_remove_accent(upper(CODE_POSTAL_18#118026)), ME)) THEN false ELSE Contains(str_remove_accent(upper(CODE_POSTAL_18#118026)), ME) END) OR CASE WHEN isnull(Contains(str_remove_accent(upper(LIBELLE_ACHEMINEMENT_19#118029)), ME)) THEN false ELSE Contains(str_remove_accent(upper(LIBELLE_ACHEMINEMENT_19#118029)), ME) END) OR CASE WHEN isnull(Contains(str_remove_accent(upper(LIGNE_5_20#118030)), ME)) THEN false ELSE Contains(str_remove_accent(upper(LIGNE_5_20#118030)), ME) END) OR CASE WHEN isnull(Contains(str_remove_accent(upper(COORDONNEES_GPS_21#118027)), ME)) THEN false ELSE Contains(str_remove_accent(upper(COORDONNEES_GPS_21#118027)), ME) END) OR CASE WHEN isnull(Contains(str_remove_accent(upper(DETAILS_DEPOT_22#118028)), ME)) THEN false ELSE Contains(str_remove_accent(upper(DETAILS_DEPOT_22#118028)), ME) END) OR CASE WHEN isnull(Contains(str_remove_accent(upper(CODE_DEPARTEMENT_23#118032)), ME)) THEN false ELSE Contains(str_remove_accent(upper(CODE_DEPARTEMENT_23#118032)), ME) END) OR CASE WHEN isnull(Contains(str_remove_accent(upper(LATITUDE_24#118033)), ME)) THEN false ELSE Contains(str_remove_accent(upper(LATITUDE_24#118033)), ME) END) OR CASE WHEN isnull(Contains(str_remove_accent(upper(LONGITUDE_25#118034)), ME)) THEN false ELSE Contains(str_remove_accent(upper(LONGITUDE_25#118034)), ME) END)

(10) Project
Output: []
Input [10]: [CODE_COMMUNE_INSEE_16#118025, CODE_POSTAL_18#118026, COORDONNEES_GPS_21#118027, DETAILS_DEPOT_22#118028, LIBELLE_ACHEMINEMENT_19#118029, LIGNE_5_20#118030, NOM_COMMUNE_17#118031, CODE_DEPARTEMENT_23#118032, LATITUDE_24#118033, LONGITUDE_25#118034]

(11) HashAggregate
Input: []
Keys: []
Functions [1]: [partial_count(1)]
Aggregate Attributes [1]: [count#118567L]
Results [1]: [count#118568L]

(12) Exchange
Input [1]: [count#118568L]
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=8601]

(13) HashAggregate
Input [1]: [count#118568L]
Keys: []
Functions [1]: [count(1)]
Aggregate Attributes [1]: [count(1)#118137L]
Results [1]: [count(1)#118137L AS COL_93fa355d-255b-4dc5-a07a-2796caea35bc#118139L]

(14) AdaptiveSparkPlan
Output [1]: [COL_93fa355d-255b-4dc5-a07a-2796caea35bc#118139L]
Arguments: isFinalPlan=true

SQL / DataFrame Properties

Name	Value
spark.sql.optimizer.nestedPredicatePushdown.supportedFileSources	parquet,orc,geoparquet