SP1 - Details for Query 1011

Submitted Time: 2026/02/24 11:03:46
Duration: 0.6 s
Succeeded Jobs: 1373 1374 1375

Show the Stage ID and Task ID that corresponds to the max metric

digraph G { 0 [labelType="html" label=" AdaptiveSparkPlan "]; 1 [labelType="html" label=" CollectLimit "]; 2 [labelType="html" label=" Union "]; subgraph cluster3 { isCluster="true"; label="WholeStageCodegen (2)\n \nduration: 9 ms"; 4 [labelType="html" label=" LocalLimit "]; 5 [labelType="html" label=" Project "]; 6 [labelType="html" label="Filter number of output rows: 0"]; } 7 [labelType="html" label="Scan csv number of output rows: 0 number of files read: 1 metadata time: 0 ms size of files read: 2.0 B"]; subgraph cluster8 { isCluster="true"; label="WholeStageCodegen (6)\n \nduration: 13 ms"; 9 [labelType="html" label=" LocalLimit "]; 10 [labelType="html" label=" Project "]; } 11 [labelType="html" label="CartesianProduct number of output rows: 0"]; subgraph cluster12 { isCluster="true"; label="WholeStageCodegen (4)\n \nduration: 46 ms"; 13 [labelType="html" label="HashAggregate spill size: 0.0 B time in aggregation build: 6 ms peak memory: 256.0 KiB number of output rows: 1 number of sort fallback tasks: 0 avg hash probes per key: 0"]; 14 [labelType="html" label="HashAggregate spill size: 0.0 B time in aggregation build: 0 ms peak memory: 256.0 KiB number of output rows: 1 number of sort fallback tasks: 0 avg hash probes per key: 0"]; 15 [labelType="html" label=" Project "]; 16 [labelType="html" label="Filter number of output rows: 3"]; } 17 [labelType="html" label="Window spill size: 0.0 B"]; subgraph cluster18 { isCluster="true"; label="WholeStageCodegen (3)\n \nduration: 124 ms"; 19 [labelType="html" label="Sort sort time: 0 ms peak memory: 16.1 MiB spill size: 0.0 B"]; 20 [labelType="html" label=" GlobalLimit "]; } 21 [labelType="html" label="Exchange shuffle records written: 15 local merged chunks fetched: 0 shuffle write time total (min, med, max (stageId: taskId)) 1 ms (0 ms, 0 ms, 0 ms (stage 2129.0: task 2540)) remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 3 local bytes read: 1688.0 B fetch wait time: 0 ms remote bytes read: 0.0 B merged fetch fallback count: 0 local blocks read: 5 remote merged chunks fetched: 0 remote blocks read: 0 data size total (min, med, max (stageId: taskId)) 3.2 KiB (528.0 B, 672.0 B, 840.0 B (stage 2129.0: task 2538)) local merged bytes read: 0.0 B number of partitions: 1 remote reqs duration: 0 ms remote bytes read to disk: 0.0 B shuffle bytes written total (min, med, max (stageId: taskId)) 1688.0 B (237.0 B, 342.0 B, 399.0 B (stage 2129.0: task 2538))"]; subgraph cluster22 { isCluster="true"; label="WholeStageCodegen (1)\n \nduration: total (min, med, max (stageId: taskId))\n120 ms (23 ms, 24 ms, 26 ms (stage 2129.0: task 2539))"; 23 [labelType="html" label=" LocalLimit "]; 24 [labelType="html" label=" Project "]; } 25 [labelType="html" label="Scan csv number of output rows: 15 number of files read: 1 metadata time: 0 ms size of files read: 18.1 MiB"]; subgraph cluster26 { isCluster="true"; label="WholeStageCodegen (5)\n \nduration: 4 ms"; 27 [labelType="html" label=" Project "]; } 28 [labelType="html" label="Scan csv number of output rows: 1 number of files read: 1 metadata time: 0 ms size of files read: 62.0 B"]; 1->0; 2->1; 4->2; 5->4; 6->5; 7->6; 9->2; 10->9; 11->10; 13->11; 14->13; 15->14; 16->15; 17->16; 19->17; 20->19; 21->20; 23->21; 24->23; 25->24; 27->11; 28->27; }

AdaptiveSparkPlan isFinalPlan=true

CollectLimit 500000

Union

LocalLimit 500000

Project [c0#316546 AS COL_E0877E55_D201_41B0_A882_1AF302053AF9#316778]

Filter isnotnull(c0#316546)

WholeStageCodegen (2)

FileScan csv [c0#316546] Batched: false, DataFilters: [isnotnull(c0#316546)], Format: CSV, Location: InMemoryFileIndex(1 paths)[file:/data/input/depot/csv/execution/empty.csv], PartitionFilters: [], PushedFilters: [IsNotNull(c0)], ReadSchema: struct<c0:string>

LocalLimit 500000

Project [str_if_with_rule((INFO_DATAAPPS_346034_COL_A9C3D939_A6C7_4DC2_8CC2_91076A4A2CFC#316742 <=> COL_467B8EF8_7CFB_46F2_8CD4_3805AEA1E17D#316738), null, Incohérence entre le FINESS saisi et le FINESS du fichier RSF) AS COL_42952156_A6E1_4158_96AE_E70CF0ED9DCE#316760]

WholeStageCodegen (6)

CartesianProduct isnotnull(str_if_with_rule((INFO_DATAAPPS_346034_COL_A9C3D939_A6C7_4DC2_8CC2_91076A4A2CFC#316742 <=> COL_467B8EF8_7CFB_46F2_8CD4_3805AEA1E17D#316738), null, Incohérence entre le FINESS saisi et le FINESS du fichier RSF))

HashAggregate(keys=[COL_CD1100A9_9106_47C2_93C6_E2EF19A395A4#316700], functions=[])

Project [str_extract_middle(COL_0B816CAF_81A1_4E69_B9E3_E6597EABCC0D#316666, 2, 9) AS COL_CD1100A9_9106_47C2_93C6_E2EF19A395A4#316700]

Filter (((COL_466B8114_A534_4700_99C5_61D90974826A#316675 <=> 1) AND (str_extract(COL_0B816CAF_81A1_4E69_B9E3_E6597EABCC0D#316666, LEFT, 1) <=> A)) OR (COL_466B8114_A534_4700_99C5_61D90974826A#316675 > 1))

WholeStageCodegen (4)

Window [row_number() windowspecdefinition(1, 1 DESC NULLS LAST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS COL_466B8114_A534_4700_99C5_61D90974826A#316675], [1], [1 DESC NULLS LAST]

Sort [1 ASC NULLS FIRST, 1 DESC NULLS LAST], false, 0

GlobalLimit 3, 0

WholeStageCodegen (3)

Exchange SinglePartition, ENSURE_REQUIREMENTS, [plan_id=39199]

LocalLimit 3

Project [_c0#316639 AS COL_0B816CAF_81A1_4E69_B9E3_E6597EABCC0D#316666]

WholeStageCodegen (1)

FileScan csv [_c0#316639] Batched: false, DataFilters: [], Format: CSV, Location: InMemoryFileIndex(1 paths)[file:/data/input/depot/csv/tmp/RSF_SOURCE_SERV_345910/Input_RSF_YYYYYY..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<_c0:string>

Project [str_create(finess_dataapps#316574) AS INFO_DATAAPPS_346034_COL_A9C3D939_A6C7_4DC2_8CC2_91076A4A2CFC#316742]

WholeStageCodegen (5)

FileScan csv [finess_dataapps#316574] Batched: false, DataFilters: [], Format: CSV, Location: InMemoryFileIndex(1 paths)[file:/data/input/depot/csv/execution/INFOETAB_DATAAP_345911/current/in..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<finess_dataapps:string>

Details

== Physical Plan ==
AdaptiveSparkPlan (43)
+- == Final Plan ==
   CollectLimit (23)
   +- Union (22)
      :- * LocalLimit (4)
      :  +- * Project (3)
      :     +- * Filter (2)
      :        +- Scan csv  (1)
      +- * LocalLimit (21)
         +- * Project (20)
            +- CartesianProduct Inner (19)
               :- * HashAggregate (16)
               :  +- * HashAggregate (15)
               :     +- * Project (14)
               :        +- * Filter (13)
               :           +- Window (12)
               :              +- * Sort (11)
               :                 +- * GlobalLimit (10)
               :                    +- ShuffleQueryStage (9), Statistics(sizeInBytes=3.2 KiB, rowCount=15)
               :                       +- Exchange (8)
               :                          +- * LocalLimit (7)
               :                             +- * Project (6)
               :                                +- Scan csv  (5)
               +- * Project (18)
                  +- Scan csv  (17)
+- == Initial Plan ==
   CollectLimit (42)
   +- Union (41)
      :- LocalLimit (26)
      :  +- Project (25)
      :     +- Filter (24)
      :        +- Scan csv  (1)
      +- LocalLimit (40)
         +- Project (39)
            +- CartesianProduct Inner (38)
               :- HashAggregate (36)
               :  +- HashAggregate (35)
               :     +- Project (34)
               :        +- Filter (33)
               :           +- Window (32)
               :              +- Sort (31)
               :                 +- GlobalLimit (30)
               :                    +- Exchange (29)
               :                       +- LocalLimit (28)
               :                          +- Project (27)
               :                             +- Scan csv  (5)
               +- Project (37)
                  +- Scan csv  (17)


(1) Scan csv 
Output [1]: [c0#316546]
Batched: false
Location: InMemoryFileIndex [file:/data/input/depot/csv/execution/empty.csv]
PushedFilters: [IsNotNull(c0)]
ReadSchema: struct<c0:string>

(2) Filter [codegen id : 2]
Input [1]: [c0#316546]
Condition : isnotnull(c0#316546)

(3) Project [codegen id : 2]
Output [1]: [c0#316546 AS COL_E0877E55_D201_41B0_A882_1AF302053AF9#316778]
Input [1]: [c0#316546]

(4) LocalLimit [codegen id : 2]
Input [1]: [COL_E0877E55_D201_41B0_A882_1AF302053AF9#316778]
Arguments: 500000

(5) Scan csv 
Output [1]: [_c0#316639]
Batched: false
Location: InMemoryFileIndex [file:/data/input/depot/csv/tmp/RSF_SOURCE_SERV_345910/Input_RSF_YYYYYYYYY.txt]
ReadSchema: struct<_c0:string>

(6) Project [codegen id : 1]
Output [1]: [_c0#316639 AS COL_0B816CAF_81A1_4E69_B9E3_E6597EABCC0D#316666]
Input [1]: [_c0#316639]

(7) LocalLimit [codegen id : 1]
Input [1]: [COL_0B816CAF_81A1_4E69_B9E3_E6597EABCC0D#316666]
Arguments: 3

(8) Exchange
Input [1]: [COL_0B816CAF_81A1_4E69_B9E3_E6597EABCC0D#316666]
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=39199]

(9) ShuffleQueryStage
Output [1]: [COL_0B816CAF_81A1_4E69_B9E3_E6597EABCC0D#316666]
Arguments: 0

(10) GlobalLimit [codegen id : 3]
Input [1]: [COL_0B816CAF_81A1_4E69_B9E3_E6597EABCC0D#316666]
Arguments: 3, 0

(11) Sort [codegen id : 3]
Input [1]: [COL_0B816CAF_81A1_4E69_B9E3_E6597EABCC0D#316666]
Arguments: [1 ASC NULLS FIRST, 1 DESC NULLS LAST], false, 0

(12) Window
Input [1]: [COL_0B816CAF_81A1_4E69_B9E3_E6597EABCC0D#316666]
Arguments: [row_number() windowspecdefinition(1, 1 DESC NULLS LAST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS COL_466B8114_A534_4700_99C5_61D90974826A#316675], [1], [1 DESC NULLS LAST]

(13) Filter [codegen id : 4]
Input [2]: [COL_0B816CAF_81A1_4E69_B9E3_E6597EABCC0D#316666, COL_466B8114_A534_4700_99C5_61D90974826A#316675]
Condition : (((COL_466B8114_A534_4700_99C5_61D90974826A#316675 <=> 1) AND (str_extract(COL_0B816CAF_81A1_4E69_B9E3_E6597EABCC0D#316666, LEFT, 1) <=> A)) OR (COL_466B8114_A534_4700_99C5_61D90974826A#316675 > 1))

(14) Project [codegen id : 4]
Output [1]: [str_extract_middle(COL_0B816CAF_81A1_4E69_B9E3_E6597EABCC0D#316666, 2, 9) AS COL_CD1100A9_9106_47C2_93C6_E2EF19A395A4#316700]
Input [2]: [COL_0B816CAF_81A1_4E69_B9E3_E6597EABCC0D#316666, COL_466B8114_A534_4700_99C5_61D90974826A#316675]

(15) HashAggregate [codegen id : 4]
Input [1]: [COL_CD1100A9_9106_47C2_93C6_E2EF19A395A4#316700]
Keys [1]: [COL_CD1100A9_9106_47C2_93C6_E2EF19A395A4#316700]
Functions: []
Aggregate Attributes: []
Results [1]: [COL_CD1100A9_9106_47C2_93C6_E2EF19A395A4#316700]

(16) HashAggregate [codegen id : 4]
Input [1]: [COL_CD1100A9_9106_47C2_93C6_E2EF19A395A4#316700]
Keys [1]: [COL_CD1100A9_9106_47C2_93C6_E2EF19A395A4#316700]
Functions: []
Aggregate Attributes: []
Results [1]: [COL_CD1100A9_9106_47C2_93C6_E2EF19A395A4#316700 AS COL_467B8EF8_7CFB_46F2_8CD4_3805AEA1E17D#316738]

(17) Scan csv 
Output [1]: [finess_dataapps#316574]
Batched: false
Location: InMemoryFileIndex [file:/data/input/depot/csv/execution/INFOETAB_DATAAP_345911/current/info_dataapps.csv]
ReadSchema: struct<finess_dataapps:string>

(18) Project [codegen id : 5]
Output [1]: [str_create(finess_dataapps#316574) AS INFO_DATAAPPS_346034_COL_A9C3D939_A6C7_4DC2_8CC2_91076A4A2CFC#316742]
Input [1]: [finess_dataapps#316574]

(19) CartesianProduct
Join type: Inner
Join condition: isnotnull(str_if_with_rule((INFO_DATAAPPS_346034_COL_A9C3D939_A6C7_4DC2_8CC2_91076A4A2CFC#316742 <=> COL_467B8EF8_7CFB_46F2_8CD4_3805AEA1E17D#316738), null, Incohérence entre le FINESS saisi et le FINESS du fichier RSF))

(20) Project [codegen id : 6]
Output [1]: [str_if_with_rule((INFO_DATAAPPS_346034_COL_A9C3D939_A6C7_4DC2_8CC2_91076A4A2CFC#316742 <=> COL_467B8EF8_7CFB_46F2_8CD4_3805AEA1E17D#316738), null, Incohérence entre le FINESS saisi et le FINESS du fichier RSF) AS COL_42952156_A6E1_4158_96AE_E70CF0ED9DCE#316760]
Input [2]: [COL_467B8EF8_7CFB_46F2_8CD4_3805AEA1E17D#316738, INFO_DATAAPPS_346034_COL_A9C3D939_A6C7_4DC2_8CC2_91076A4A2CFC#316742]

(21) LocalLimit [codegen id : 6]
Input [1]: [COL_42952156_A6E1_4158_96AE_E70CF0ED9DCE#316760]
Arguments: 500000

(22) Union

(23) CollectLimit
Input [1]: [COL_E0877E55_D201_41B0_A882_1AF302053AF9#316778]
Arguments: 500000

(24) Filter
Input [1]: [c0#316546]
Condition : isnotnull(c0#316546)

(25) Project
Output [1]: [c0#316546 AS COL_E0877E55_D201_41B0_A882_1AF302053AF9#316778]
Input [1]: [c0#316546]

(26) LocalLimit
Input [1]: [COL_E0877E55_D201_41B0_A882_1AF302053AF9#316778]
Arguments: 500000

(27) Project
Output [1]: [_c0#316639 AS COL_0B816CAF_81A1_4E69_B9E3_E6597EABCC0D#316666]
Input [1]: [_c0#316639]

(28) LocalLimit
Input [1]: [COL_0B816CAF_81A1_4E69_B9E3_E6597EABCC0D#316666]
Arguments: 3

(29) Exchange
Input [1]: [COL_0B816CAF_81A1_4E69_B9E3_E6597EABCC0D#316666]
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=39176]

(30) GlobalLimit
Input [1]: [COL_0B816CAF_81A1_4E69_B9E3_E6597EABCC0D#316666]
Arguments: 3, 0

(31) Sort
Input [1]: [COL_0B816CAF_81A1_4E69_B9E3_E6597EABCC0D#316666]
Arguments: [1 ASC NULLS FIRST, 1 DESC NULLS LAST], false, 0

(32) Window
Input [1]: [COL_0B816CAF_81A1_4E69_B9E3_E6597EABCC0D#316666]
Arguments: [row_number() windowspecdefinition(1, 1 DESC NULLS LAST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS COL_466B8114_A534_4700_99C5_61D90974826A#316675], [1], [1 DESC NULLS LAST]

(33) Filter
Input [2]: [COL_0B816CAF_81A1_4E69_B9E3_E6597EABCC0D#316666, COL_466B8114_A534_4700_99C5_61D90974826A#316675]
Condition : (((COL_466B8114_A534_4700_99C5_61D90974826A#316675 <=> 1) AND (str_extract(COL_0B816CAF_81A1_4E69_B9E3_E6597EABCC0D#316666, LEFT, 1) <=> A)) OR (COL_466B8114_A534_4700_99C5_61D90974826A#316675 > 1))

(34) Project
Output [1]: [str_extract_middle(COL_0B816CAF_81A1_4E69_B9E3_E6597EABCC0D#316666, 2, 9) AS COL_CD1100A9_9106_47C2_93C6_E2EF19A395A4#316700]
Input [2]: [COL_0B816CAF_81A1_4E69_B9E3_E6597EABCC0D#316666, COL_466B8114_A534_4700_99C5_61D90974826A#316675]

(35) HashAggregate
Input [1]: [COL_CD1100A9_9106_47C2_93C6_E2EF19A395A4#316700]
Keys [1]: [COL_CD1100A9_9106_47C2_93C6_E2EF19A395A4#316700]
Functions: []
Aggregate Attributes: []
Results [1]: [COL_CD1100A9_9106_47C2_93C6_E2EF19A395A4#316700]

(36) HashAggregate
Input [1]: [COL_CD1100A9_9106_47C2_93C6_E2EF19A395A4#316700]
Keys [1]: [COL_CD1100A9_9106_47C2_93C6_E2EF19A395A4#316700]
Functions: []
Aggregate Attributes: []
Results [1]: [COL_CD1100A9_9106_47C2_93C6_E2EF19A395A4#316700 AS COL_467B8EF8_7CFB_46F2_8CD4_3805AEA1E17D#316738]

(37) Project
Output [1]: [str_create(finess_dataapps#316574) AS INFO_DATAAPPS_346034_COL_A9C3D939_A6C7_4DC2_8CC2_91076A4A2CFC#316742]
Input [1]: [finess_dataapps#316574]

(38) CartesianProduct
Join type: Inner
Join condition: isnotnull(str_if_with_rule((INFO_DATAAPPS_346034_COL_A9C3D939_A6C7_4DC2_8CC2_91076A4A2CFC#316742 <=> COL_467B8EF8_7CFB_46F2_8CD4_3805AEA1E17D#316738), null, Incohérence entre le FINESS saisi et le FINESS du fichier RSF))

(39) Project
Output [1]: [str_if_with_rule((INFO_DATAAPPS_346034_COL_A9C3D939_A6C7_4DC2_8CC2_91076A4A2CFC#316742 <=> COL_467B8EF8_7CFB_46F2_8CD4_3805AEA1E17D#316738), null, Incohérence entre le FINESS saisi et le FINESS du fichier RSF) AS COL_42952156_A6E1_4158_96AE_E70CF0ED9DCE#316760]
Input [2]: [COL_467B8EF8_7CFB_46F2_8CD4_3805AEA1E17D#316738, INFO_DATAAPPS_346034_COL_A9C3D939_A6C7_4DC2_8CC2_91076A4A2CFC#316742]

(40) LocalLimit
Input [1]: [COL_42952156_A6E1_4158_96AE_E70CF0ED9DCE#316760]
Arguments: 500000

(41) Union

(42) CollectLimit
Input [1]: [COL_E0877E55_D201_41B0_A882_1AF302053AF9#316778]
Arguments: 500000

(43) AdaptiveSparkPlan
Output [1]: [COL_E0877E55_D201_41B0_A882_1AF302053AF9#316778]
Arguments: isFinalPlan=true

SQL / DataFrame Properties

Name	Value
spark.sql.optimizer.nestedPredicatePushdown.supportedFileSources	parquet,orc,geoparquet