hdh-spark-job - Details for Query 1819

Submitted Time: 2026/09/03 12:15:42
Duration: 0.4 s
Succeeded Jobs: 2268 2269 2270

Show the Stage ID and Task ID that corresponds to the max metric

digraph G { 0 [labelType="html" label=" AdaptiveSparkPlan "]; 1 [labelType="html" label=" CollectLimit "]; subgraph cluster2 { isCluster="true"; label="WholeStageCodegen (5)\n \nduration: 0 ms"; 3 [labelType="html" label=" Project "]; 4 [labelType="html" label="SortMergeJoin number of output rows: 1,500 spill size: 0.0 B"]; } subgraph cluster5 { isCluster="true"; label="WholeStageCodegen (3)\n \nduration: 0 ms"; 6 [labelType="html" label="Sort sort time: 4 ms peak memory: 64.5 MiB spill size: 0.0 B"]; } 7 [labelType="html" label="AQEShuffleRead number of partitions: 1 partition data size: 171.0 KiB number of coalesced partitions: 1"]; 8 [labelType="html" label="Exchange shuffle records written: 12,000 local merged chunks fetched: 0 shuffle write time total (min, med, max (stageId: taskId)) 23 ms (1 ms, 3 ms, 7 ms (stage 3300.0: task 26059)) remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 12,000 local bytes read: 162.5 KiB fetch wait time: 0 ms remote bytes read: 0.0 B merged fetch fallback count: 0 local blocks read: 8 remote merged chunks fetched: 0 remote blocks read: 0 data size total (min, med, max (stageId: taskId)) 1312.5 KiB (164.1 KiB, 164.1 KiB, 164.1 KiB (stage 3300.0: task 26059)) local merged bytes read: 0.0 B number of partitions: 10 remote reqs duration: 0 ms remote bytes read to disk: 0.0 B shuffle bytes written total (min, med, max (stageId: taskId)) 162.5 KiB (19.9 KiB, 20.3 KiB, 20.7 KiB (stage 3300.0: task 26061))"]; subgraph cluster9 { isCluster="true"; label="WholeStageCodegen (1)\n \nduration: total (min, med, max (stageId: taskId))\n107 ms (11 ms, 12 ms, 25 ms (stage 3300.0: task 26059))"; 10 [labelType="html" label=" LocalLimit "]; 11 [labelType="html" label=" Project "]; } 12 [labelType="html" label="Scan csv number of output rows: 12,000 number of files read: 1 metadata time: 0 ms size of files read: 29.7 MiB"]; subgraph cluster13 { isCluster="true"; label="WholeStageCodegen (4)\n \nduration: 0 ms"; 14 [labelType="html" label="Sort sort time: 0 ms peak memory: 64.1 MiB spill size: 0.0 B"]; } 15 [labelType="html" label="AQEShuffleRead number of partitions: 1 partition data size: 1883.0 B number of coalesced partitions: 1"]; 16 [labelType="html" label="Exchange shuffle records written: 31 local merged chunks fetched: 0 shuffle write time: 1 ms remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 31 local bytes read: 1816.0 B fetch wait time: 0 ms remote bytes read: 0.0 B merged fetch fallback count: 0 local blocks read: 1 remote merged chunks fetched: 0 remote blocks read: 0 data size: 1776.0 B local merged bytes read: 0.0 B number of partitions: 10 remote reqs duration: 0 ms remote bytes read to disk: 0.0 B shuffle bytes written: 1816.0 B"]; subgraph cluster17 { isCluster="true"; label="WholeStageCodegen (2)\n \nduration: 6 ms"; 18 [labelType="html" label=" Project "]; 19 [labelType="html" label="Filter number of output rows: 31"]; } 20 [labelType="html" label="Scan csv number of output rows: 31 number of files read: 1 metadata time: 0 ms size of files read: 812.0 B"]; 1->0; 3->1; 4->3; 6->4; 7->6; 8->7; 10->8; 11->10; 12->11; 14->4; 15->14; 16->15; 18->16; 19->18; 20->19; }

AdaptiveSparkPlan isFinalPlan=true

CollectLimit 1500

Project [COL_2F4B6682_E156_4C20_BE7D_D44470CF5BE0#453917, COL_B50EA623_693D_446D_A303_ACA69FEAD959#453918, COL_E226E62C_37DE_4649_8D35_5E2EA725FACC#453919, COL_C6728F41_A55A_45B1_B7A5_FC74A3A0E5FE#453920, COL_AFD799C8_68A7_4102_A165_EAFF501D0780#453921, CODEOFF_426387__LABEL#453930 AS COL_1601C1F0_4948_40A5_BFC0_054407FC2872#453959]

SortMergeJoin [COL_AFD799C8_68A7_4102_A165_EAFF501D0780#453921], [CODEOFF_426387__CODEOFF#453929], LeftOuter

WholeStageCodegen (5)

Sort [COL_AFD799C8_68A7_4102_A165_EAFF501D0780#453921 ASC NULLS FIRST], false, 0

WholeStageCodegen (3)

AQEShuffleRead coalesced

Exchange hashpartitioning(COL_AFD799C8_68A7_4102_A165_EAFF501D0780#453921, 10), ENSURE_REQUIREMENTS, [plan_id=108651]

LocalLimit 1500

Project [DOSNUM#453760 AS COL_2F4B6682_E156_4C20_BE7D_D44470CF5BE0#453917, ACTACC1#453761 AS COL_B50EA623_693D_446D_A303_ACA69FEAD959#453918, TYPE_ACTE#453762 AS COL_E226E62C_37DE_4649_8D35_5E2EA725FACC#453919, str_concat(array(DATE_ACTE#453763, :00)) AS COL_C6728F41_A55A_45B1_B7A5_FC74A3A0E5FE#453920, SPECACC#453764 AS COL_AFD799C8_68A7_4102_A165_EAFF501D0780#453921]

WholeStageCodegen (1)

FileScan csv [DOSNUM#453760,ACTACC1#453761,TYPE_ACTE#453762,DATE_ACTE#453763,SPECACC#453764] Batched: false, DataFilters: [], Format: CSV, Location: InMemoryFileIndex(1 paths)[file:/data/input/depot/csv/execution/0B169EC1_E45B_447C_B262_7BC2954E3..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<DOSNUM:string,ACTACC1:string,TYPE_ACTE:string,DATE_ACTE:string,SPECACC:string>

Sort [CODEOFF_426387__CODEOFF#453929 ASC NULLS FIRST], false, 0

WholeStageCodegen (4)

AQEShuffleRead coalesced

Exchange hashpartitioning(CODEOFF_426387__CODEOFF#453929, 10), ENSURE_REQUIREMENTS, [plan_id=108672]

Project [CodeOff#453833 AS CODEOFF_426387__CODEOFF#453929, Label#453832 AS CODEOFF_426387__LABEL#453930]

Filter isnotnull(CodeOff#453833)

WholeStageCodegen (2)

FileScan csv [Label#453832,CodeOff#453833] Batched: false, DataFilters: [isnotnull(CodeOff#453833)], Format: CSV, Location: InMemoryFileIndex(1 paths)[file:/data/input/depot/csv/execution/CB74BA10_C402_4727_9E22_0E29596EF..., PartitionFilters: [], PushedFilters: [IsNotNull(CodeOff)], ReadSchema: struct<Label:string,CodeOff:string>

Details

== Physical Plan ==
AdaptiveSparkPlan (29)
+- == Final Plan ==
   CollectLimit (17)
   +- * Project (16)
      +- * SortMergeJoin LeftOuter (15)
         :- * Sort (7)
         :  +- AQEShuffleRead (6)
         :     +- ShuffleQueryStage (5), Statistics(sizeInBytes=1312.5 KiB, rowCount=1.20E+4)
         :        +- Exchange (4)
         :           +- * LocalLimit (3)
         :              +- * Project (2)
         :                 +- Scan csv  (1)
         +- * Sort (14)
            +- AQEShuffleRead (13)
               +- ShuffleQueryStage (12), Statistics(sizeInBytes=1776.0 B, rowCount=31)
                  +- Exchange (11)
                     +- * Project (10)
                        +- * Filter (9)
                           +- Scan csv  (8)
+- == Initial Plan ==
   CollectLimit (28)
   +- Project (27)
      +- SortMergeJoin LeftOuter (26)
         :- Sort (21)
         :  +- Exchange (20)
         :     +- LocalLimit (19)
         :        +- Project (18)
         :           +- Scan csv  (1)
         +- Sort (25)
            +- Exchange (24)
               +- Project (23)
                  +- Filter (22)
                     +- Scan csv  (8)


(1) Scan csv 
Output [5]: [DOSNUM#453760, ACTACC1#453761, TYPE_ACTE#453762, DATE_ACTE#453763, SPECACC#453764]
Batched: false
Location: InMemoryFileIndex [file:/data/input/depot/csv/execution/0B169EC1_E45B_447C_B262_7BC2954E3BEE/current/RSF_Actes__patient.csv]
ReadSchema: struct<DOSNUM:string,ACTACC1:string,TYPE_ACTE:string,DATE_ACTE:string,SPECACC:string>

(2) Project [codegen id : 1]
Output [5]: [DOSNUM#453760 AS COL_2F4B6682_E156_4C20_BE7D_D44470CF5BE0#453917, ACTACC1#453761 AS COL_B50EA623_693D_446D_A303_ACA69FEAD959#453918, TYPE_ACTE#453762 AS COL_E226E62C_37DE_4649_8D35_5E2EA725FACC#453919, str_concat(array(DATE_ACTE#453763, :00)) AS COL_C6728F41_A55A_45B1_B7A5_FC74A3A0E5FE#453920, SPECACC#453764 AS COL_AFD799C8_68A7_4102_A165_EAFF501D0780#453921]
Input [5]: [DOSNUM#453760, ACTACC1#453761, TYPE_ACTE#453762, DATE_ACTE#453763, SPECACC#453764]

(3) LocalLimit [codegen id : 1]
Input [5]: [COL_2F4B6682_E156_4C20_BE7D_D44470CF5BE0#453917, COL_B50EA623_693D_446D_A303_ACA69FEAD959#453918, COL_E226E62C_37DE_4649_8D35_5E2EA725FACC#453919, COL_C6728F41_A55A_45B1_B7A5_FC74A3A0E5FE#453920, COL_AFD799C8_68A7_4102_A165_EAFF501D0780#453921]
Arguments: 1500

(4) Exchange
Input [5]: [COL_2F4B6682_E156_4C20_BE7D_D44470CF5BE0#453917, COL_B50EA623_693D_446D_A303_ACA69FEAD959#453918, COL_E226E62C_37DE_4649_8D35_5E2EA725FACC#453919, COL_C6728F41_A55A_45B1_B7A5_FC74A3A0E5FE#453920, COL_AFD799C8_68A7_4102_A165_EAFF501D0780#453921]
Arguments: hashpartitioning(COL_AFD799C8_68A7_4102_A165_EAFF501D0780#453921, 10), ENSURE_REQUIREMENTS, [plan_id=108651]

(5) ShuffleQueryStage
Output [5]: [COL_2F4B6682_E156_4C20_BE7D_D44470CF5BE0#453917, COL_B50EA623_693D_446D_A303_ACA69FEAD959#453918, COL_E226E62C_37DE_4649_8D35_5E2EA725FACC#453919, COL_C6728F41_A55A_45B1_B7A5_FC74A3A0E5FE#453920, COL_AFD799C8_68A7_4102_A165_EAFF501D0780#453921]
Arguments: 0

(6) AQEShuffleRead
Input [5]: [COL_2F4B6682_E156_4C20_BE7D_D44470CF5BE0#453917, COL_B50EA623_693D_446D_A303_ACA69FEAD959#453918, COL_E226E62C_37DE_4649_8D35_5E2EA725FACC#453919, COL_C6728F41_A55A_45B1_B7A5_FC74A3A0E5FE#453920, COL_AFD799C8_68A7_4102_A165_EAFF501D0780#453921]
Arguments: coalesced

(7) Sort [codegen id : 3]
Input [5]: [COL_2F4B6682_E156_4C20_BE7D_D44470CF5BE0#453917, COL_B50EA623_693D_446D_A303_ACA69FEAD959#453918, COL_E226E62C_37DE_4649_8D35_5E2EA725FACC#453919, COL_C6728F41_A55A_45B1_B7A5_FC74A3A0E5FE#453920, COL_AFD799C8_68A7_4102_A165_EAFF501D0780#453921]
Arguments: [COL_AFD799C8_68A7_4102_A165_EAFF501D0780#453921 ASC NULLS FIRST], false, 0

(8) Scan csv 
Output [2]: [Label#453832, CodeOff#453833]
Batched: false
Location: InMemoryFileIndex [file:/data/input/depot/csv/execution/CB74BA10_C402_4727_9E22_0E29596EFCA3/current/CodOff.csv]
PushedFilters: [IsNotNull(CodeOff)]
ReadSchema: struct<Label:string,CodeOff:string>

(9) Filter [codegen id : 2]
Input [2]: [Label#453832, CodeOff#453833]
Condition : isnotnull(CodeOff#453833)

(10) Project [codegen id : 2]
Output [2]: [CodeOff#453833 AS CODEOFF_426387__CODEOFF#453929, Label#453832 AS CODEOFF_426387__LABEL#453930]
Input [2]: [Label#453832, CodeOff#453833]

(11) Exchange
Input [2]: [CODEOFF_426387__CODEOFF#453929, CODEOFF_426387__LABEL#453930]
Arguments: hashpartitioning(CODEOFF_426387__CODEOFF#453929, 10), ENSURE_REQUIREMENTS, [plan_id=108672]

(12) ShuffleQueryStage
Output [2]: [CODEOFF_426387__CODEOFF#453929, CODEOFF_426387__LABEL#453930]
Arguments: 1

(13) AQEShuffleRead
Input [2]: [CODEOFF_426387__CODEOFF#453929, CODEOFF_426387__LABEL#453930]
Arguments: coalesced

(14) Sort [codegen id : 4]
Input [2]: [CODEOFF_426387__CODEOFF#453929, CODEOFF_426387__LABEL#453930]
Arguments: [CODEOFF_426387__CODEOFF#453929 ASC NULLS FIRST], false, 0

(15) SortMergeJoin [codegen id : 5]
Left keys [1]: [COL_AFD799C8_68A7_4102_A165_EAFF501D0780#453921]
Right keys [1]: [CODEOFF_426387__CODEOFF#453929]
Join type: LeftOuter
Join condition: None

(16) Project [codegen id : 5]
Output [6]: [COL_2F4B6682_E156_4C20_BE7D_D44470CF5BE0#453917, COL_B50EA623_693D_446D_A303_ACA69FEAD959#453918, COL_E226E62C_37DE_4649_8D35_5E2EA725FACC#453919, COL_C6728F41_A55A_45B1_B7A5_FC74A3A0E5FE#453920, COL_AFD799C8_68A7_4102_A165_EAFF501D0780#453921, CODEOFF_426387__LABEL#453930 AS COL_1601C1F0_4948_40A5_BFC0_054407FC2872#453959]
Input [7]: [COL_2F4B6682_E156_4C20_BE7D_D44470CF5BE0#453917, COL_B50EA623_693D_446D_A303_ACA69FEAD959#453918, COL_E226E62C_37DE_4649_8D35_5E2EA725FACC#453919, COL_C6728F41_A55A_45B1_B7A5_FC74A3A0E5FE#453920, COL_AFD799C8_68A7_4102_A165_EAFF501D0780#453921, CODEOFF_426387__CODEOFF#453929, CODEOFF_426387__LABEL#453930]

(17) CollectLimit
Input [6]: [COL_2F4B6682_E156_4C20_BE7D_D44470CF5BE0#453917, COL_B50EA623_693D_446D_A303_ACA69FEAD959#453918, COL_E226E62C_37DE_4649_8D35_5E2EA725FACC#453919, COL_C6728F41_A55A_45B1_B7A5_FC74A3A0E5FE#453920, COL_AFD799C8_68A7_4102_A165_EAFF501D0780#453921, COL_1601C1F0_4948_40A5_BFC0_054407FC2872#453959]
Arguments: 1500

(18) Project
Output [5]: [DOSNUM#453760 AS COL_2F4B6682_E156_4C20_BE7D_D44470CF5BE0#453917, ACTACC1#453761 AS COL_B50EA623_693D_446D_A303_ACA69FEAD959#453918, TYPE_ACTE#453762 AS COL_E226E62C_37DE_4649_8D35_5E2EA725FACC#453919, str_concat(array(DATE_ACTE#453763, :00)) AS COL_C6728F41_A55A_45B1_B7A5_FC74A3A0E5FE#453920, SPECACC#453764 AS COL_AFD799C8_68A7_4102_A165_EAFF501D0780#453921]
Input [5]: [DOSNUM#453760, ACTACC1#453761, TYPE_ACTE#453762, DATE_ACTE#453763, SPECACC#453764]

(19) LocalLimit
Input [5]: [COL_2F4B6682_E156_4C20_BE7D_D44470CF5BE0#453917, COL_B50EA623_693D_446D_A303_ACA69FEAD959#453918, COL_E226E62C_37DE_4649_8D35_5E2EA725FACC#453919, COL_C6728F41_A55A_45B1_B7A5_FC74A3A0E5FE#453920, COL_AFD799C8_68A7_4102_A165_EAFF501D0780#453921]
Arguments: 1500

(20) Exchange
Input [5]: [COL_2F4B6682_E156_4C20_BE7D_D44470CF5BE0#453917, COL_B50EA623_693D_446D_A303_ACA69FEAD959#453918, COL_E226E62C_37DE_4649_8D35_5E2EA725FACC#453919, COL_C6728F41_A55A_45B1_B7A5_FC74A3A0E5FE#453920, COL_AFD799C8_68A7_4102_A165_EAFF501D0780#453921]
Arguments: hashpartitioning(COL_AFD799C8_68A7_4102_A165_EAFF501D0780#453921, 10), ENSURE_REQUIREMENTS, [plan_id=108634]

(21) Sort
Input [5]: [COL_2F4B6682_E156_4C20_BE7D_D44470CF5BE0#453917, COL_B50EA623_693D_446D_A303_ACA69FEAD959#453918, COL_E226E62C_37DE_4649_8D35_5E2EA725FACC#453919, COL_C6728F41_A55A_45B1_B7A5_FC74A3A0E5FE#453920, COL_AFD799C8_68A7_4102_A165_EAFF501D0780#453921]
Arguments: [COL_AFD799C8_68A7_4102_A165_EAFF501D0780#453921 ASC NULLS FIRST], false, 0

(22) Filter
Input [2]: [Label#453832, CodeOff#453833]
Condition : isnotnull(CodeOff#453833)

(23) Project
Output [2]: [CodeOff#453833 AS CODEOFF_426387__CODEOFF#453929, Label#453832 AS CODEOFF_426387__LABEL#453930]
Input [2]: [Label#453832, CodeOff#453833]

(24) Exchange
Input [2]: [CODEOFF_426387__CODEOFF#453929, CODEOFF_426387__LABEL#453930]
Arguments: hashpartitioning(CODEOFF_426387__CODEOFF#453929, 10), ENSURE_REQUIREMENTS, [plan_id=108635]

(25) Sort
Input [2]: [CODEOFF_426387__CODEOFF#453929, CODEOFF_426387__LABEL#453930]
Arguments: [CODEOFF_426387__CODEOFF#453929 ASC NULLS FIRST], false, 0

(26) SortMergeJoin
Left keys [1]: [COL_AFD799C8_68A7_4102_A165_EAFF501D0780#453921]
Right keys [1]: [CODEOFF_426387__CODEOFF#453929]
Join type: LeftOuter
Join condition: None

(27) Project
Output [6]: [COL_2F4B6682_E156_4C20_BE7D_D44470CF5BE0#453917, COL_B50EA623_693D_446D_A303_ACA69FEAD959#453918, COL_E226E62C_37DE_4649_8D35_5E2EA725FACC#453919, COL_C6728F41_A55A_45B1_B7A5_FC74A3A0E5FE#453920, COL_AFD799C8_68A7_4102_A165_EAFF501D0780#453921, CODEOFF_426387__LABEL#453930 AS COL_1601C1F0_4948_40A5_BFC0_054407FC2872#453959]
Input [7]: [COL_2F4B6682_E156_4C20_BE7D_D44470CF5BE0#453917, COL_B50EA623_693D_446D_A303_ACA69FEAD959#453918, COL_E226E62C_37DE_4649_8D35_5E2EA725FACC#453919, COL_C6728F41_A55A_45B1_B7A5_FC74A3A0E5FE#453920, COL_AFD799C8_68A7_4102_A165_EAFF501D0780#453921, CODEOFF_426387__CODEOFF#453929, CODEOFF_426387__LABEL#453930]

(28) CollectLimit
Input [6]: [COL_2F4B6682_E156_4C20_BE7D_D44470CF5BE0#453917, COL_B50EA623_693D_446D_A303_ACA69FEAD959#453918, COL_E226E62C_37DE_4649_8D35_5E2EA725FACC#453919, COL_C6728F41_A55A_45B1_B7A5_FC74A3A0E5FE#453920, COL_AFD799C8_68A7_4102_A165_EAFF501D0780#453921, COL_1601C1F0_4948_40A5_BFC0_054407FC2872#453959]
Arguments: 1500

(29) AdaptiveSparkPlan
Output [6]: [COL_2F4B6682_E156_4C20_BE7D_D44470CF5BE0#453917, COL_B50EA623_693D_446D_A303_ACA69FEAD959#453918, COL_E226E62C_37DE_4649_8D35_5E2EA725FACC#453919, COL_C6728F41_A55A_45B1_B7A5_FC74A3A0E5FE#453920, COL_AFD799C8_68A7_4102_A165_EAFF501D0780#453921, COL_1601C1F0_4948_40A5_BFC0_054407FC2872#453959]
Arguments: isFinalPlan=true

SQL / DataFrame Properties

Name	Value
spark.sql.optimizer.nestedPredicatePushdown.supportedFileSources	parquet,orc,geoparquet