hdh-spark-job - Details for Query 1829

Submitted Time: 2026/09/03 12:16:25
Duration: 0.2 s
Succeeded Jobs: 2279 2280 2282

Show the Stage ID and Task ID that corresponds to the max metric

digraph G { 0 [labelType="html" label=" AdaptiveSparkPlan "]; 1 [labelType="html" label=" CollectLimit "]; subgraph cluster2 { isCluster="true"; label="WholeStageCodegen (5)\n \nduration: 0 ms"; 3 [labelType="html" label=" Project "]; 4 [labelType="html" label="SortMergeJoin number of output rows: 1,500 spill size: 0.0 B"]; } subgraph cluster5 { isCluster="true"; label="WholeStageCodegen (3)\n \nduration: 0 ms"; 6 [labelType="html" label="Sort sort time: 2 ms peak memory: 64.5 MiB spill size: 0.0 B"]; } 7 [labelType="html" label="AQEShuffleRead number of partitions: 1 partition data size: 171.0 KiB number of coalesced partitions: 1"]; 8 [labelType="html" label="Exchange shuffle records written: 12,000 local merged chunks fetched: 0 shuffle write time total (min, med, max (stageId: taskId)) 10 ms (1 ms, 1 ms, 1 ms (stage 3318.0: task 26087)) remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 12,000 local bytes read: 162.5 KiB fetch wait time: 0 ms remote bytes read: 0.0 B merged fetch fallback count: 0 local blocks read: 8 remote merged chunks fetched: 0 remote blocks read: 0 data size total (min, med, max (stageId: taskId)) 1312.5 KiB (164.1 KiB, 164.1 KiB, 164.1 KiB (stage 3318.0: task 26084)) local merged bytes read: 0.0 B number of partitions: 10 remote reqs duration: 0 ms remote bytes read to disk: 0.0 B shuffle bytes written total (min, med, max (stageId: taskId)) 162.5 KiB (19.9 KiB, 20.3 KiB, 20.7 KiB (stage 3318.0: task 26086))"]; subgraph cluster9 { isCluster="true"; label="WholeStageCodegen (1)\n \nduration: total (min, med, max (stageId: taskId))\n89 ms (11 ms, 11 ms, 12 ms (stage 3318.0: task 26084))"; 10 [labelType="html" label=" LocalLimit "]; 11 [labelType="html" label=" Project "]; } 12 [labelType="html" label="Scan csv number of output rows: 12,000 number of files read: 1 metadata time: 0 ms size of files read: 29.7 MiB"]; subgraph cluster13 { isCluster="true"; label="WholeStageCodegen (4)\n \nduration: 0 ms"; 14 [labelType="html" label="Sort sort time: 0 ms peak memory: 64.1 MiB spill size: 0.0 B"]; } 15 [labelType="html" label="AQEShuffleRead number of partitions: 1 partition data size: 1883.0 B number of coalesced partitions: 1"]; 16 [labelType="html" label="Exchange shuffle records written: 31 local merged chunks fetched: 0 shuffle write time: 1 ms remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 31 local bytes read: 1816.0 B fetch wait time: 0 ms remote bytes read: 0.0 B merged fetch fallback count: 0 local blocks read: 1 remote merged chunks fetched: 0 remote blocks read: 0 data size: 1776.0 B local merged bytes read: 0.0 B number of partitions: 10 remote reqs duration: 0 ms remote bytes read to disk: 0.0 B shuffle bytes written: 1816.0 B"]; subgraph cluster17 { isCluster="true"; label="WholeStageCodegen (2)\n \nduration: 4 ms"; 18 [labelType="html" label=" Project "]; 19 [labelType="html" label="Filter number of output rows: 31"]; } 20 [labelType="html" label="Scan csv number of output rows: 31 number of files read: 1 metadata time: 0 ms size of files read: 812.0 B"]; 1->0; 3->1; 4->3; 6->4; 7->6; 8->7; 10->8; 11->10; 12->11; 14->4; 15->14; 16->15; 18->16; 19->18; 20->19; }

AdaptiveSparkPlan isFinalPlan=true

CollectLimit 1500

Project [COL_2F4B6682_E156_4C20_BE7D_D44470CF5BE0#454387, COL_B50EA623_693D_446D_A303_ACA69FEAD959#454388, COL_E226E62C_37DE_4649_8D35_5E2EA725FACC#454389, COL_C6728F41_A55A_45B1_B7A5_FC74A3A0E5FE#454390, CODEOFF_426387__LABEL#454400 AS COL_1601C1F0_4948_40A5_BFC0_054407FC2872#454428]

SortMergeJoin [COL_AFD799C8_68A7_4102_A165_EAFF501D0780#454391], [CODEOFF_426387__CODEOFF#454399], LeftOuter

WholeStageCodegen (5)

Sort [COL_AFD799C8_68A7_4102_A165_EAFF501D0780#454391 ASC NULLS FIRST], false, 0

WholeStageCodegen (3)

AQEShuffleRead coalesced

Exchange hashpartitioning(COL_AFD799C8_68A7_4102_A165_EAFF501D0780#454391, 10), ENSURE_REQUIREMENTS, [plan_id=109074]

LocalLimit 1500

Project [DOSNUM#454230 AS COL_2F4B6682_E156_4C20_BE7D_D44470CF5BE0#454387, ACTACC1#454231 AS COL_B50EA623_693D_446D_A303_ACA69FEAD959#454388, TYPE_ACTE#454232 AS COL_E226E62C_37DE_4649_8D35_5E2EA725FACC#454389, str_concat(array(DATE_ACTE#454233, :00)) AS COL_C6728F41_A55A_45B1_B7A5_FC74A3A0E5FE#454390, SPECACC#454234 AS COL_AFD799C8_68A7_4102_A165_EAFF501D0780#454391]

WholeStageCodegen (1)

FileScan csv [DOSNUM#454230,ACTACC1#454231,TYPE_ACTE#454232,DATE_ACTE#454233,SPECACC#454234] Batched: false, DataFilters: [], Format: CSV, Location: InMemoryFileIndex(1 paths)[file:/data/input/depot/csv/execution/0B169EC1_E45B_447C_B262_7BC2954E3..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<DOSNUM:string,ACTACC1:string,TYPE_ACTE:string,DATE_ACTE:string,SPECACC:string>

Sort [CODEOFF_426387__CODEOFF#454399 ASC NULLS FIRST], false, 0

WholeStageCodegen (4)

AQEShuffleRead coalesced

Exchange hashpartitioning(CODEOFF_426387__CODEOFF#454399, 10), ENSURE_REQUIREMENTS, [plan_id=109095]

Project [CodeOff#454303 AS CODEOFF_426387__CODEOFF#454399, Label#454302 AS CODEOFF_426387__LABEL#454400]

Filter isnotnull(CodeOff#454303)

WholeStageCodegen (2)

FileScan csv [Label#454302,CodeOff#454303] Batched: false, DataFilters: [isnotnull(CodeOff#454303)], Format: CSV, Location: InMemoryFileIndex(1 paths)[file:/data/input/depot/csv/execution/CB74BA10_C402_4727_9E22_0E29596EF..., PartitionFilters: [], PushedFilters: [IsNotNull(CodeOff)], ReadSchema: struct<Label:string,CodeOff:string>

Details

== Physical Plan ==
AdaptiveSparkPlan (29)
+- == Final Plan ==
   CollectLimit (17)
   +- * Project (16)
      +- * SortMergeJoin LeftOuter (15)
         :- * Sort (7)
         :  +- AQEShuffleRead (6)
         :     +- ShuffleQueryStage (5), Statistics(sizeInBytes=1312.5 KiB, rowCount=1.20E+4)
         :        +- Exchange (4)
         :           +- * LocalLimit (3)
         :              +- * Project (2)
         :                 +- Scan csv  (1)
         +- * Sort (14)
            +- AQEShuffleRead (13)
               +- ShuffleQueryStage (12), Statistics(sizeInBytes=1776.0 B, rowCount=31)
                  +- Exchange (11)
                     +- * Project (10)
                        +- * Filter (9)
                           +- Scan csv  (8)
+- == Initial Plan ==
   CollectLimit (28)
   +- Project (27)
      +- SortMergeJoin LeftOuter (26)
         :- Sort (21)
         :  +- Exchange (20)
         :     +- LocalLimit (19)
         :        +- Project (18)
         :           +- Scan csv  (1)
         +- Sort (25)
            +- Exchange (24)
               +- Project (23)
                  +- Filter (22)
                     +- Scan csv  (8)


(1) Scan csv 
Output [5]: [DOSNUM#454230, ACTACC1#454231, TYPE_ACTE#454232, DATE_ACTE#454233, SPECACC#454234]
Batched: false
Location: InMemoryFileIndex [file:/data/input/depot/csv/execution/0B169EC1_E45B_447C_B262_7BC2954E3BEE/current/RSF_Actes__patient.csv]
ReadSchema: struct<DOSNUM:string,ACTACC1:string,TYPE_ACTE:string,DATE_ACTE:string,SPECACC:string>

(2) Project [codegen id : 1]
Output [5]: [DOSNUM#454230 AS COL_2F4B6682_E156_4C20_BE7D_D44470CF5BE0#454387, ACTACC1#454231 AS COL_B50EA623_693D_446D_A303_ACA69FEAD959#454388, TYPE_ACTE#454232 AS COL_E226E62C_37DE_4649_8D35_5E2EA725FACC#454389, str_concat(array(DATE_ACTE#454233, :00)) AS COL_C6728F41_A55A_45B1_B7A5_FC74A3A0E5FE#454390, SPECACC#454234 AS COL_AFD799C8_68A7_4102_A165_EAFF501D0780#454391]
Input [5]: [DOSNUM#454230, ACTACC1#454231, TYPE_ACTE#454232, DATE_ACTE#454233, SPECACC#454234]

(3) LocalLimit [codegen id : 1]
Input [5]: [COL_2F4B6682_E156_4C20_BE7D_D44470CF5BE0#454387, COL_B50EA623_693D_446D_A303_ACA69FEAD959#454388, COL_E226E62C_37DE_4649_8D35_5E2EA725FACC#454389, COL_C6728F41_A55A_45B1_B7A5_FC74A3A0E5FE#454390, COL_AFD799C8_68A7_4102_A165_EAFF501D0780#454391]
Arguments: 1500

(4) Exchange
Input [5]: [COL_2F4B6682_E156_4C20_BE7D_D44470CF5BE0#454387, COL_B50EA623_693D_446D_A303_ACA69FEAD959#454388, COL_E226E62C_37DE_4649_8D35_5E2EA725FACC#454389, COL_C6728F41_A55A_45B1_B7A5_FC74A3A0E5FE#454390, COL_AFD799C8_68A7_4102_A165_EAFF501D0780#454391]
Arguments: hashpartitioning(COL_AFD799C8_68A7_4102_A165_EAFF501D0780#454391, 10), ENSURE_REQUIREMENTS, [plan_id=109074]

(5) ShuffleQueryStage
Output [5]: [COL_2F4B6682_E156_4C20_BE7D_D44470CF5BE0#454387, COL_B50EA623_693D_446D_A303_ACA69FEAD959#454388, COL_E226E62C_37DE_4649_8D35_5E2EA725FACC#454389, COL_C6728F41_A55A_45B1_B7A5_FC74A3A0E5FE#454390, COL_AFD799C8_68A7_4102_A165_EAFF501D0780#454391]
Arguments: 0

(6) AQEShuffleRead
Input [5]: [COL_2F4B6682_E156_4C20_BE7D_D44470CF5BE0#454387, COL_B50EA623_693D_446D_A303_ACA69FEAD959#454388, COL_E226E62C_37DE_4649_8D35_5E2EA725FACC#454389, COL_C6728F41_A55A_45B1_B7A5_FC74A3A0E5FE#454390, COL_AFD799C8_68A7_4102_A165_EAFF501D0780#454391]
Arguments: coalesced

(7) Sort [codegen id : 3]
Input [5]: [COL_2F4B6682_E156_4C20_BE7D_D44470CF5BE0#454387, COL_B50EA623_693D_446D_A303_ACA69FEAD959#454388, COL_E226E62C_37DE_4649_8D35_5E2EA725FACC#454389, COL_C6728F41_A55A_45B1_B7A5_FC74A3A0E5FE#454390, COL_AFD799C8_68A7_4102_A165_EAFF501D0780#454391]
Arguments: [COL_AFD799C8_68A7_4102_A165_EAFF501D0780#454391 ASC NULLS FIRST], false, 0

(8) Scan csv 
Output [2]: [Label#454302, CodeOff#454303]
Batched: false
Location: InMemoryFileIndex [file:/data/input/depot/csv/execution/CB74BA10_C402_4727_9E22_0E29596EFCA3/current/CodOff.csv]
PushedFilters: [IsNotNull(CodeOff)]
ReadSchema: struct<Label:string,CodeOff:string>

(9) Filter [codegen id : 2]
Input [2]: [Label#454302, CodeOff#454303]
Condition : isnotnull(CodeOff#454303)

(10) Project [codegen id : 2]
Output [2]: [CodeOff#454303 AS CODEOFF_426387__CODEOFF#454399, Label#454302 AS CODEOFF_426387__LABEL#454400]
Input [2]: [Label#454302, CodeOff#454303]

(11) Exchange
Input [2]: [CODEOFF_426387__CODEOFF#454399, CODEOFF_426387__LABEL#454400]
Arguments: hashpartitioning(CODEOFF_426387__CODEOFF#454399, 10), ENSURE_REQUIREMENTS, [plan_id=109095]

(12) ShuffleQueryStage
Output [2]: [CODEOFF_426387__CODEOFF#454399, CODEOFF_426387__LABEL#454400]
Arguments: 1

(13) AQEShuffleRead
Input [2]: [CODEOFF_426387__CODEOFF#454399, CODEOFF_426387__LABEL#454400]
Arguments: coalesced

(14) Sort [codegen id : 4]
Input [2]: [CODEOFF_426387__CODEOFF#454399, CODEOFF_426387__LABEL#454400]
Arguments: [CODEOFF_426387__CODEOFF#454399 ASC NULLS FIRST], false, 0

(15) SortMergeJoin [codegen id : 5]
Left keys [1]: [COL_AFD799C8_68A7_4102_A165_EAFF501D0780#454391]
Right keys [1]: [CODEOFF_426387__CODEOFF#454399]
Join type: LeftOuter
Join condition: None

(16) Project [codegen id : 5]
Output [5]: [COL_2F4B6682_E156_4C20_BE7D_D44470CF5BE0#454387, COL_B50EA623_693D_446D_A303_ACA69FEAD959#454388, COL_E226E62C_37DE_4649_8D35_5E2EA725FACC#454389, COL_C6728F41_A55A_45B1_B7A5_FC74A3A0E5FE#454390, CODEOFF_426387__LABEL#454400 AS COL_1601C1F0_4948_40A5_BFC0_054407FC2872#454428]
Input [7]: [COL_2F4B6682_E156_4C20_BE7D_D44470CF5BE0#454387, COL_B50EA623_693D_446D_A303_ACA69FEAD959#454388, COL_E226E62C_37DE_4649_8D35_5E2EA725FACC#454389, COL_C6728F41_A55A_45B1_B7A5_FC74A3A0E5FE#454390, COL_AFD799C8_68A7_4102_A165_EAFF501D0780#454391, CODEOFF_426387__CODEOFF#454399, CODEOFF_426387__LABEL#454400]

(17) CollectLimit
Input [5]: [COL_2F4B6682_E156_4C20_BE7D_D44470CF5BE0#454387, COL_B50EA623_693D_446D_A303_ACA69FEAD959#454388, COL_E226E62C_37DE_4649_8D35_5E2EA725FACC#454389, COL_C6728F41_A55A_45B1_B7A5_FC74A3A0E5FE#454390, COL_1601C1F0_4948_40A5_BFC0_054407FC2872#454428]
Arguments: 1500

(18) Project
Output [5]: [DOSNUM#454230 AS COL_2F4B6682_E156_4C20_BE7D_D44470CF5BE0#454387, ACTACC1#454231 AS COL_B50EA623_693D_446D_A303_ACA69FEAD959#454388, TYPE_ACTE#454232 AS COL_E226E62C_37DE_4649_8D35_5E2EA725FACC#454389, str_concat(array(DATE_ACTE#454233, :00)) AS COL_C6728F41_A55A_45B1_B7A5_FC74A3A0E5FE#454390, SPECACC#454234 AS COL_AFD799C8_68A7_4102_A165_EAFF501D0780#454391]
Input [5]: [DOSNUM#454230, ACTACC1#454231, TYPE_ACTE#454232, DATE_ACTE#454233, SPECACC#454234]

(19) LocalLimit
Input [5]: [COL_2F4B6682_E156_4C20_BE7D_D44470CF5BE0#454387, COL_B50EA623_693D_446D_A303_ACA69FEAD959#454388, COL_E226E62C_37DE_4649_8D35_5E2EA725FACC#454389, COL_C6728F41_A55A_45B1_B7A5_FC74A3A0E5FE#454390, COL_AFD799C8_68A7_4102_A165_EAFF501D0780#454391]
Arguments: 1500

(20) Exchange
Input [5]: [COL_2F4B6682_E156_4C20_BE7D_D44470CF5BE0#454387, COL_B50EA623_693D_446D_A303_ACA69FEAD959#454388, COL_E226E62C_37DE_4649_8D35_5E2EA725FACC#454389, COL_C6728F41_A55A_45B1_B7A5_FC74A3A0E5FE#454390, COL_AFD799C8_68A7_4102_A165_EAFF501D0780#454391]
Arguments: hashpartitioning(COL_AFD799C8_68A7_4102_A165_EAFF501D0780#454391, 10), ENSURE_REQUIREMENTS, [plan_id=109057]

(21) Sort
Input [5]: [COL_2F4B6682_E156_4C20_BE7D_D44470CF5BE0#454387, COL_B50EA623_693D_446D_A303_ACA69FEAD959#454388, COL_E226E62C_37DE_4649_8D35_5E2EA725FACC#454389, COL_C6728F41_A55A_45B1_B7A5_FC74A3A0E5FE#454390, COL_AFD799C8_68A7_4102_A165_EAFF501D0780#454391]
Arguments: [COL_AFD799C8_68A7_4102_A165_EAFF501D0780#454391 ASC NULLS FIRST], false, 0

(22) Filter
Input [2]: [Label#454302, CodeOff#454303]
Condition : isnotnull(CodeOff#454303)

(23) Project
Output [2]: [CodeOff#454303 AS CODEOFF_426387__CODEOFF#454399, Label#454302 AS CODEOFF_426387__LABEL#454400]
Input [2]: [Label#454302, CodeOff#454303]

(24) Exchange
Input [2]: [CODEOFF_426387__CODEOFF#454399, CODEOFF_426387__LABEL#454400]
Arguments: hashpartitioning(CODEOFF_426387__CODEOFF#454399, 10), ENSURE_REQUIREMENTS, [plan_id=109058]

(25) Sort
Input [2]: [CODEOFF_426387__CODEOFF#454399, CODEOFF_426387__LABEL#454400]
Arguments: [CODEOFF_426387__CODEOFF#454399 ASC NULLS FIRST], false, 0

(26) SortMergeJoin
Left keys [1]: [COL_AFD799C8_68A7_4102_A165_EAFF501D0780#454391]
Right keys [1]: [CODEOFF_426387__CODEOFF#454399]
Join type: LeftOuter
Join condition: None

(27) Project
Output [5]: [COL_2F4B6682_E156_4C20_BE7D_D44470CF5BE0#454387, COL_B50EA623_693D_446D_A303_ACA69FEAD959#454388, COL_E226E62C_37DE_4649_8D35_5E2EA725FACC#454389, COL_C6728F41_A55A_45B1_B7A5_FC74A3A0E5FE#454390, CODEOFF_426387__LABEL#454400 AS COL_1601C1F0_4948_40A5_BFC0_054407FC2872#454428]
Input [7]: [COL_2F4B6682_E156_4C20_BE7D_D44470CF5BE0#454387, COL_B50EA623_693D_446D_A303_ACA69FEAD959#454388, COL_E226E62C_37DE_4649_8D35_5E2EA725FACC#454389, COL_C6728F41_A55A_45B1_B7A5_FC74A3A0E5FE#454390, COL_AFD799C8_68A7_4102_A165_EAFF501D0780#454391, CODEOFF_426387__CODEOFF#454399, CODEOFF_426387__LABEL#454400]

(28) CollectLimit
Input [5]: [COL_2F4B6682_E156_4C20_BE7D_D44470CF5BE0#454387, COL_B50EA623_693D_446D_A303_ACA69FEAD959#454388, COL_E226E62C_37DE_4649_8D35_5E2EA725FACC#454389, COL_C6728F41_A55A_45B1_B7A5_FC74A3A0E5FE#454390, COL_1601C1F0_4948_40A5_BFC0_054407FC2872#454428]
Arguments: 1500

(29) AdaptiveSparkPlan
Output [5]: [COL_2F4B6682_E156_4C20_BE7D_D44470CF5BE0#454387, COL_B50EA623_693D_446D_A303_ACA69FEAD959#454388, COL_E226E62C_37DE_4649_8D35_5E2EA725FACC#454389, COL_C6728F41_A55A_45B1_B7A5_FC74A3A0E5FE#454390, COL_1601C1F0_4948_40A5_BFC0_054407FC2872#454428]
Arguments: isFinalPlan=true

SQL / DataFrame Properties

Name	Value
spark.sql.optimizer.nestedPredicatePushdown.supportedFileSources	parquet,orc,geoparquet