hdh-spark-job - Details for Query 1560

Submitted Time: 2026/09/01 08:53:47
Duration: 0.1 s
Succeeded Jobs: 2010

Show the Stage ID and Task ID that corresponds to the max metric

digraph G { 0 [labelType="html" label=" AdaptiveSparkPlan "]; 1 [labelType="html" label="Execute InsertIntoHadoopFsRelationCommand task commit time: 1 ms number of written files: 1 job commit time: 6 ms number of output rows: 101 number of dynamic part: 0 written output: 2007.7 KiB"]; 2 [labelType="html" label=" WriteFiles "]; 3 [labelType="html" label="InMemoryTableScan number of output rows: 101"]; subgraph cluster4 { isCluster="true"; label="WholeStageCodegen (1)\n \nduration: 0 ms"; 5 [labelType="html" label=" Project "]; 6 [labelType="html" label="Generate number of output rows: 0"]; 7 [labelType="html" label=" Project "]; } 8 [labelType="html" label="Scan json number of output rows: 0"]; 1->0; 2->1; 3->2; 5->3; 6->5; 7->6; 8->7; }

AdaptiveSparkPlan isFinalPlan=true

Execute InsertIntoHadoopFsRelationCommand file:/data/data/subscriptions/NOMENCLATURES_A_425642/bcb2829d-01af-47f3-a9b9-d04a6a136bda/2026-09-01_08-53-46-812_LOCK_aa690d42-62e4-4723-8b6f-42b23e733261, false, Parquet, [path=file:///data/data/subscriptions/NOMENCLATURES_A_425642/bcb2829d-01af-47f3-a9b9-d04a6a136bda/2026-09-01_08-53-46-812_LOCK_aa690d42-62e4-4723-8b6f-42b23e733261], ErrorIfExists, [DD949BC3_0B6D_4794_9931_FBDE5AA578F6, B9DBAA59_BCA5_407D_978B_503F6506C191, 71114C29_6B7C_4DE0_924E_E46ACE77220A]

WriteFiles

InMemoryTableScan [DD949BC3_0B6D_4794_9931_FBDE5AA578F6#338549, B9DBAA59_BCA5_407D_978B_503F6506C191#338550, 71114C29_6B7C_4DE0_924E_E46ACE77220A#338551]

Project [f3b74279-4a39-436c-9c7b-11e3690a65f8#338462.geometry AS DD949BC3_0B6D_4794_9931_FBDE5AA578F6#338549, f3b74279-4a39-436c-9c7b-11e3690a65f8#338462.properties.code AS B9DBAA59_BCA5_407D_978B_503F6506C191#338550, f3b74279-4a39-436c-9c7b-11e3690a65f8#338462.properties.nom AS 71114C29_6B7C_4DE0_924E_E46ACE77220A#338551]

Generate explode(c14d4058-9056-411f-97e6-b4fefa16fa0a#338456), true, [f3b74279-4a39-436c-9c7b-11e3690a65f8#338462]

Project [features#338447 AS c14d4058-9056-411f-97e6-b4fefa16fa0a#338456]

WholeStageCodegen (1)

FileScan json [features#338447] Batched: false, DataFilters: [], Format: JSON, Location: InMemoryFileIndex(1 paths)[file:/data/input/depot/json/execution/62DE089E_A2D6_4F97_9EC1_F53A7F2A..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<features:array<struct<geometry:string,properties:struct<code:string,nom:string>,type:strin...

Details

== Physical Plan ==
AdaptiveSparkPlan (12)
+- == Final Plan ==
   Execute InsertIntoHadoopFsRelationCommand (9)
   +- WriteFiles (8)
      +- TableCacheQueryStage (7), Statistics(sizeInBytes=3.5 MiB, rowCount=101)
         +- InMemoryTableScan (1)
               +- InMemoryRelation (2)
                     +- * Project (6)
                        +- * Generate (5)
                           +- * Project (4)
                              +- Scan json  (3)
+- == Initial Plan ==
   Execute InsertIntoHadoopFsRelationCommand (11)
   +- WriteFiles (10)
      +- InMemoryTableScan (1)
            +- InMemoryRelation (2)
                  +- * Project (6)
                     +- * Generate (5)
                        +- * Project (4)
                           +- Scan json  (3)


(1) InMemoryTableScan
Output [3]: [DD949BC3_0B6D_4794_9931_FBDE5AA578F6#338549, B9DBAA59_BCA5_407D_978B_503F6506C191#338550, 71114C29_6B7C_4DE0_924E_E46ACE77220A#338551]
Arguments: [DD949BC3_0B6D_4794_9931_FBDE5AA578F6#338549, B9DBAA59_BCA5_407D_978B_503F6506C191#338550, 71114C29_6B7C_4DE0_924E_E46ACE77220A#338551]

(2) InMemoryRelation
Arguments: [DD949BC3_0B6D_4794_9931_FBDE5AA578F6#338549, B9DBAA59_BCA5_407D_978B_503F6506C191#338550, 71114C29_6B7C_4DE0_924E_E46ACE77220A#338551], CachedRDDBuilder(org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer@68f4cb77,StorageLevel(disk, memory, deserialized, 1 replicas),*(1) Project [f3b74279-4a39-436c-9c7b-11e3690a65f8#338462.geometry AS DD949BC3_0B6D_4794_9931_FBDE5AA578F6#338549, f3b74279-4a39-436c-9c7b-11e3690a65f8#338462.properties.code AS B9DBAA59_BCA5_407D_978B_503F6506C191#338550, f3b74279-4a39-436c-9c7b-11e3690a65f8#338462.properties.nom AS 71114C29_6B7C_4DE0_924E_E46ACE77220A#338551]
+- *(1) Generate explode(c14d4058-9056-411f-97e6-b4fefa16fa0a#338456), true, [f3b74279-4a39-436c-9c7b-11e3690a65f8#338462]
   +- *(1) Project [features#338447 AS c14d4058-9056-411f-97e6-b4fefa16fa0a#338456]
      +- FileScan json [features#338447] Batched: false, DataFilters: [], Format: JSON, Location: InMemoryFileIndex(1 paths)[file:/data/input/depot/json/execution/62DE089E_A2D6_4F97_9EC1_F53A7F2A..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<features:array<struct<geometry:string,properties:struct<code:string,nom:string>,type:strin...
,None)

(3) Scan json 
Output [1]: [features#338447]
Batched: false
Location: InMemoryFileIndex [file:/data/input/depot/json/execution/62DE089E_A2D6_4F97_9EC1_F53A7F2A78CC/current/departements-avec-outre-mer.geojson]
ReadSchema: struct<features:array<struct<geometry:string,properties:struct<code:string,nom:string>,type:string>>>

(4) Project [codegen id : 1]
Output [1]: [features#338447 AS c14d4058-9056-411f-97e6-b4fefa16fa0a#338456]
Input [1]: [features#338447]

(5) Generate [codegen id : 1]
Input [1]: [c14d4058-9056-411f-97e6-b4fefa16fa0a#338456]
Arguments: explode(c14d4058-9056-411f-97e6-b4fefa16fa0a#338456), true, [f3b74279-4a39-436c-9c7b-11e3690a65f8#338462]

(6) Project [codegen id : 1]
Output [3]: [f3b74279-4a39-436c-9c7b-11e3690a65f8#338462.geometry AS DD949BC3_0B6D_4794_9931_FBDE5AA578F6#338549, f3b74279-4a39-436c-9c7b-11e3690a65f8#338462.properties.code AS B9DBAA59_BCA5_407D_978B_503F6506C191#338550, f3b74279-4a39-436c-9c7b-11e3690a65f8#338462.properties.nom AS 71114C29_6B7C_4DE0_924E_E46ACE77220A#338551]
Input [1]: [f3b74279-4a39-436c-9c7b-11e3690a65f8#338462]

(7) TableCacheQueryStage
Output [3]: [DD949BC3_0B6D_4794_9931_FBDE5AA578F6#338549, B9DBAA59_BCA5_407D_978B_503F6506C191#338550, 71114C29_6B7C_4DE0_924E_E46ACE77220A#338551]
Arguments: 0

(8) WriteFiles
Input [3]: [DD949BC3_0B6D_4794_9931_FBDE5AA578F6#338549, B9DBAA59_BCA5_407D_978B_503F6506C191#338550, 71114C29_6B7C_4DE0_924E_E46ACE77220A#338551]

(9) Execute InsertIntoHadoopFsRelationCommand
Input: []
Arguments: file:/data/data/subscriptions/NOMENCLATURES_A_425642/bcb2829d-01af-47f3-a9b9-d04a6a136bda/2026-09-01_08-53-46-812_LOCK_aa690d42-62e4-4723-8b6f-42b23e733261, false, Parquet, [path=file:///data/data/subscriptions/NOMENCLATURES_A_425642/bcb2829d-01af-47f3-a9b9-d04a6a136bda/2026-09-01_08-53-46-812_LOCK_aa690d42-62e4-4723-8b6f-42b23e733261], ErrorIfExists, [DD949BC3_0B6D_4794_9931_FBDE5AA578F6, B9DBAA59_BCA5_407D_978B_503F6506C191, 71114C29_6B7C_4DE0_924E_E46ACE77220A]

(10) WriteFiles
Input [3]: [DD949BC3_0B6D_4794_9931_FBDE5AA578F6#338549, B9DBAA59_BCA5_407D_978B_503F6506C191#338550, 71114C29_6B7C_4DE0_924E_E46ACE77220A#338551]

(11) Execute InsertIntoHadoopFsRelationCommand
Input: []
Arguments: file:/data/data/subscriptions/NOMENCLATURES_A_425642/bcb2829d-01af-47f3-a9b9-d04a6a136bda/2026-09-01_08-53-46-812_LOCK_aa690d42-62e4-4723-8b6f-42b23e733261, false, Parquet, [path=file:///data/data/subscriptions/NOMENCLATURES_A_425642/bcb2829d-01af-47f3-a9b9-d04a6a136bda/2026-09-01_08-53-46-812_LOCK_aa690d42-62e4-4723-8b6f-42b23e733261], ErrorIfExists, [DD949BC3_0B6D_4794_9931_FBDE5AA578F6, B9DBAA59_BCA5_407D_978B_503F6506C191, 71114C29_6B7C_4DE0_924E_E46ACE77220A]

(12) AdaptiveSparkPlan
Output: []
Arguments: isFinalPlan=true

SQL / DataFrame Properties

Name	Value
spark.sql.optimizer.nestedPredicatePushdown.supportedFileSources	parquet,orc,geoparquet