hdh-spark-job - Details for Query 1388

Submitted Time: 2026/08/31 15:02:37
Duration: 0.5 s
Succeeded Jobs: 1801

Show the Stage ID and Task ID that corresponds to the max metric

digraph G { 0 [labelType="html" label="Execute InsertIntoHadoopFsRelationCommand task commit time total (min, med, max (stageId: taskId)) 11 ms (0 ms, 1 ms, 6 ms (stage 2703.0: task 21525)) number of written files: 6 job commit time: 8 ms number of output rows: 1,519 number of dynamic part: 1 written output: 89.8 KiB"]; 1 [labelType="html" label=" WriteFiles "]; subgraph cluster2 { isCluster="true"; label="WholeStageCodegen (2)\n \nduration: total (min, med, max (stageId: taskId))\n416 ms (10 ms, 35 ms, 122 ms (stage 2703.0: task 21526))"; 3 [labelType="html" label="Sort sort time total (min, med, max (stageId: taskId)) 0 ms (0 ms, 0 ms, 0 ms (stage 2703.0: task 21526)) peak memory total (min, med, max (stageId: taskId)) 384.6 MiB (64.0 KiB, 64.1 MiB, 64.1 MiB (stage 2703.0: task 21526)) spill size total (min, med, max (stageId: taskId)) 0.0 B (0.0 B, 0.0 B, 0.0 B (stage 2703.0: task 21526))"]; } 4 [labelType="html" label=" Project "]; subgraph cluster5 { isCluster="true"; label="WholeStageCodegen (1)\n \nduration: total (min, med, max (stageId: taskId))\n267 ms (6 ms, 15 ms, 92 ms (stage 2703.0: task 21526))"; 6 [labelType="html" label="Generate number of output rows: 1,519"]; } 7 [labelType="html" label=" Project "]; 8 [labelType="html" label="Filter number of output rows: 16"]; 9 [labelType="html" label="Scan text number of output rows: 30 number of files read: 30 metadata time: 0 ms size of files read: 268.0 KiB"]; 1->0; 3->1; 4->3; 6->4; 7->6; 8->7; 9->8; }

Execute InsertIntoHadoopFsRelationCommand file:/data/input/depot/json/execution/INFORMATION_STA_119302/extraction, false, [date_566bc84309361176421a95193a0eb006#294599], Parquet, [__partition_columns=["date_566bc84309361176421a95193a0eb006"], path=file:///data/input/depot/json/execution/INFORMATION_STA_119302/extraction], Append, [capacity, coordonnees_geo, name, station_opening_hours, stationcode, date_566bc84309361176421a95193a0eb006]

WriteFiles

Sort [date_566bc84309361176421a95193a0eb006#294599 ASC NULLS FIRST], false, 0

WholeStageCodegen (2)

Project [root#294539.capacity AS capacity#294564L, to_json(root#294539.coordonnees_geo, Some(Etc/UTC)) AS coordonnees_geo#294571, root#294539.name AS name#294578, root#294539.station_opening_hours AS station_opening_hours#294585, root#294539.stationcode AS stationcode#294592, 2026-08-31 15:02:37.380552 AS date_566bc84309361176421a95193a0eb006#294599]

Generate explode(root#294536), false, [root#294539]

WholeStageCodegen (1)

Project [from_json(ArrayType(StructType(StructField(capacity,LongType,true),StructField(coordonnees_geo,StructType(StructField(lat,DoubleType,true),StructField(lon,DoubleType,true)),true),StructField(name,StringType,true),StructField(station_opening_hours,StringType,true),StructField(stationcode,StringType,true)),true), get_json_object(value#294521, $.results), Some(Etc/UTC)) AS root#294536]

Filter ((size(from_json(ArrayType(StructType(StructField(capacity,LongType,true),StructField(coordonnees_geo,StructType(StructField(lat,DoubleType,true),StructField(lon,DoubleType,true)),true),StructField(name,StringType,true),StructField(station_opening_hours,StringType,true),StructField(stationcode,StringType,true)),true), get_json_object(value#294521, $.results), Some(Etc/UTC)), true) > 0) AND isnotnull(from_json(ArrayType(StructType(StructField(capacity,LongType,true),StructField(coordonnees_geo,StructType(StructField(lat,DoubleType,true),StructField(lon,DoubleType,true)),true),StructField(name,StringType,true),StructField(station_opening_hours,StringType,true),StructField(stationcode,StringType,true)),true), get_json_object(value#294521, $.results), Some(Etc/UTC))))

FileScan text [value#294521] Batched: false, DataFilters: [(size(from_json(ArrayType(StructType(StructField(capacity,LongType,true),StructField(coordonnees..., Format: Text, Location: InMemoryFileIndex(30 paths)[file:/data/input/depot/json/tmp/INFORMATION_STA_119302144/HTTP_FILE_0..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<value:string>

Details

== Physical Plan ==
Execute InsertIntoHadoopFsRelationCommand (8)
+- WriteFiles (7)
   +- * Sort (6)
      +- Project (5)
         +- * Generate (4)
            +- Project (3)
               +- Filter (2)
                  +- Scan text  (1)


(1) Scan text 
Output [1]: [value#294521]
Batched: false
Location: InMemoryFileIndex [file:/data/input/depot/json/tmp/INFORMATION_STA_119302144/HTTP_FILE_0.json, ... 29 entries]
ReadSchema: struct<value:string>

(2) Filter
Input [1]: [value#294521]
Condition : ((size(from_json(ArrayType(StructType(StructField(capacity,LongType,true),StructField(coordonnees_geo,StructType(StructField(lat,DoubleType,true),StructField(lon,DoubleType,true)),true),StructField(name,StringType,true),StructField(station_opening_hours,StringType,true),StructField(stationcode,StringType,true)),true), get_json_object(value#294521, $.results), Some(Etc/UTC)), true) > 0) AND isnotnull(from_json(ArrayType(StructType(StructField(capacity,LongType,true),StructField(coordonnees_geo,StructType(StructField(lat,DoubleType,true),StructField(lon,DoubleType,true)),true),StructField(name,StringType,true),StructField(station_opening_hours,StringType,true),StructField(stationcode,StringType,true)),true), get_json_object(value#294521, $.results), Some(Etc/UTC))))

(3) Project
Output [1]: [from_json(ArrayType(StructType(StructField(capacity,LongType,true),StructField(coordonnees_geo,StructType(StructField(lat,DoubleType,true),StructField(lon,DoubleType,true)),true),StructField(name,StringType,true),StructField(station_opening_hours,StringType,true),StructField(stationcode,StringType,true)),true), get_json_object(value#294521, $.results), Some(Etc/UTC)) AS root#294536]
Input [1]: [value#294521]

(4) Generate [codegen id : 1]
Input [1]: [root#294536]
Arguments: explode(root#294536), false, [root#294539]

(5) Project
Output [6]: [root#294539.capacity AS capacity#294564L, to_json(root#294539.coordonnees_geo, Some(Etc/UTC)) AS coordonnees_geo#294571, root#294539.name AS name#294578, root#294539.station_opening_hours AS station_opening_hours#294585, root#294539.stationcode AS stationcode#294592, 2026-08-31 15:02:37.380552 AS date_566bc84309361176421a95193a0eb006#294599]
Input [1]: [root#294539]

(6) Sort [codegen id : 2]
Input [6]: [capacity#294564L, coordonnees_geo#294571, name#294578, station_opening_hours#294585, stationcode#294592, date_566bc84309361176421a95193a0eb006#294599]
Arguments: [date_566bc84309361176421a95193a0eb006#294599 ASC NULLS FIRST], false, 0

(7) WriteFiles
Input [6]: [capacity#294564L, coordonnees_geo#294571, name#294578, station_opening_hours#294585, stationcode#294592, date_566bc84309361176421a95193a0eb006#294599]

(8) Execute InsertIntoHadoopFsRelationCommand
Input: []
Arguments: file:/data/input/depot/json/execution/INFORMATION_STA_119302/extraction, false, [date_566bc84309361176421a95193a0eb006#294599], Parquet, [__partition_columns=["date_566bc84309361176421a95193a0eb006"], path=file:///data/input/depot/json/execution/INFORMATION_STA_119302/extraction], Append, [capacity, coordonnees_geo, name, station_opening_hours, stationcode, date_566bc84309361176421a95193a0eb006]

SQL / DataFrame Properties

Name	Value
spark.sql.optimizer.nestedPredicatePushdown.supportedFileSources	parquet,orc,geoparquet