hdh-spark-job - Details for Query 1391

Submitted Time: 2026/08/31 15:02:49
Duration: 0.3 s
Succeeded Jobs: 1804

Show the Stage ID and Task ID that corresponds to the max metric

digraph G { 0 [labelType="html" label="Execute InsertIntoHadoopFsRelationCommand task commit time total (min, med, max (stageId: taskId)) 12 ms (0 ms, 1 ms, 7 ms (stage 2706.0: task 21536)) number of written files: 6 job commit time: 6 ms number of output rows: 1,519 number of dynamic part: 1 written output: 89.7 KiB"]; 1 [labelType="html" label=" WriteFiles "]; subgraph cluster2 { isCluster="true"; label="WholeStageCodegen (2)\n \nduration: total (min, med, max (stageId: taskId))\n215 ms (8 ms, 30 ms, 37 ms (stage 2706.0: task 21537))"; 3 [labelType="html" label="Sort sort time total (min, med, max (stageId: taskId)) 0 ms (0 ms, 0 ms, 0 ms (stage 2706.0: task 21537)) peak memory total (min, med, max (stageId: taskId)) 384.6 MiB (64.0 KiB, 64.1 MiB, 64.1 MiB (stage 2706.0: task 21537)) spill size total (min, med, max (stageId: taskId)) 0.0 B (0.0 B, 0.0 B, 0.0 B (stage 2706.0: task 21537))"]; } 4 [labelType="html" label=" Project "]; subgraph cluster5 { isCluster="true"; label="WholeStageCodegen (1)\n \nduration: total (min, med, max (stageId: taskId))\n100 ms (5 ms, 13 ms, 18 ms (stage 2706.0: task 21537))"; 6 [labelType="html" label="Generate number of output rows: 1,519"]; } 7 [labelType="html" label=" Project "]; 8 [labelType="html" label="Filter number of output rows: 16"]; 9 [labelType="html" label="Scan text number of output rows: 30 number of files read: 30 metadata time: 0 ms size of files read: 268.0 KiB"]; 1->0; 3->1; 4->3; 6->4; 7->6; 8->7; 9->8; }

Execute InsertIntoHadoopFsRelationCommand file:/data/input/depot/json/execution/INFORMATION_STA_119302/extraction, false, [date_566bc84309361176421a95193a0eb006#294691], Parquet, [__partition_columns=["date_566bc84309361176421a95193a0eb006"], path=file:///data/input/depot/json/execution/INFORMATION_STA_119302/extraction], Append, [capacity, coordonnees_geo, name, station_opening_hours, stationcode, date_566bc84309361176421a95193a0eb006]

WriteFiles

Sort [date_566bc84309361176421a95193a0eb006#294691 ASC NULLS FIRST], false, 0

WholeStageCodegen (2)

Project [root#294631.capacity AS capacity#294656L, to_json(root#294631.coordonnees_geo, Some(Etc/UTC)) AS coordonnees_geo#294663, root#294631.name AS name#294670, root#294631.station_opening_hours AS station_opening_hours#294677, root#294631.stationcode AS stationcode#294684, 2026-08-31 15:02:49.330127 AS date_566bc84309361176421a95193a0eb006#294691]

Generate explode(root#294628), false, [root#294631]

WholeStageCodegen (1)

Project [from_json(ArrayType(StructType(StructField(capacity,LongType,true),StructField(coordonnees_geo,StructType(StructField(lat,DoubleType,true),StructField(lon,DoubleType,true)),true),StructField(name,StringType,true),StructField(station_opening_hours,StringType,true),StructField(stationcode,StringType,true)),true), get_json_object(value#294613, $.results), Some(Etc/UTC)) AS root#294628]

Filter ((size(from_json(ArrayType(StructType(StructField(capacity,LongType,true),StructField(coordonnees_geo,StructType(StructField(lat,DoubleType,true),StructField(lon,DoubleType,true)),true),StructField(name,StringType,true),StructField(station_opening_hours,StringType,true),StructField(stationcode,StringType,true)),true), get_json_object(value#294613, $.results), Some(Etc/UTC)), true) > 0) AND isnotnull(from_json(ArrayType(StructType(StructField(capacity,LongType,true),StructField(coordonnees_geo,StructType(StructField(lat,DoubleType,true),StructField(lon,DoubleType,true)),true),StructField(name,StringType,true),StructField(station_opening_hours,StringType,true),StructField(stationcode,StringType,true)),true), get_json_object(value#294613, $.results), Some(Etc/UTC))))

FileScan text [value#294613] Batched: false, DataFilters: [(size(from_json(ArrayType(StructType(StructField(capacity,LongType,true),StructField(coordonnees..., Format: Text, Location: InMemoryFileIndex(30 paths)[file:/data/input/depot/json/tmp/INFORMATION_STA_119302143/HTTP_FILE_0..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<value:string>

Details

== Physical Plan ==
Execute InsertIntoHadoopFsRelationCommand (8)
+- WriteFiles (7)
   +- * Sort (6)
      +- Project (5)
         +- * Generate (4)
            +- Project (3)
               +- Filter (2)
                  +- Scan text  (1)


(1) Scan text 
Output [1]: [value#294613]
Batched: false
Location: InMemoryFileIndex [file:/data/input/depot/json/tmp/INFORMATION_STA_119302143/HTTP_FILE_0.json, ... 29 entries]
ReadSchema: struct<value:string>

(2) Filter
Input [1]: [value#294613]
Condition : ((size(from_json(ArrayType(StructType(StructField(capacity,LongType,true),StructField(coordonnees_geo,StructType(StructField(lat,DoubleType,true),StructField(lon,DoubleType,true)),true),StructField(name,StringType,true),StructField(station_opening_hours,StringType,true),StructField(stationcode,StringType,true)),true), get_json_object(value#294613, $.results), Some(Etc/UTC)), true) > 0) AND isnotnull(from_json(ArrayType(StructType(StructField(capacity,LongType,true),StructField(coordonnees_geo,StructType(StructField(lat,DoubleType,true),StructField(lon,DoubleType,true)),true),StructField(name,StringType,true),StructField(station_opening_hours,StringType,true),StructField(stationcode,StringType,true)),true), get_json_object(value#294613, $.results), Some(Etc/UTC))))

(3) Project
Output [1]: [from_json(ArrayType(StructType(StructField(capacity,LongType,true),StructField(coordonnees_geo,StructType(StructField(lat,DoubleType,true),StructField(lon,DoubleType,true)),true),StructField(name,StringType,true),StructField(station_opening_hours,StringType,true),StructField(stationcode,StringType,true)),true), get_json_object(value#294613, $.results), Some(Etc/UTC)) AS root#294628]
Input [1]: [value#294613]

(4) Generate [codegen id : 1]
Input [1]: [root#294628]
Arguments: explode(root#294628), false, [root#294631]

(5) Project
Output [6]: [root#294631.capacity AS capacity#294656L, to_json(root#294631.coordonnees_geo, Some(Etc/UTC)) AS coordonnees_geo#294663, root#294631.name AS name#294670, root#294631.station_opening_hours AS station_opening_hours#294677, root#294631.stationcode AS stationcode#294684, 2026-08-31 15:02:49.330127 AS date_566bc84309361176421a95193a0eb006#294691]
Input [1]: [root#294631]

(6) Sort [codegen id : 2]
Input [6]: [capacity#294656L, coordonnees_geo#294663, name#294670, station_opening_hours#294677, stationcode#294684, date_566bc84309361176421a95193a0eb006#294691]
Arguments: [date_566bc84309361176421a95193a0eb006#294691 ASC NULLS FIRST], false, 0

(7) WriteFiles
Input [6]: [capacity#294656L, coordonnees_geo#294663, name#294670, station_opening_hours#294677, stationcode#294684, date_566bc84309361176421a95193a0eb006#294691]

(8) Execute InsertIntoHadoopFsRelationCommand
Input: []
Arguments: file:/data/input/depot/json/execution/INFORMATION_STA_119302/extraction, false, [date_566bc84309361176421a95193a0eb006#294691], Parquet, [__partition_columns=["date_566bc84309361176421a95193a0eb006"], path=file:///data/input/depot/json/execution/INFORMATION_STA_119302/extraction], Append, [capacity, coordonnees_geo, name, station_opening_hours, stationcode, date_566bc84309361176421a95193a0eb006]

SQL / DataFrame Properties

Name	Value
spark.sql.optimizer.nestedPredicatePushdown.supportedFileSources	parquet,orc,geoparquet