SP1 - Details for Query 424

Submitted Time: 2026/04/22 08:56:22
Duration: 0.2 s
Succeeded Jobs: 458

Show the Stage ID and Task ID that corresponds to the max metric

digraph G { 0 [labelType="html" label=" CollectLimit "]; subgraph cluster1 { isCluster="true"; label="WholeStageCodegen (1)\n \nduration: 13 ms"; 2 [labelType="html" label=" Project "]; 3 [labelType="html" label="ColumnarToRow number of output rows: 16 number of input batches: 1"]; } 4 [labelType="html" label="Scan parquet number of files read: 1 scan time: 11 ms metadata time: 0 ms size of files read: 51.9 KiB number of output rows: 16"]; 2->0; 3->2; 4->3; }

CollectLimit 1500

Project [colonne#135097 AS COL_COLONNE_392304#136058, dtype_pandas#135098 AS COL_DTYPE_PANDAS_392305#136059, type_infere#135099 AS COL_TYPE_INFERE_392306#136060, cast(REFORMAT_FOR_DOUBLE(analyse__nb_total#135100) as double) AS COL_ANALYSE_NB_TOTA_392307#136061, cast(REFORMAT_FOR_DOUBLE(analyse__nb_null_nan#135101) as double) AS COL_ANALYSE_NB_NULL_392308#136062, analyse__pct_null_nan#135102 AS COL_ANALYSE_PCT_NUL_392309#136063, cast(REFORMAT_FOR_DOUBLE(analyse__nb_non_null#135103) as double) AS COL_ANALYSE_NB_NON__392310#136064, cast(REFORMAT_FOR_DOUBLE(analyse__nb_distinct#135104) as double) AS COL_ANALYSE_NB_DIST_392311#136065, analyse__nb_vide#135105 AS COL_ANALYSE_NB_VIDE_392312#136066, analyse__pct_vide#135106 AS COL_ANALYSE_PCT_VID_392313#136067, analyse__nb_whitespace_only#135107 AS COL_ANALYSE_NB_WHIT_392314#136068, analyse__nb_leading_spaces#135108 AS COL_ANALYSE_NB_LEAD_392315#136069, analyse__nb_trailing_spaces#135109 AS COL_ANALYSE_NB_TRAI_392316#136070, analyse__nb_leading_et_trailing#135110 AS COL_ANALYSE_NB_LEAD_392317#136071, analyse__longueur_min#135111 AS COL_ANALYSE_LONGUEU_392318#136072, analyse__longueur_max#135112 AS COL_ANALYSE_LONGUEU_392319#136073, analyse__longueur_moyenne#135113 AS COL_ANALYSE_LONGUEU_392320#136074, analyse__longueur_mediane#135114 AS COL_ANALYSE_LONGUEU_392321#136075, analyse__longueur_ecart_type#135115 AS COL_ANALYSE_LONGUEU_392322#136076, analyse__valeur_min_alpha#135116 AS COL_ANALYSE_VALEUR__392323#136077, analyse__valeur_max_alpha#135117 AS COL_ANALYSE_VALEUR__392324#136078, analyse__nb_avec_chiffre#135118 AS COL_ANALYSE_NB_AVEC_392325#136079, analyse__nb_avec_special#135119 AS COL_ANALYSE_NB_AVEC_392326#136080, analyse__nb_uppercase_only#135120 AS COL_ANALYSE_NB_UPPE_392327#136081, ... 36 more fields]

ColumnarToRow

WholeStageCodegen (1)

FileScan parquet [colonne#135097,dtype_pandas#135098,type_infere#135099,analyse__nb_total#135100,analyse__nb_null_nan#135101,analyse__pct_null_nan#135102,analyse__nb_non_null#135103,analyse__nb_distinct#135104,analyse__nb_vide#135105,analyse__pct_vide#135106,analyse__nb_whitespace_only#135107,analyse__nb_leading_spaces#135108,analyse__nb_trailing_spaces#135109,analyse__nb_leading_et_trailing#135110,analyse__longueur_min#135111,analyse__longueur_max#135112,analyse__longueur_moyenne#135113,analyse__longueur_mediane#135114,analyse__longueur_ecart_type#135115,analyse__valeur_min_alpha#135116,analyse__valeur_max_alpha#135117,analyse__nb_avec_chiffre#135118,analyse__nb_avec_special#135119,analyse__nb_uppercase_only#135120,... 36 more fields] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/data/input/depot/parquet/execution/BIO_PAT_S_API_QUALITY_REPORT/..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<colonne:string,dtype_pandas:string,type_infere:string,analyse__nb_total:decimal(4,0),analy...

Details

== Physical Plan ==
CollectLimit (4)
+- * Project (3)
   +- * ColumnarToRow (2)
      +- Scan parquet  (1)


(1) Scan parquet 
Output [60]: [colonne#135097, dtype_pandas#135098, type_infere#135099, analyse__nb_total#135100, analyse__nb_null_nan#135101, analyse__pct_null_nan#135102, analyse__nb_non_null#135103, analyse__nb_distinct#135104, analyse__nb_vide#135105, analyse__pct_vide#135106, analyse__nb_whitespace_only#135107, analyse__nb_leading_spaces#135108, analyse__nb_trailing_spaces#135109, analyse__nb_leading_et_trailing#135110, analyse__longueur_min#135111, analyse__longueur_max#135112, analyse__longueur_moyenne#135113, analyse__longueur_mediane#135114, analyse__longueur_ecart_type#135115, analyse__valeur_min_alpha#135116, analyse__valeur_max_alpha#135117, analyse__nb_avec_chiffre#135118, analyse__nb_avec_special#135119, analyse__nb_uppercase_only#135120, analyse__nb_lowercase_only#135121, analyse__nb_mixte_case#135122, analyse__top_10_valeurs#135123, analyse__min#135124, analyse__max#135125, analyse__etendue#135126, analyse__moyenne#135127, analyse__mediane#135128, analyse__variance#135129, analyse__ecart_type#135130, analyse__skewness#135131, analyse__kurtosis#135132, analyse__q1#135133, analyse__mediane_q2#135134, analyse__q3#135135, analyse__iqr#135136, analyse__nb_outliers_iqr#135137, analyse__pct_outliers_iqr#135138, analyse__nb_zero#135139, analyse__nb_negatif#135140, analyse__distribution_10_bins#135141, analyse__datetime_min#135142, analyse__datetime_max#135143, analyse__etendue_jours#135144, analyse__etendue_secondes_total#135145, analyse__annee_min#135146, analyse__annee_max#135147, analyse__timezone#135148, analyse__nb_weekend#135149, analyse__nb_semaine#135150, analyse__distribution_annee#135151, analyse__distribution_mois#135152, analyse__distribution_heure#135153, analyse__distribution_jour_semaine#135154, Source#135155, horodatage-traitement#135156]
Batched: true
Location: InMemoryFileIndex [file:/data/input/depot/parquet/execution/BIO_PAT_S_API_QUALITY_REPORT/current/pydc_s6098ada.parquet]
ReadSchema: struct<colonne:string,dtype_pandas:string,type_infere:string,analyse__nb_total:decimal(4,0),analyse__nb_null_nan:decimal(3,0),analyse__pct_null_nan:double,analyse__nb_non_null:decimal(4,0),analyse__nb_distinct:decimal(4,0),analyse__nb_vide:double,analyse__pct_vide:double,analyse__nb_whitespace_only:double,analyse__nb_leading_spaces:double,analyse__nb_trailing_spaces:double,analyse__nb_leading_et_trailing:double,analyse__longueur_min:double,analyse__longueur_max:double,analyse__longueur_moyenne:double,analyse__longueur_mediane:double,analyse__longueur_ecart_type:double,analyse__valeur_min_alpha:string,analyse__valeur_max_alpha:string,analyse__nb_avec_chiffre:double,analyse__nb_avec_special:double,analyse__nb_uppercase_only:double,analyse__nb_lowercase_only:double,analyse__nb_mixte_case:double,analyse__top_10_valeurs:string,analyse__min:double,analyse__max:double,analyse__etendue:double,analyse__moyenne:double,analyse__mediane:double,analyse__variance:double,analyse__ecart_type:double,analyse__skewness:double,analyse__kurtosis:double,analyse__q1:double,analyse__mediane_q2:double,analyse__q3:double,analyse__iqr:double,analyse__nb_outliers_iqr:double,analyse__pct_outliers_iqr:double,analyse__nb_zero:double,analyse__nb_negatif:double,analyse__distribution_10_bins:string,analyse__datetime_min:string,analyse__datetime_max:string,analyse__etendue_jours:double,analyse__etendue_secondes_total:double,analyse__annee_min:double,analyse__annee_max:double,analyse__timezone:string,analyse__nb_weekend:double,analyse__nb_semaine:double,analyse__distribution_annee:string,analyse__distribution_mois:string,analyse__distribution_heure:string,analyse__distribution_jour_semaine:string,Source:string,horodatage-traitement:timestamp_ntz>

(2) ColumnarToRow [codegen id : 1]
Input [60]: [colonne#135097, dtype_pandas#135098, type_infere#135099, analyse__nb_total#135100, analyse__nb_null_nan#135101, analyse__pct_null_nan#135102, analyse__nb_non_null#135103, analyse__nb_distinct#135104, analyse__nb_vide#135105, analyse__pct_vide#135106, analyse__nb_whitespace_only#135107, analyse__nb_leading_spaces#135108, analyse__nb_trailing_spaces#135109, analyse__nb_leading_et_trailing#135110, analyse__longueur_min#135111, analyse__longueur_max#135112, analyse__longueur_moyenne#135113, analyse__longueur_mediane#135114, analyse__longueur_ecart_type#135115, analyse__valeur_min_alpha#135116, analyse__valeur_max_alpha#135117, analyse__nb_avec_chiffre#135118, analyse__nb_avec_special#135119, analyse__nb_uppercase_only#135120, analyse__nb_lowercase_only#135121, analyse__nb_mixte_case#135122, analyse__top_10_valeurs#135123, analyse__min#135124, analyse__max#135125, analyse__etendue#135126, analyse__moyenne#135127, analyse__mediane#135128, analyse__variance#135129, analyse__ecart_type#135130, analyse__skewness#135131, analyse__kurtosis#135132, analyse__q1#135133, analyse__mediane_q2#135134, analyse__q3#135135, analyse__iqr#135136, analyse__nb_outliers_iqr#135137, analyse__pct_outliers_iqr#135138, analyse__nb_zero#135139, analyse__nb_negatif#135140, analyse__distribution_10_bins#135141, analyse__datetime_min#135142, analyse__datetime_max#135143, analyse__etendue_jours#135144, analyse__etendue_secondes_total#135145, analyse__annee_min#135146, analyse__annee_max#135147, analyse__timezone#135148, analyse__nb_weekend#135149, analyse__nb_semaine#135150, analyse__distribution_annee#135151, analyse__distribution_mois#135152, analyse__distribution_heure#135153, analyse__distribution_jour_semaine#135154, Source#135155, horodatage-traitement#135156]

(3) Project [codegen id : 1]
Output [60]: [colonne#135097 AS COL_COLONNE_392304#136058, dtype_pandas#135098 AS COL_DTYPE_PANDAS_392305#136059, type_infere#135099 AS COL_TYPE_INFERE_392306#136060, cast(REFORMAT_FOR_DOUBLE(analyse__nb_total#135100) as double) AS COL_ANALYSE_NB_TOTA_392307#136061, cast(REFORMAT_FOR_DOUBLE(analyse__nb_null_nan#135101) as double) AS COL_ANALYSE_NB_NULL_392308#136062, analyse__pct_null_nan#135102 AS COL_ANALYSE_PCT_NUL_392309#136063, cast(REFORMAT_FOR_DOUBLE(analyse__nb_non_null#135103) as double) AS COL_ANALYSE_NB_NON__392310#136064, cast(REFORMAT_FOR_DOUBLE(analyse__nb_distinct#135104) as double) AS COL_ANALYSE_NB_DIST_392311#136065, analyse__nb_vide#135105 AS COL_ANALYSE_NB_VIDE_392312#136066, analyse__pct_vide#135106 AS COL_ANALYSE_PCT_VID_392313#136067, analyse__nb_whitespace_only#135107 AS COL_ANALYSE_NB_WHIT_392314#136068, analyse__nb_leading_spaces#135108 AS COL_ANALYSE_NB_LEAD_392315#136069, analyse__nb_trailing_spaces#135109 AS COL_ANALYSE_NB_TRAI_392316#136070, analyse__nb_leading_et_trailing#135110 AS COL_ANALYSE_NB_LEAD_392317#136071, analyse__longueur_min#135111 AS COL_ANALYSE_LONGUEU_392318#136072, analyse__longueur_max#135112 AS COL_ANALYSE_LONGUEU_392319#136073, analyse__longueur_moyenne#135113 AS COL_ANALYSE_LONGUEU_392320#136074, analyse__longueur_mediane#135114 AS COL_ANALYSE_LONGUEU_392321#136075, analyse__longueur_ecart_type#135115 AS COL_ANALYSE_LONGUEU_392322#136076, analyse__valeur_min_alpha#135116 AS COL_ANALYSE_VALEUR__392323#136077, analyse__valeur_max_alpha#135117 AS COL_ANALYSE_VALEUR__392324#136078, analyse__nb_avec_chiffre#135118 AS COL_ANALYSE_NB_AVEC_392325#136079, analyse__nb_avec_special#135119 AS COL_ANALYSE_NB_AVEC_392326#136080, analyse__nb_uppercase_only#135120 AS COL_ANALYSE_NB_UPPE_392327#136081, analyse__nb_lowercase_only#135121 AS COL_ANALYSE_NB_LOWE_392328#136082, analyse__nb_mixte_case#135122 AS COL_ANALYSE_NB_MIXT_392329#136083, analyse__top_10_valeurs#135123 AS COL_ANALYSE_TOP_10__392330#136084, analyse__min#135124 AS COL_ANALYSE_MIN_392331#136085, analyse__max#135125 AS COL_ANALYSE_MAX_392332#136086, analyse__etendue#135126 AS COL_ANALYSE_ETENDUE_392333#136087, analyse__moyenne#135127 AS COL_ANALYSE_MOYENNE_392334#136088, analyse__mediane#135128 AS COL_ANALYSE_MEDIANE_392335#136089, analyse__variance#135129 AS COL_ANALYSE_VARIANC_392336#136090, analyse__ecart_type#135130 AS COL_ANALYSE_ECART_T_392337#136091, analyse__skewness#135131 AS COL_ANALYSE_SKEWNES_392338#136092, analyse__kurtosis#135132 AS COL_ANALYSE_KURTOSI_392339#136093, analyse__q1#135133 AS COL_ANALYSE_Q1_392340#136094, analyse__mediane_q2#135134 AS COL_ANALYSE_MEDIANE_392341#136095, analyse__q3#135135 AS COL_ANALYSE_Q3_392342#136096, analyse__iqr#135136 AS COL_ANALYSE_IQR_392343#136097, analyse__nb_outliers_iqr#135137 AS COL_ANALYSE_NB_OUTL_392344#136098, analyse__pct_outliers_iqr#135138 AS COL_ANALYSE_PCT_OUT_392345#136099, analyse__nb_zero#135139 AS COL_ANALYSE_NB_ZERO_392346#136100, analyse__nb_negatif#135140 AS COL_ANALYSE_NB_NEGA_392347#136101, analyse__distribution_10_bins#135141 AS COL_ANALYSE_DISTRIB_392348#136102, analyse__datetime_min#135142 AS COL_ANALYSE_DATETIM_392349#136103, analyse__datetime_max#135143 AS COL_ANALYSE_DATETIM_392350#136104, analyse__etendue_jours#135144 AS COL_ANALYSE_ETENDUE_392351#136105, analyse__etendue_secondes_total#135145 AS COL_ANALYSE_ETENDUE_392352#136106, analyse__annee_min#135146 AS COL_ANALYSE_ANNEE_M_392353#136107, analyse__annee_max#135147 AS COL_ANALYSE_ANNEE_M_392354#136108, analyse__timezone#135148 AS COL_ANALYSE_TIMEZON_392355#136109, analyse__nb_weekend#135149 AS COL_ANALYSE_NB_WEEK_392356#136110, analyse__nb_semaine#135150 AS COL_ANALYSE_NB_SEMA_392357#136111, analyse__distribution_annee#135151 AS COL_ANALYSE_DISTRIB_392358#136112, analyse__distribution_mois#135152 AS COL_ANALYSE_DISTRIB_392359#136113, analyse__distribution_heure#135153 AS COL_ANALYSE_DISTRIB_392360#136114, analyse__distribution_jour_semaine#135154 AS COL_ANALYSE_DISTRIB_392361#136115, Source#135155 AS COL_SOURCE_392362#136116, cast(horodatage-traitement#135156 as string) AS COL_HORODATAGE_TRAI_392363#136117]
Input [60]: [colonne#135097, dtype_pandas#135098, type_infere#135099, analyse__nb_total#135100, analyse__nb_null_nan#135101, analyse__pct_null_nan#135102, analyse__nb_non_null#135103, analyse__nb_distinct#135104, analyse__nb_vide#135105, analyse__pct_vide#135106, analyse__nb_whitespace_only#135107, analyse__nb_leading_spaces#135108, analyse__nb_trailing_spaces#135109, analyse__nb_leading_et_trailing#135110, analyse__longueur_min#135111, analyse__longueur_max#135112, analyse__longueur_moyenne#135113, analyse__longueur_mediane#135114, analyse__longueur_ecart_type#135115, analyse__valeur_min_alpha#135116, analyse__valeur_max_alpha#135117, analyse__nb_avec_chiffre#135118, analyse__nb_avec_special#135119, analyse__nb_uppercase_only#135120, analyse__nb_lowercase_only#135121, analyse__nb_mixte_case#135122, analyse__top_10_valeurs#135123, analyse__min#135124, analyse__max#135125, analyse__etendue#135126, analyse__moyenne#135127, analyse__mediane#135128, analyse__variance#135129, analyse__ecart_type#135130, analyse__skewness#135131, analyse__kurtosis#135132, analyse__q1#135133, analyse__mediane_q2#135134, analyse__q3#135135, analyse__iqr#135136, analyse__nb_outliers_iqr#135137, analyse__pct_outliers_iqr#135138, analyse__nb_zero#135139, analyse__nb_negatif#135140, analyse__distribution_10_bins#135141, analyse__datetime_min#135142, analyse__datetime_max#135143, analyse__etendue_jours#135144, analyse__etendue_secondes_total#135145, analyse__annee_min#135146, analyse__annee_max#135147, analyse__timezone#135148, analyse__nb_weekend#135149, analyse__nb_semaine#135150, analyse__distribution_annee#135151, analyse__distribution_mois#135152, analyse__distribution_heure#135153, analyse__distribution_jour_semaine#135154, Source#135155, horodatage-traitement#135156]

(4) CollectLimit
Input [60]: [COL_COLONNE_392304#136058, COL_DTYPE_PANDAS_392305#136059, COL_TYPE_INFERE_392306#136060, COL_ANALYSE_NB_TOTA_392307#136061, COL_ANALYSE_NB_NULL_392308#136062, COL_ANALYSE_PCT_NUL_392309#136063, COL_ANALYSE_NB_NON__392310#136064, COL_ANALYSE_NB_DIST_392311#136065, COL_ANALYSE_NB_VIDE_392312#136066, COL_ANALYSE_PCT_VID_392313#136067, COL_ANALYSE_NB_WHIT_392314#136068, COL_ANALYSE_NB_LEAD_392315#136069, COL_ANALYSE_NB_TRAI_392316#136070, COL_ANALYSE_NB_LEAD_392317#136071, COL_ANALYSE_LONGUEU_392318#136072, COL_ANALYSE_LONGUEU_392319#136073, COL_ANALYSE_LONGUEU_392320#136074, COL_ANALYSE_LONGUEU_392321#136075, COL_ANALYSE_LONGUEU_392322#136076, COL_ANALYSE_VALEUR__392323#136077, COL_ANALYSE_VALEUR__392324#136078, COL_ANALYSE_NB_AVEC_392325#136079, COL_ANALYSE_NB_AVEC_392326#136080, COL_ANALYSE_NB_UPPE_392327#136081, COL_ANALYSE_NB_LOWE_392328#136082, COL_ANALYSE_NB_MIXT_392329#136083, COL_ANALYSE_TOP_10__392330#136084, COL_ANALYSE_MIN_392331#136085, COL_ANALYSE_MAX_392332#136086, COL_ANALYSE_ETENDUE_392333#136087, COL_ANALYSE_MOYENNE_392334#136088, COL_ANALYSE_MEDIANE_392335#136089, COL_ANALYSE_VARIANC_392336#136090, COL_ANALYSE_ECART_T_392337#136091, COL_ANALYSE_SKEWNES_392338#136092, COL_ANALYSE_KURTOSI_392339#136093, COL_ANALYSE_Q1_392340#136094, COL_ANALYSE_MEDIANE_392341#136095, COL_ANALYSE_Q3_392342#136096, COL_ANALYSE_IQR_392343#136097, COL_ANALYSE_NB_OUTL_392344#136098, COL_ANALYSE_PCT_OUT_392345#136099, COL_ANALYSE_NB_ZERO_392346#136100, COL_ANALYSE_NB_NEGA_392347#136101, COL_ANALYSE_DISTRIB_392348#136102, COL_ANALYSE_DATETIM_392349#136103, COL_ANALYSE_DATETIM_392350#136104, COL_ANALYSE_ETENDUE_392351#136105, COL_ANALYSE_ETENDUE_392352#136106, COL_ANALYSE_ANNEE_M_392353#136107, COL_ANALYSE_ANNEE_M_392354#136108, COL_ANALYSE_TIMEZON_392355#136109, COL_ANALYSE_NB_WEEK_392356#136110, COL_ANALYSE_NB_SEMA_392357#136111, COL_ANALYSE_DISTRIB_392358#136112, COL_ANALYSE_DISTRIB_392359#136113, COL_ANALYSE_DISTRIB_392360#136114, COL_ANALYSE_DISTRIB_392361#136115, COL_SOURCE_392362#136116, COL_HORODATAGE_TRAI_392363#136117]
Arguments: 1500

SQL / DataFrame Properties

Name	Value
spark.sql.optimizer.nestedPredicatePushdown.supportedFileSources	parquet,orc,geoparquet