This article explores how logging is integrated into the pipeline, referencing relevant Python scripts in dependencies/logging_utils, Hydra configs under configs/logging_utils, and how each stage’s logs tie back to DVC and MLflow runs.
Introduction
Logging is essential for traceability, debugging, and performance monitoring within an MLOps pipeline. When multiple components—Hydra configuration management, DVC versioning, Optuna hyperparameter optimization, MLflow experiment tracking, and tools like Prefect—must interoperate, logging becomes the glue that keeps the entire process auditable and reproducible. This article explores how logging is integrated into the pipeline, referencing relevant Python scripts in dependencies/logging_utils, Hydra configs under configs/logging_utils, and how each stage’s logs tie back to DVC and MLflow runs.
configs/logging_utils/base.yaml stores default logging parameters (log_file_path, formatting, level). By loading these configurations through Hydra, the pipeline ensures every step—whether a data ingestion script or an Optuna trial—follows consistent logging conventions.
# configs/logging_utils/base.yaml
defaults:
- paths: base
- _self_
log_directory_path: "${paths.directories.logs}/pipeline"
log_file_path: "${.log_directory_path}/${now:%Y-%m-%d_%H-%M-%S}.log"
level: 20
formatter: "%(asctime)s %(levelname)s:%(message)s"
Tools like Prefect and MLflow also have their own logging mechanisms. By funneling all logs into a single log directory (and optionally shipping them to a centralized system like Elasticsearch or Splunk), the pipeline avoids fragmentation. Meanwhile, environment variables and Hydra overrides let you switch log levels (for example, from INFO to DEBUG) at runtime without altering code.
Each pipeline step logs relevant events (for example, reading a CSV, computing file hashes, applying transformations). DVC references these logs as artifacts whenever a step is re-run. The script scripts/universal_step.py initializes logging by calling dependencies/logging_utils/setup_logging.py. This ensures consistent logging across data ingestion, cleaning, feature engineering, and model training stages.
DVC’s dvc repro triggers steps in sequence; the logs from each step get appended or stored in a new timestamped file, depending on your Hydra config. Because each stage is versioned, rolling back to a prior commit recovers both the scripts and the logs at that point in time.
[2025-03-21 16:40:29,477][dependencies.general.mkdir_if_not_exists][INFO] - Directory exists, skipping creation
/Users/tobias/.local/projects/portfolio_medical_drg_ny/logs/pipeline
[2025-03-21 16:40:30,476][dependencies.io.csv_to_dataframe][INFO] - Read /Users/tobias/.local/projects/portfolio_medical_drg_ny/data/v7/v7.csv, created df
[2025-03-21 16:54:21,317][dependencies.transformations.agg_severities][INFO] - Done with core transformation: agg_severities
[2025-03-21 16:54:21,325][dependencies.general.mkdir_if_not_exists][INFO] - Directory exists, skipping creation
/Users/tobias/.local/projects/portfolio_medical_drg_ny/data/v8
[2025-03-21 16:54:24,451][dependencies.io.dataframe_to_csv][INFO] - Exported df to csv using filepath: /Users/tobias/.local/projects/portfolio_medical_drg_ny/data/v8/v8.csv
[2025-03-21 16:54:27,709][dependencies.metadata.compute_file_hash][INFO] - Generated file hash: 0cee898257560d8e67d10b502b136054d5340a30fa1836d59e40cc53cbd45144
[2025-03-21 16:54:27,857][dependencies.metadata.calculate_metadata][INFO] - Generated metadata for file: /Users/tobias/.local/projects/portfolio_medical_drg_ny/data/v8/v8.csv
[2025-03-21 16:54:27,857][dependencies.metadata.calculate_metadata][INFO] - Metadata successfully saved to /Users/tobias/.local/projects/portfolio_medical_drg_ny/data/v8/v8_metadata.json
[2025-03-21 16:54:27,857][__main__][INFO] - Sucessfully executed step: agg_severities
When running hyperparameter tuning with Optuna (for example, dependencies/modeling/rf_optuna_trial.py), the pipeline logs each trial’s metrics to MLflow. dependencies/logging_utils/calculate_and_log_importances_as_artifact.py demonstrates how model artifacts (such as feature importances) are logged as well.
Each trial also streams standard log messages (for example, “Trial 1 => RMSE=764781.853”) to the console and the .log file, allowing real-time monitoring. By referencing the unique MLflow run ID in logs, it’s straightforward to cross-check experiment results with pipeline events.
Prefect tasks can leverage the built-in prefect.context.get_run_logger() for simpler orchestration, but in many teams, a shared Python logger is used to unify logs across both Prefect tasks and local scripts. By pulling configuration from Hydra at the start of each flow, you can maintain a consistent format, level, and output location. This avoids the pitfall of scattering logs into separate systems.
Inconsistent Run Identifiers
Failing to tie logs to unique run IDs or MLflow experiment tags leads to confusion about which logs pertain to which runs.
Overly Verbose Logging
Dumping entire dataframes or large model objects floods your log system, making it hard to find meaningful events.
Lack of Structure
Using free-form strings instead of structured logging (JSON or key-value pairs) complicates downstream parsing.
Ignoring Config Changes
Neglecting to log Hydra overrides or environment variable changes can destroy reproducibility.
Forgetting Error Handling
Not capturing stack traces or skipping logging on exceptions can hamper debugging.
Structured and Level-Based
Using log levels (INFO, DEBUG, WARNING, ERROR) consistently. One can consider a JSON format if robust search and filtering capabilities are needed.
Centralization
It is important to centralize logs in a known directory or logging service; Unifying logs from local runs, MLflow, Optuna, Prefect, etc.
Tie Logs to Pipeline Artifacts
It is good practice to ensure each run is associated with a Hydra config, a DVC stage, and an MLflow run ID. This single run ID is the backbone of reproducibility.
Minimal Noise
Using appropriate log levels means only major transformations, hyperparameters, and relevant errors get logged, not every micro-step.
Retention
Decide how long logs remain accessible, especially if your pipeline is regularly re-run. For large volumes, adopt efficient rotation or archiving.
A robust logging strategy is an essential pillar for any MLOps pipeline. By pairing consistent Hydra-based configs with DVC tracking, MLflow experiment logging, and optional Prefect orchestration, teams ensure they capture all details necessary to debug, audit, and refine each stage of the project. Combined with unique run identifiers, structured logging, and a disciplined approach to error handling, this pipeline design yields highly traceable and reproducible machine learning workflows.