Debugging

Last updated on 2024-11-26 | Edit this page

Overview

Questions

  • How can I handle errors/warnings?

Objectives

  • Fix a broken recipe

Every user encounters errors. Once you know why you get certain types of errors, they become much easier to fix. The good news is that ESMValTool creates a record of the output messages and stores them in log files. They can be used for debugging or monitoring the process. This lesson helps you understand the different types of errors and when you are likely to encounter them.

Log files

Each time we run ESMValTool, it will produce a new output directory. This directory should contain the run folder that is automatically generated by ESMValTool. To examine this, we run a recipe_python.yml that can be found in lesson Running your first recipe . Check lesson Configuration on how to set paths.

In a new terminal, run the recipe:

BASH 

  cd esmvaltool_tutorial
  esmvaltool run examples/recipe_python.yml

ERROR 

esmvaltool: command not found

ESMValTool encounters this error because the conda environment esmvaltool has not been activated. To fix the error, before running the recipe, activate the environment:

BASH 

conda activate esmvaltool

conda environment

More information about the conda environment can be found at Installation.

Let’s list the files in the run directory:

BASH 

  ls esmvaltool_output/recipe_python_#_#/run

OUTPUT 

main_log_debug.txt  main_log.txt  map  recipe_python.yml  resource_usage.txt
timeseries

In the main_log_debug.txt and main_log.txt, ESMValTool writes the output messages, warnings and possible errors that might happen during pre-processings. To inspect them, we can look inside the files. For example:

BASH 

  cat esmvaltool_output/recipe_python_#_#/run/main_log.txt

Now, let’s have a look inside the folder timeseries/script1:

BASH 

  ls esmvaltool_output/recipe_python_#_#/run/timeseries/script1/

OUTPUT 

diagnostic_provenance.yml log.txt  resource_usage.txt  settings.yml

In the log.txt, ESMValTool writes the output messages, warnings and possible errors that are related to the diagnostic script.

If you encounter an error and don’t know what it means, it is important to read the log information. Sometimes knowing where the error occurred is enough to fix it, even if you don’t entirely understand the message. However, note that you may not always be able to find the error or fix it. In that case, ESMValTool community helps you figure out what went wrong.

Different log files

In the run directory, there are two log files main_log_debug.txt and main_log.txt. What are their differences?

The main_log_debug.txt contains the output messages from the pre-processor whereas the main_log.txt shows general errors and warnings that might happen in running the recipe and diagnostics script.

Let’s change some settings in the recipe to run a regional pre-processor. We use a text editor called nano to open the recipe file:

BASH 

  nano recipe_python.yml

Text editor side note

No matter what editor you use, you will need to know where it searches for and saves files. If you start it from the shell, it will (probably) use your current working directory as its default location. We use nano in examples here because it is one of the least complex text editors. Press ctrl + O to save the file, and then ctrl + X to exit nano.

YAML 

# ESMValTool
# recipe_python.yml
---
# See https://docs.esmvaltool.org/en/latest/recipes/recipe_examples.html
# for a description of this recipe.

# See https://docs.esmvaltool.org/projects/esmvalcore/en/latest/recipe/overview.html
# for a description of the recipe format.
---
documentation:
  description: |
    Example recipe that plots a map and timeseries of temperature.

  title: Recipe that runs an example diagnostic written in Python.

  authors:
    - andela_bouwe
    - righi_mattia

  maintainer:
    - schlund_manuel

  references:
    - acknow_project

  projects:
    - esmval
    - c3s-magic

datasets:
  - {dataset: BCC-ESM1, project: CMIP6, exp: historical, ensemble: r1i1p1f1, grid: gn}
  - {dataset: bcc-csm1-1, version: v1, project: CMIP5, exp: historical, ensemble: r1i1p1}

preprocessors:
# See https://docs.esmvaltool.org/projects/esmvalcore/en/latest/recipe/preprocessor.html
# for a description of the preprocessor functions.

  to_degrees_c:
    convert_units:
      units: degrees_C

  annual_mean_amsterdam:
    extract_location:
      location: Amsterdam
      scheme: linear
    annual_statistics:
      operator: mean
    multi_model_statistics:
      statistics:
        - mean
      span: overlap
    convert_units:
      units: degrees_C

  annual_mean_global:
    area_statistics:
      operator: mean
    annual_statistics:
      operator: mean
    convert_units:
      units: degrees_C

diagnostics:

  map:
    description: Global map of temperature in January 2000.
    themes:
      - phys
    realms:
      - atmos
    variables:
      tas:
        mip: Amon
        preprocessor: to_degrees_c
        timerange: 2000/P1M
        caption: |
          Global map of {long_name} in January 2000 according to {dataset}.
    scripts:
      script1:
        script: examples/diagnostic.py
        quickplot:
          plot_type: pcolormesh
          cmap: Reds

  timeseries:
    description: Annual mean temperature in Amsterdam and global mean since 1850.
    themes:
      - phys
    realms:
      - atmos
    variables:
      tas_amsterdam:
        short_name: tas
        mip: Amon
        preprocessor: annual_mean_amsterdam
        timerange: 1850/2000
        caption: Annual mean {long_name} in Amsterdam according to {dataset}.
      tas_global:
        short_name: tas
        mip: Amon
        preprocessor: annual_mean_global
        timerange: 1850/2000
        caption: Annual global mean {long_name} according to {dataset}.
    scripts:
      script1:
        script: examples/diagnostic.py
        quickplot:
          plot_type: plot

Keys and values in recipe settings

The [ESMValTool pre-processors](https://docs.esmvaltool.org/projects/ESMValCore/en/latest/ recipe/preprocessor.html?highlight=preprocessor) cover a broad range of operations on the input data, like time manipulation, area manipulation, land-sea masking, variable derivation, etc. Let’s add the preprocessor extract_region to a new section annual_mean_regional:

YAML 

preprocessors:
  annual_mean_regional:
    annual_statistics:
      operator: mean
    extract_region:
      start_longitude: -10
      end_longitude: 40
      start_latitude: 27
      end_latitude: 70

Also, we change the projects value esmval to tutorial:

YAML 

projects:
  - tutorial
  - c3s-magic

Then, we save the file and run the recipe:

BASH 

  esmvaltool run recipe_python.yml

ERROR 

ValueError: Tag 'tutorial' does not exist in section 'projects' of
esmvaltool/config-references.yml 2020-06-29 18:09:56,641 UTC [46055] INFO If you
have a question or need help, please start a new discussion on
https://github.com/ESMValGroup/ESMValTool/discussions If you suspect this is a
bug, please open an issue on https://github.com/ESMValGroup/ESMValTool/issues To
make it easier to find out what the problem is, please consider attaching the
files run/recipe_*.yml and run/main_log_debug.txt from the output directory.

The values for the keys author, maintainer, projects and references in the recipe should be known by ESMValTool:

ESMValTool can’t locate the data

You are assisting a colleague with ESMValTool. The colleague replaces the CanESM2 entry in dataset: CanESM2, project: CMIP5 to ACCESS1-3 and runs the recipe. However, ESMValTool encounters an error like:

BASH 

ERROR   No input files found for variable {'short_name': 'tas', 'mip': 'Amon',
'preprocessor': 'annual_mean_amsterdam', 'variable_group': 'tas_amsterdam',
'diagnostic': 'timeseries', 'dataset': 'ACCESS1-3', 'project': 'CMIP5', 'exp':
'historical', 'ensemble': 'r1i1p1', 'recipe_dataset_index': 1, 'institute':
['CSIRO-BOM'], 'product': ['output1', 'output2'], 'timerange': '1850/2000',
'alias': 'CMIP5', 'original_short_name': 'tas', 'standard_name':
'air_temperature', 'long_name': 'Near-Surface Air Temperature', 'units': 'K',
'modeling_realm': ['atmos'], 'frequency': 'mon', 'start_year': 1850,
'end_year': 2000}
ERROR   Looked for files matching
['tas_Amon_ACCESS1-3_historical_r1i1p1*.nc'], but did not find any existing
input directory
ERROR   Set 'log_level' to 'debug' to get more information

What suggestions would you give the researcher for fixing the error?

Challenge

  1. Check user-config.yml to see if the correct directory for input data is introduced
  2. Check the available data, regarding exp, mip, ensemble, start_year, and end_year
  3. Check the variable names in the diagnostics section in the recipe

Check pre-processed data

The setting save_intermediary_cubes in the configuration file can be used to save the pre-processed data. More information about this setting can be found at Configuration.

save_intermediary_cubes

Note that this setting should only be used for debugging, as it significantly slows down the recipe and increases disk usage because a lot of output files need to be stored.

Check diagnostic script path

The result of the pre-processor is passed to the examples/diagnostic.py script, that is introduced in the recipe as:

YAML 

scripts:
  script1:
    script:
      script: examples/diagnostic.py

The diagnostic scripts are located in the folder diag_scripts in the ESMValTool installation directory <path_to_esmvaltool>. To find where ESMValTool is located on your system, see Installation .

Let’s see what happens if we can change the script path as:

YAML 

scripts:
  script1:
    script:
      script: diag_scripts/ocean/diagnostic_timeseries.py

BASH 

  esmvaltool run examples/recipe_python.yml

ERROR 

esmvalcore._task.DiagnosticError: Cannot execute script
'diag_scripts/ocean/diagnostic_timeseries.py'
(~/mambaforge/envs/esmvaltool2.6/lib/python3.10/site-packages/esmvaltool/
diag_scripts/diag_scripts/ocean/diagnostic_timeseries.py):
file does not exist. 2022-10-18 11:42:34,136 UTC [39323] INFO If you have a
question or need help, please start a new discussion on
https://github.com/ESMValGroup/ESMValTool/discussions If you suspect this is a
bug, please open an issue on https://github.com/ESMValGroup/ESMValTool/issues To
make it easier to find out what the problem is, please consider attaching the
files run/recipe_*.yml and run/main_log_debug.txt from the output directory.

The script path should be relative to diag_scripts directory. It means that the script diagnostic_timeseries.py is located in <path_to_esmvaltool>/diag_scripts/ocean/. Alternatively, the script path can be an absolute path. To examine this, we can download the script from the ESMValTool repository:

BASH 

wget https://raw.githubusercontent.com/ESMValGroup/ESMValTool/main/esmvaltool/\
diag_scripts/ocean/diagnostic_timeseries.py

One way to get the absolute path is to run:

BASH 

readlink -f diagnostic_timeseries.py

Then we can update the script path :

YAML 

scripts:
  script1:
    script:
      script: <path_to_script>/diagnostic_timeseries.py

Then, run the recipe again and examine the output to see Run was successful!

Available recipe and diagnostic scripts

ESMValTool provides a broad suite of recipes and diagnostic scripts for different disciplines like atmosphere, climate metrics, future projections, IPCC, land, ocean, ….

Re-running a diagnostic

Look at the main_log.txt file and answer the following question: How to re-run the diagnostic script?

Solution

The main_log.txt file contains information on how to re-run the diagnostic script without re-running the pre-processors:

BASH 

2020-06-29 20:36:32,844 UTC [52810] INFO    To re-run this diagnostic script, run:

Challenge

If you run the command in a terminal, you will be able to re-run the diagnostic.

Memory issues

If you run out of memory, try setting max_parallel_tasks to 1 in the configuration file. Then, check the amount of memory you need for that by inspecting the file run/resource_usage.txt in the output directory. Using the number, there you can increase the number of parallel tasks again to a reasonable number for the amount of memory available in your system.

Key Points

  • There are three different kinds of log files: main_log.txt, and main_log_debug.txt and log.txt.