Welcome to PyCast S2S’s documentation!

PyCast S2S provides a suite of tools for processing and post-processing regional climate forecast data. It facilitates tasks such as:

Data Preparation: Truncating and remapping forecast and reference data to specific domains and grids.
Bias Correction: Applying the Bias Correction Spatial Disaggregation (BCSD) method to improve forecast accuracy.
Aggregation and Analysis: Calculating climatologies, monthly statistics, and other derived variables.
Parallel Processing: Leveraging Dask for efficient computation on large datasets.
Flexible Configuration: Utilizing JSON configuration files for easy customization of processing parameters.

This framework is designed to be modular and extensible, allowing users to adapt it to their specific needs. It is built upon popular Python libraries like Xarray, Dask, and others, ensuring interoperability with existing scientific workflows.

Process_regional_forecasts.py - The tool for setting up your data

Processes regional forecasts by performing operations like truncation, remapping, and concatenation.

This script handles various processing steps for regional climate forecasts, including:

Truncating forecasts and reference data to a specified bounding box.
Remapping forecasts and reference data to a target grid.
Concatenating daily and monthly forecasts and reference data for calibration and other periods.
Rechunking data for optimized storage and access.
Calculating temperature-related variables (t2plus, t2minus).

The script uses command-line arguments to specify the domain, processing mode, years, months, variables, and other options. It relies on configuration files (domain_config.json, attribute_config.json, variable_config.json) for domain-specific parameters, variable attributes, and directory paths. The script utilizes Dask for parallel processing and Xarray for handling NetCDF and Zarr data formats.

process_regional_forecasts.get_clas()

Parses command-line arguments for regional forecast processing.

This function defines and parses command-line arguments used to control the regional forecast processing script. It uses argparse to handle arguments related to domain selection, processing mode, time ranges, variables, cluster configuration, and input/output files.

Returns

argparse.Namespaceargparse.Namespace: An object containing the parsed arguments.

process_regional_forecasts.process_forecast(mode, func, args, domain_config, variable_config, reg_dir_dict, grid_file)

Generalized function to iterate over variables, years, and months while applying a specific processing function.

Parameters

modestr: The processing mode (e.g., “truncate_forecasts”, “remap_forecasts”).
funccallable: The function to execute for each combination.
argsNamespace: Parsed command-line arguments.
domain_configdict: Domain-specific configuration.
variable_configdict: Variable-specific configuration.
reg_dir_dictdict: Directory paths.
grid_filestr: Grid file path.

Run_bcsd - The main file of PyCast S2S

Applies bias correction to regional climate forecasts using the BCSD method.

This script performs bias correction on regional climate forecasts using the Bias Correction Spatial Disaggregation (BCSD) method. It takes raw forecast data, reforecasts, and observational reference data as input and produces bias-corrected forecasts. The script supports various configuration options, including domain selection, processing years and months, variable selection, and cross-validation. It utilizes Dask for parallel processing and Xarray for handling NetCDF and Zarr data formats.

run_bcsd.get_clas()

run_bcsd.within_window(time, start_window, end_window)

Check if a date falls within a defined time window, handling year boundaries.

This function determines if a given date falls within a time window defined by start_window and end_window, correctly handling cases where the window spans across the end of a year and the beginning of the next.

Args:: time: The date to check, represented as a pandas Timestamp.
Returns:: True if the date falls within the window, False otherwise.

Regional processing modules - Helper files for the regional proccesing

This module provides functions for processing regional climate data, including:

Preprocessing functions for forecasts and reference data (e.g., renaming coordinates, handling time dimensions).
Truncating and remapping functions for both forecasts and reference data to a specified domain and grid.
Calibration functions for forecasts, involving concatenation and rechunking of data.
Utility functions for creating grid files and setting encoding parameters for NetCDF files.

The module utilizes libraries like Xarray, Dask, and CDO for data manipulation, parallel processing, and remapping operations.

modules.regional_processing_modules.calib_forecasts(domain_config, variable_config, dir_dict, syr, eyr, month_str)

Calibrates forecasts by concatenating and rechunking data.

This function concatenates forecast data for a specified period (syr to eyr) and rechunks it for optimized storage and access. It handles both merged and individual variable files based on the ‘merged_variables’ configuration.

Args:: domain_config (dict): Domain configuration dictionary. variable_config (dict): Variable configuration dictionary. dir_dict (dict): Directory paths dictionary. syr (int): Start year for calibration. eyr (int): End year for calibration. month_str (str): Month string (e.g., “01”).

modules.regional_processing_modules.create_grd_file(domain_config: dict, grid_file: str) → str: Creates a grid description file that is used for remapping the forecasts to the final resolution.

modules.regional_processing_modules.create_grid_ds(grid_file_path)

Reads a grid definition file and creates an xarray.Dataset for regridding with xESMF.

Parameters:: grid_file_path (str): Path to the grid definition file.
Returns:: xr.Dataset: An xarray dataset containing the latitude and longitude grid.

modules.regional_processing_modules.preprocess(ds)

Preprocesses an xarray Dataset by renaming and sorting coordinates.

This function standardizes the longitude and latitude coordinates of a dataset, ensuring correct naming and ascending order. It also converts longitudes from the [0, 360] range to [-180, 180] if necessary.

Args:: ds (xr.Dataset): The input xarray Dataset.
Returns:: xr.Dataset: The preprocessed xarray Dataset.

modules.regional_processing_modules.preprocess_reference(ds)

Preprocesses reference data by adjusting time coordinates and renaming variables.

This function modifies the time dimension of a reference dataset to ensure consistent daily values throughout the year. It also standardizes longitude and latitude coordinates, converting longitudes to the [-180, 180] range if necessary. Additionally, it removes the “bnds” dimension if present.

Args:: ds (xr.Dataset): The input xarray Dataset.
Returns:: xr.Dataset: The preprocessed xarray Dataset.

modules.regional_processing_modules.rechunker_forecasts(domain_config: dict, variable_config: dict, dir_dict: dict, year: int, month: int, variable: str)

The helper_modules - General helper tools for PyCast

modules.helper_modules.create_4d_netcdf(file_out, global_config, domain_config, variable_config, coordinates, variable) → Dataset

Create a 4D NetCDF dataset with specified metadata and coordinates.

Parameters

file_outstr: The path to the output NetCDF file.
global_configdict: Global attributes for the NetCDF file.
domain_configdict: Domain-specific configuration.
variable_configdict: Variable-specific configuration containing standard_name, long_name, and units.
coordinatesdict: Dictionary containing coordinate arrays for time, ens (ensemble), lat (latitude), and lon (longitude).
variabledict: Name of the variable for which the NetCDF dataset is created.

Returns

dsxr.Dataset: A 4D xarray Dataset with the specified variable, dimensions, coordinates, and attributes.

Example:

file_out = "output.nc"
global_config = {"history": "Created by create_4d_netcdf function"}
domain_config = {...}  # Domain-specific configuration
variable_config = {
    "temperature": {"standard_name": "air_temperature", "long_name": "Temperature", "units": "K"}
}
coordinates = {
    "time": pd.date_range("2023-01-01", periods=10, freq="D"),
    "ens": np.arange(1, 6),
    "lat": np.linspace(-90, 90, 180),
    "lon": np.linspace(-180, 180, 360),
}
variable = "temperature"

ds = create_4d_netcdf(file_out, global_config, domain_config, variable_config, coordinates, variable)

The function generates an xarray Dataset with a DataArray for the specified variable. The coordinates dictionary should contain arrays for “time,” “ens,” “lat,” and “lon.” The variable_config dictionary should contain metadata information for the specified variable. The resulting dataset can be written to a NetCDF file using ds.to_netcdf() method (currently commented out in the code).

modules.helper_modules.decode_processing_months(months_string)

Decodes a string of months into a list of months for processing.

This function takes a comma-separated string of months and converts it into a list of integers. It handles different input formats: single month, start and end month, or a sequence of months.

Parameters

months_stringstr: A comma-separated string of months.

Returns

monthslist: A list of months to process.

modules.helper_modules.decode_processing_years(years_string)

Decodes a string of years into a list of years for processing.

This function takes a comma-separated string of years and converts it into a list of integers. It handles different input formats: single year, start and end year, or a sequence of years.

Parameters

years_stringstr: A comma-separated string of years.

Returns

yearslist: A list of years to process.

modules.helper_modules.get_coords_from_frcst(filename, filetype='netcdf')

Retrieves coordinates from a forecast file.

This function extracts time, latitude, longitude, and ensemble coordinates from a forecast file, supporting both NetCDF and GRIB formats.

Parameters

filenamestr: Path to the forecast file.
filetypestr: (Optional) Type of the forecast file (‘netcdf’ or ‘grib’). Defaults to ‘netcdf’.

Returns

coords: dict: A dictionary containing the extracted coordinates.

modules.helper_modules.get_coords_from_ref(filename)

Retrieves coordinates from a reference file.

This function extracts time, latitude, and longitude coordinates from a reference file.

Parameters

filenamestr: Path to the reference file.

Returns

dict: dict: A dictionary containing the extracted coordinates.

modules.helper_modules.load_json(file_path)

Load JSON data from a file.

This function reads JSON data from the specified file path and returns it as a Python object. It raises a FileNotFoundError if the file does not exist.

Parameters

file_pathstr: The path to the JSON file.

Returns

dict or listdict or list: The loaded JSON data as a Python dictionary or list.

Raises

FileNotFoundError: If the file specified by file_path does not exist.

modules.helper_modules.preprocess_mdl_hist(filename, month, variable)

Preprocesses model history data by creating a consistent time dimension.

This function takes model history data with separate ‘year’ and ‘time’ dimensions and combines them into a single ‘time’ dimension with datetime values. It assumes the input data has monthly chunks and processes a specific variable within the dataset.

Parameters

filenamestr: Path to the model history NetCDF file.
monthint: The month for which to process the data (1-12).
variablestr: The name of the variable to extract and process.

Returns

xr.Datasetxr.Dataset: The preprocessed dataset with a unified ‘time’ dimension.

modules.helper_modules.run_cmd(cmd, path_extra=PosixPath('/home/docs/checkouts/readthedocs.org/user_builds/pycast-s2s/envs/latest/bin'))

Runs a bash command.

This function executes a bash command using the subprocess module. It extends the environment’s PATH variable with an additional path and raises a RuntimeError if the command fails.

Parameters

cmdlist: The bash command as a list of strings.
path_extraPath: (Optional) Additional path to add to the PATH environment variable. Defaults to Path(sys.exec_prefix) / “bin”.

Returns

str: str: The standard output of the command.

Raises

RuntimeError: If the command fails (non-zero return code).

modules.helper_modules.set_and_make_dirs(domain_config: dict) → dict

Prepare directory names

For running the whole BCSD-workflow, we need to set various directories for all the different processing steps.

Parameters

domain_configdict: A dictionary with all information about the current domain

Returns

reg_dir_dict: A dictionary with all required directories for the regional processing stels
glob_dir_dict: A dictionary which contains the directories of the global data; this needs to be made more flexible in a future release…

modules.helper_modules.set_encoding(variable_config, coordinates, type='maps')

Prepares an encoding-dictionary

When writing data to NetCDF-files, we need to set some parameters that can substantially imact the I/O-performance as well as the filesize. A standard set of parameters is defined with this little function.

Parameters

variable_configdict: A dictionary with all the variables that will be processed during the current workflw
coordinatesdict: A dictionary that holds the coordinates of the data for which we need encoding parameter.
type: str: Can be set to “maps” or “lines” for defining the chunking of the output data; map-chunks imporove the performance when whole maps are needed while line-chunks work much better if data along the time-axis is required.

Returns

encoding: A dictionary with the variables from variable_config and the corresponding parameter that will be used for writing the NetCDF-files

modules.helper_modules.set_input_files(domain_config: dict, reg_dir_dict: dict, month: int, year: int, variable: str)

Generate file paths for input data based on configuration and time information.

Parameters

domain_configdict: Dictionary containing configuration parameters for the domain.
reg_dir_dictdict: Dictionary containing directory paths for different types of files.
monthint: Numeric representation of the month (1 to 12).
yearint: Year for which the input files are generated.
variablestr: Variable for which the input files are generated.

Returns

raw_fullstr: File path for raw forecasts
pp_fullstr: File path for processed forecasts
refrcst_fullstr: File path for calibrated raw forecasts
ref_fullstr: File path for reference data

Example:

domain_config = {...}  # Dictionary with domain configuration parameters
reg_dir_dict = {...}   # Dictionary with directory paths
month = 3
year = 2023
variable = "temperature"

raw_file, pp_file, refrcst_zarr, ref_zarr = set_input_files(domain_config, reg_dir_dict, month, year, variable)

The function constructs file paths for different types of input data based on the provided parameters. The generated file paths include: - raw_full: Path to the raw forecasts NetCDF file. - pp_full: Path to the processed forecasts NetCDF file. - refrcst_full: Path to the calibrated raw forecasts Zarr file. - ref_full: Path to the reference data Zarr file.

Note: - The function uses configuration parameters from domain_config and directory paths from reg_dir_dict to construct file paths. - The month and year parameters determine the time period for which the files are generated. - The variable parameter specifies the variable for which the input files are generated.

modules.helper_modules.set_zarr_encoding(variable_config)

Generate Zarr encoding configuration for variables based on provided variable configurations.

Parameters

variable_configdict: Dictionary containing variable-specific configuration parameters.

Returns

encoding: dict: Dictionary containing Zarr encoding configurations for each variable.

Example:

variable_config = {
    "temperature": {"_FillValue": -9999, "scale_factor": 1.0, "add_offset": 0.0, "dtype": "float32"},
    "precipitation": {"_FillValue": -9999, "scale_factor": 1.0, "add_offset": 0.0, "dtype": "float32"},
}

encoding = set_zarr_encoding(variable_config)

The function constructs Zarr encoding configurations for each variable based on the provided variable_config dictionary. The generated encoding dictionary includes compression settings and other metadata for each variable, as well as default configurations for ‘lat’, ‘lon’, and ‘time’ variables.

Note: - The function iterates through each variable in variable_config and generates an encoding configuration based on specified parameters. - Default encoding configurations for ‘lat’, ‘lon’, and ‘time’ variables are also included.

modules.helper_modules.update_global_attributes(global_config, bc_params, coords, domain)

Add information about the current workflow to the output NetCDFs

The global attributes of the output NetCDFs should contain some information about the current domain and the parameters of the bias-correction. This function reads the respective information from the different parameter dictionaries and updates the global attributes accordingly.

Parameters

global_configdict: A dictionary with all global attributes for the NetCDFs
bc_params: dict: A dictionary that holds the settings for the bias correction
coords: dict: A dictionary with the coordinates of the output NetCDFs
domain: str: The name of the current domain

Returns

global_config: A dictionary with global attributes for the output NetCDFs

The cluster_modules - Tools for setting up your cluster environment

modules.cluster_modules.getCluster(queue, nodes, jobs_per_node, scheduler_file=None, walltime=None, cores=None, memory=None)

Set up a Dask cluster for SLURM or local execution.

Parameters: - queue (str): The queue to which to submit the job or ‘local’ for local execution. - nodes (int): Number of nodes requested for the current workflow. - jobs_per_node (int): Number of jobs submitted to a single node. - scheduler_file (str, optional): Path to a scheduler file for an existing Dask cluster. - walltime (str, optional): Walltime for SLURM jobs. - cores (int, optional): Number of cores for SLURM jobs. - memory (str, optional): Memory specification for SLURM jobs.

Returns: - client (Client): Object for interacting with HPC resources and submitting jobs. - cluster (SLURMCluster or LocalCluster): Object with information about the cluster infrastructure.

Usage Example: `python client, cluster = getCluster('rome', 2, 4) `

modules.cluster_modules.get_default_cluster_options(queue)

Get default cluster options based on the specified queue.

Parameters: - queue (str): The queue for which to retrieve default options.

Returns: - dict: A dictionary containing default cluster options (cores, memory, walltime).

modules.cluster_modules.s3_init()

Initialize and configure an S3FileSystem instance for interacting with an S3-compatible storage.

Returns: s3fs.core.S3FileSystem: An S3FileSystem instance configured with the provided access key,

secret key, region, endpoint URL, and verification settings.

Example:

s3_instance = s3_init()
file_list = s3_instance.ls("s3://your-bucket-name/")
print("List of files in the bucket:", file_list)

Note: - Replace the access key, secret key, region, and endpoint URL with your specific credentials

and S3-compatible storage information.

The ‘verify’ parameter is set to ‘False’ to disable SSL certificate verification. Adjust accordingly based on your security requirements.

Welcome to PyCast S2S’s documentation!

Process_regional_forecasts.py - The tool for setting up your data

Returns

Parameters

Run_bcsd - The main file of PyCast S2S

Regional processing modules - Helper files for the regional proccesing

The helper_modules - General helper tools for PyCast

Parameters

Returns

Parameters

Returns

Parameters

Returns

Parameters

Returns

Parameters

Returns

Parameters

Returns

Raises

Parameters

Returns

Parameters

Returns

Raises

Parameters

Returns

Parameters

Returns

Parameters

Returns

Parameters

Returns

Parameters

Returns

The cluster_modules - Tools for setting up your cluster environment

Indices and tables