How to maintain YAML configurations without duplication¶

Goal

Create and maintain YAML configuration files that share a common base while avoiding duplication of configuration code.

This guide presents three options to define a base YAML configuration file and extend it to create case-specific variants, such as development, production, or testing. Each option is illustrated with an example that starts from a base MATLAB simulation configuration and extends it into a high-precision configuration, inheriting from the base while overriding specific parameters.

Prerequisites:

MATLAB R2019b or later.
yaml package v1.6.0, installed and added to the path.
Intermediate familiarity with YAML syntax.

Options summary:

Option	Scalability	Use this is you want…
YAML merge keys	Low	to keep everything in a single config file
Implicit composition	Medium	to have separate config files
Explicit composition	High	to have complex inheritance chains between config files

YAML merge keys¶

How to implement¶

Create a single YAML file to store all configuration cases.
Define the shared base configuration in that file.
Use YAML anchors and merge keys to derive the remaining configurations in that file.
Load the file with yaml.loadFile().
Select the required configuration from the loaded data.

Example¶

Create a single YAML, config.yaml, to store all the configuration cases. Define the shared configuration in that file in the default field. Use YAML anchors (&default and &solver_options) and merge keys (<<: *default and <<: *solver_options) to derive base and high-precicion configurations in base and high_precision fields.

config.yaml¶

default: &default
  odefun: exponential_decay
  tspan: [0, 10]
  y0: 1
  solver: ode45
  solver_options: &solver_options
    RelTol: 1.0e-3
    AbsTol: 1.0e-6
    MaxStep: 0.1
    InitialStep: 0.01

base:
  <<: *default

high_precision:
  <<: *default
  solver_options:
    <<: *solver_options
    RelTol: 1.0e-6
    AbsTol: 1.0e-9

Load the config.yaml with yaml.loadFile() and select the required configuration from the loaded data:

config = yaml.loadFile("config.yaml", "ConvertToArray", true);

% Load base configuration.
cfg = config.base;

% Load high-precision configuration.
cfg = config.high_precision;

Implicit composition¶

How to implement¶

Create a YAML file that defines the base configuration.
Put all shared configuration values in that file.
Create a separate YAML file for each configuration case.
In each case-specific file, define only the values that differ from the base.
Select an external tool or implement a custom function to load the base configuration and apply the case-specific overrides.
Load the required configuration using that tool or function.

Example¶

Create a YAML file, base.yaml, with the base configuration values.

base.yaml¶

odefun: exponential_decay
tspan: [0, 10]
y0: 1
solver: ode45
solver_options:
  RelTol: 1.0e-3
  AbsTol: 1.0e-6
  MaxStep: 0.1
  InitialStep: 0.01

Create a separate YAML file, high_precision.yaml, defining only the values that differ from the base.yaml:

high_precision.yaml¶

solver_options:
  RelTol: 1.0e-6
  AbsTol: 1.0e-9

Implement the following custom funtion, mergeStructs(), to override the base configuration with the high-precision configuration.

mergeStructs.m¶

function [result] = mergeStructs(override, base)
%MERGESTRUCTS Recursively merge two structures.
%
%   RESULT = MERGESTRUCTS(OVERRIDE, BASE) returns a structure where fields
%   in OVERRIDE replace those in BASE and nested structs are also merged
%   recursively.

    result = base;

    for fn = fieldnames(override)'
        field = fn{1};

        if isfield(result, field) && isstruct(result.(field)) ...
            && isstruct(override.(field))
            % Recursively merge nested structs.
            result.(field) = mergeStructs(override.(field), result.(field));
        else
            % Override or add new field.
            result.(field) = override.(field);
        end
    end
end

Load the required configuration using yaml.loadFile() and mergeStructs():

% For loading base configuration.
cfg = yaml.loadFile("base.yaml", "ConvertToArray", true);

% For loading high-precision configuration.
base = yaml.loadFile("base.yaml", "ConvertToArray", true);
high_precision = yaml.loadFile("high_precision.yaml", "ConvertToArray", true);
cfg = mergeStructs(high_precision, base);

Explicit composition¶

How to implement¶

Create a separate YAML file for each base configuration.
Define all values for each base configuration case in its file.
Create a separate YAML file for each extended configuration.
In each extended configuration file, explicitly specify which base configuration it extends and define only the values that differ from that base.
Implement a custom function to load configurations, taking extensions into account.
Load the required configuration using this custom function.

Example¶

Create a YAML file, base.yaml, and define the base configuration:

base.yaml¶

odefun: exponential_decay
tspan: [0, 10]
y0: 1
solver: ode45
solver_options:
  RelTol: 1.0e-3
  AbsTol: 1.0e-6
  MaxStep: 0.1
  InitialStep: 0.01

Create a YAML file, high_precision.yaml, define it is a extension of base.yaml writing extends: base.yaml and define the only the values that differ from base.yaml.

high_precision.yaml¶

extends: base.yaml
solver_options:
  RelTol: 1.0e-6
  AbsTol: 1.0e-9

Implement the custom function, loadSimulationConfig(), to load a configuration file taking extensions into account.

loadSimulationConfig.m¶

function [config] = loadSimulationConfig(filename)
%LOADSIMULATIONCONFIG Load a simulation configuration from a YAML file.
%
%   CONFIG = LOADSIMULATIONCONFIG(FILENAME) loads the YAML configuration
%   stored in FILENAME. If the configuration contains an 'extends' field,
%   the parent configuration is loaded and merged recursively.

    % Load the configuration from YAML file.
    config = yaml.loadFile(filename, "ConvertToArray", true);

    % Merge with parent config if 'extends' is specified
    if isfield(config, "extends")
        parentConfig = loadSimulationConfig(config.extends);
        config = rmfield(config, "extends");
        config = mergeStructs(config, parentConfig);
    end
end

Load the required configuration file with loadSimulationConfig():

% Load base configuration.
cfg = loadSimulationConfig("base.yaml");

% Load high-precision configuration.
cfg = loadSimulationConfig("high_precision.yaml");

Appendix¶

Here is the ODE function definition:

exponential_decay.m¶

function dydt = exponential_decay(t, y)
%EXPONENTIAL_DECAY Simple first-order ODE.
%
%   dydt = EXPONENTIAL_DECAY(t, y) returns the time derivative of y
%   for the equation: dy/dt = -k*y, where k is the decay rate.

    k = 0.5;
    dydt = -k*y;
end

Here is how to run a simulation with the loaded configuration:

% Use the selected configuration for the simulator.
solver = str2func(cfg.solver);
odefun = str2func(cfg.odefun);
[t, y] = solver(odefun, cfg.tspan, cfg.y0, cfg.solver_options);