!include statement with certain files not working #25
Description
I want to implement dependency injection with my hyperpyyaml config files. This means keeping configs of datasets, experiments and as such as independent as possible (in different files) and then combining them using include statements. However, the following does not work:
Minimal (not-)working example:
dataset.yaml:
- name: libri_dev
data: libri
set: devmain.py
from hyperpyyaml import load_hyperpyyaml
yaml_string = """
datasets: !include:dataset.yaml #should be the list of datasets in the dataset.yaml file
"""
loaded_yaml = load_hyperpyyaml(yaml_string)Running main.py results in:
Traceback (most recent call last):
File "env/lib/python3.11/site-packages/hyperpyyaml/core.py", line 316, in resolve_references
recursive_update(preview, overrides, must_match=overrides_must_match)
File "env/lib/python3.11/site-packages/hyperpyyaml/core.py", line 768, in recursive_update
raise TypeError(f"Expected to update a mapping, but got: {d}")
TypeError: Expected to update a mapping, but got: [{'name': 'libri_dev', 'data': 'libri', 'set': 'dev'}]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "main.py", line 13, in <module>
loaded_yaml = load_hyperpyyaml(yaml_string)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "env/lib/python3.11/site-packages/hyperpyyaml/core.py", line 157, in load_hyperpyyaml
yaml_stream = resolve_references(yaml_stream, overrides, overrides_must_match)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "env/lib/python3.11/site-packages/hyperpyyaml/core.py", line 325, in resolve_references
_walk_tree_and_resolve("root", preview, preview, file_path)
File "env/lib/python3.11/site-packages/hyperpyyaml/core.py", line 406, in _walk_tree_and_resolve
included_yaml = resolve_references(f, overrides)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "env/lib/python3.11/site-packages/hyperpyyaml/core.py", line 318, in resolve_references
raise ValueError(
ValueError: ("The structure of the overrides doesn't match the structure of the document: ", {'datasets': []})
Tracing the error, it boils down to incomplete handling of the overrides in
HyperPyYAML/hyperpyyaml/core.py
Lines 312 to 322 in 0c54f07
The code is not able to understand the lack of overrides if it comes in the form {}.
I will submit a fix.