Preprocessor / Macroprocessor YAML

Question

Is there an easy way to use a preprocessor / macroprocessor with YAML files? (Ie I'm thinking of something in the lines of the C preprocessor)?

We have many text files that describe various data structures. They are currently in their own proprietary format and read using their own parser. I would like to switch to YAML files to use various existing libraries for reading and writing.

However, our files are hierarchical, because we “include” the main files in the subfiles and, by replacing the variables, generate new data structures.

As an example of toys, I would like something like:

country_master.yaml

name: $COUNTRY$ file: C:\data\$COUNTRY$ 

UK_country.yaml

 #define $COUNTRY$ UK #include <country_master.yaml> 

USA_country.yaml

 #define $COUNTRY$ USA #include <country_master.yaml> 

Then, after preprocessing, we get something like:

 name: USA file: C:\data\USA 

C preprocessor will not work with the # character used in YAML comments. In addition, ideally, we would like to have loops that are extended by the preprocessor, so in the above example we will create the UK and the USA together with the loop (and I don't think you can use the cpp loop).

Any ideas?

Answer 1

You are trying to make a difference at the YAML string representation level, and I think you shouldn't. YAML can load objects, and these objects can affect loaded elements by connecting to the parser. Thus, you can replace complete nodes with data, change values ​​in scalars, etc.

Suppose you have this YAML main.yml file:

 - !YAMLPreProcessor verbose: '3' escape: ♦ - ♦replace(verbose) - abcd - ♦include(xyz.yml) - xyz 

and what xyz.yml :

 k: 9 l: 8 m: [7. 6] # can be either 

and you have ♦ as a special character (it can be anything if the YAMLPreProcessor value for special matches starts with the keyword action ( replace and include ). You want this to be rounded (loaded into data in memory, and then reset to next yaml:

 - !YAMLPreProcessor verbose: '3' escape: ♦ - '3' - abcd - k: 9 l: 8 m: [7. 6] # can be either - xyz 

This can be done by overloading the scalar constructor, which is called for each scalar and the corresponding class YAMLPreProcessor :

 # coding: utf-8 from __future__ import print_function import ruamel.yaml as yaml def construct_scalar(loader, node): self = getattr(loader, '_yaml_preprocessor', None) if self and self.d.get('escape'): if node.value and node.value.startswith(self.d['escape']): key_word, rest = node.value[1:].split('(', 1) args, rest = rest.split(')', 1) if key_word == 'replace': res = u'' for arg in args.split(','): res += str(self.d[arg]) node.value = res + rest elif key_word == 'include': inc_yml = yaml.load( open(args), Loader=yaml.RoundTripLoader ) # this needs ruamel.yaml>=0.9.6 return inc_yml else: print('keyword not found:', key_word) ret_val = loader._org_construct_scalar(node) # print('ret_val', type(ret_val), ret_val) return ret_val class YAMLPreProcessor: def __init__(self, escape=None, verbose=0): self.d = dict(escape=escape, verbose=verbose) def __repr__(self): return "YAMLPreProcessor({escape!r}, {verbose})".format(**self.d) @staticmethod def __yaml_out__(dumper, self): return dumper.represent_mapping('!YAMLPreProcessor', self.d) @staticmethod def __yaml_in__(loader, data): from ruamel.yaml.comments import CommentedMap result = YAMLPreProcessor() loader._yaml_preprocessor = result z = dict() loader.construct_mapping(data, z) result.d = z yield result def __delete__(self): loader._yaml_preprocessor = None def construct_yaml_str(self, node): value = self.construct_scalar(node) if isinstance(value, ScalarString): return value if PY3: return value try: return value.encode('ascii') except AttributeError: # in case you replace the node dynamically eg with a dict return value except UnicodeEncodeError: return value loader = yaml.RoundTripLoader loader.add_constructor('!YAMLPreProcessor', YAMLPreProcessor.__yaml_in__) loader._org_construct_scalar = loader.construct_scalar loader.construct_scalar = construct_scalar data_from_yaml = yaml.load(open('main.yml'), Loader=loader) #print ('out', data_from_yaml) dumper = yaml.RoundTripDumper # need to be able to represent '!YAMLPreProcessor' # but you can of course also remove the first element # from data_from_yaml if you don't want the preprocessor in your output dumper.add_representer(YAMLPreProcessor, YAMLPreProcessor.__yaml_out__) print(yaml.dump(data_from_yaml, Dumper=dumper, allow_unicode=True)) 

The above version of ruamel.yaml (0.9.6) requires an older version of the choke if construct_scalar returns a non-string object.

Please note that the position of the comment after the line with the key m refers to the beginning of the line, and in the example it is not a compensation for the indentation level of the node where the xyz.yml file is xyz.yml .

Answer 2

 # Yamp - YAML Macro-Processor # https://github.com/birchb1024/yamp # in master.yaml defmacro: name: country args: [$COUNTRY$] value: name: $COUNTRY$ file: C:\data\{{$COUNTRY$}} --- # in some file - include: [master.yaml] # Call with wherever needed: { country: USA }