mkdocs.yml parsing error with !! in format: !!python/name:pymdownx.superfences.fence_div_format

[pbek]

Thank you very much for this wonderful service, it makes my life at documenting on https://docs.qownnotes.org/ so much easier!

Details

• Read the Docs project URL: https://docs.qownnotes.org/
• Build URL (if applicable): https://readthedocs.org/projects/qownnotes/builds/10814332
• Read the Docs username (if applicable): pbek

Expected Result

Being able to use !!python/name:pymdownx.superfences.fence_div_format in mkdocs.yml, like it works when building with MkDocs locally.

  - pymdownx.superfences:
      custom_fences:
        - name: mermaid
          class: mermaid
          # this doesn't work on ReadTheDocs, we need to use `<div class="mermaid"></div>` instead
          format: !!python/name:pymdownx.superfences.fence_div_format

Actual Result

I get a parsing error in https://github.com/pbek/QOwnNotes/blob/develop/docs/mkdocs.yml#L85 at the !! (I've now commented the line out).

Your mkdocs.yml could not be loaded, possibly due to a syntax error (line 84, column 19).

Thank you very much.

[humitos]

Being able to use !!python/name:pymdownx.superfences.fence_div_format in mkdocs.yml, like it works when building with MkDocs locally.

Is this valid in YAML?

We are parsing the mkdocs.yaml file using pyyaml and it seems that it does not recognize !!. Example,

>>> import yaml
>>> document = """
  a: 1
  b:
    c: 3
    d: !!4
"""
>>> print(yaml.dump(yaml.load(document)))
<stdin>:1: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.8/site-packages/yaml/__init__.py", line 114, in load
    return loader.get_single_data()
  File "/usr/lib/python3.8/site-packages/yaml/constructor.py", line 51, in get_single_data
    return self.construct_document(node)
  File "/usr/lib/python3.8/site-packages/yaml/constructor.py", line 60, in construct_document
    for dummy in generator:
  File "/usr/lib/python3.8/site-packages/yaml/constructor.py", line 413, in construct_yaml_map
    value = self.construct_mapping(node)
  File "/usr/lib/python3.8/site-packages/yaml/constructor.py", line 218, in construct_mapping
    return super().construct_mapping(node, deep=deep)
  File "/usr/lib/python3.8/site-packages/yaml/constructor.py", line 143, in construct_mapping
    value = self.construct_object(value_node, deep=deep)
  File "/usr/lib/python3.8/site-packages/yaml/constructor.py", line 100, in construct_object
    data = constructor(self, node)
  File "/usr/lib/python3.8/site-packages/yaml/constructor.py", line 427, in construct_undefined
    raise ConstructorError(None, None,
yaml.constructor.ConstructorError: could not determine a constructor for the tag 'tag:yaml.org,2002:4'
  in "<unicode string>", line 5, column 8:
        d: !!4
           ^
>>> 

[pbek]

I feared as much... The pymdown-extensions makes heavy use of it (like in https://github.com/facelessuser/pymdown-extensions/blob/master/mkdocs.yml) and it's considered the default way to use custom code blocks (see squidfunk/mkdocs-material#693 (comment)).

[pbek]

I asked now on facelessuser/pymdown-extensions#892 if there is a way around the !!.

With mkdocs alone those !! work fine.

[pbek]

The same issue was also "reported" on https://stackoverflow.com/questions/59246344/are-explicit-tags-in-mkdocs-config-supported-in-readthedocs.

[humitos]

!! is valid YAML. See https://yaml.org/spec/1.2/spec.html#id2761292

Although, I think the problem is that RTD can't find the tag referenced (!!python/name) because it's not loaded.

[pbek]

Oh, another plot-twist. 😄

[pbek]

facelessuser/pymdown-extensions#892 (comment) said there is no way around the !!.

[humitos]

OK! I took a look at this and I understand the problem. I had no idea what was the !! in YAML since I never seen it before.

For example, using !! here we can cast the 2 as a float, instead of a integer.

In [13]: import yaml 
    ...: print(yaml.load(""" 
    ...: gold: !!float 2 
    ...: """))                                                                                                                                    
{'gold': 2.0}

Using !!python we can convert that field to a Python object,

In [15]: import yaml 
    ...: print(yaml.load(""" 
    ...: gold: !!python/name:int 
    ...: """))                                                                                                                                    
{'gold': <class 'int'>}

So, I think the problem is that we are parsing your file before installing all the dependencies of your project, and so, Python can't find the module and it fails.

The solution, should be to try to load your YAML configuration file skipping these calls to !!python/name:.

[pbek]

Thank you very much, @humitos!

[pbek]

Thank you for the quick fix, @ericholscher and @humitos! In case it should already work, I still get the same error message in https://readthedocs.org/projects/qownnotes/builds/10854620/:

Your mkdocs.yml could not be loaded, possibly due to a syntax error (line 85, column 19)

https://github.com/pbek/QOwnNotes/blob/develop/docs/mkdocs.yml#L85

[humitos]

@pbek It's not deployed yet, but I suppose it will go out next week.

[pbek]

Ok, great. Thank you!

[pbek]

If I try to enable the setting now I get new errors:
https://readthedocs.org/projects/qownnotes/builds/10904583/

INFO    -  Cleaning site directory 
INFO    -  Building documentation to directory: /home/docs/checkouts/readthedocs.org/user_builds/qownnotes/checkouts/latest/docs/_build/html 
INFO    -  The following pages exist in the docs directory, but are not included in the "nav" configuration:
  - assets/graphs/concept.md 
ERROR   -  Error reading page 'assets/graphs/concept.md': 'NoneType' object is not callable 
Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/qownnotes/envs/latest/lib/python3.7/site-packages/pymdownx/superfences.py", line 529, in process_nested_block
    id_value=self.id
  File "/home/docs/checkouts/readthedocs.org/user_builds/qownnotes/envs/latest/lib/python3.7/site-packages/pymdownx/superfences.py", line 208, in _formatter
    return _fmt(source, language, class_name, options, md, **kwargs)
TypeError: 'NoneType' object is not callable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/docs/.pyenv/versions/3.7.3/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/docs/.pyenv/versions/3.7.3/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/docs/checkouts/readthedocs.org/user_builds/qownnotes/envs/latest/lib/python3.7/site-packages/mkdocs/__main__.py", line 202, in <module>
    cli()
  File "/home/docs/checkouts/readthedocs.org/user_builds/qownnotes/envs/latest/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/docs/checkouts/readthedocs.org/user_builds/qownnotes/envs/latest/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/docs/checkouts/readthedocs.org/user_builds/qownnotes/envs/latest/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/docs/checkouts/readthedocs.org/user_builds/qownnotes/envs/latest/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/docs/checkouts/readthedocs.org/user_builds/qownnotes/envs/latest/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/docs/checkouts/readthedocs.org/user_builds/qownnotes/envs/latest/lib/python3.7/site-packages/mkdocs/__main__.py", line 163, in build_command
    ), dirty=not clean)
  File "/home/docs/checkouts/readthedocs.org/user_builds/qownnotes/envs/latest/lib/python3.7/site-packages/mkdocs/commands/build.py", line 274, in build
    _populate_page(file.page, config, files, dirty)
  File "/home/docs/checkouts/readthedocs.org/user_builds/qownnotes/envs/latest/lib/python3.7/site-packages/mkdocs/commands/build.py", line 177, in _populate_page
    page.render(config, files)
  File "/home/docs/checkouts/readthedocs.org/user_builds/qownnotes/envs/latest/lib/python3.7/site-packages/mkdocs/structure/pages.py", line 184, in render
    self.content = md.convert(self.markdown)
  File "/home/docs/checkouts/readthedocs.org/user_builds/qownnotes/envs/latest/lib/python3.7/site-packages/markdown/core.py", line 262, in convert
    self.lines = prep.run(self.lines)
  File "/home/docs/checkouts/readthedocs.org/user_builds/qownnotes/envs/latest/lib/python3.7/site-packages/pymdownx/superfences.py", line 849, in run
    return self.search_nested(lines)
  File "/home/docs/checkouts/readthedocs.org/user_builds/qownnotes/envs/latest/lib/python3.7/site-packages/pymdownx/superfences.py", line 708, in search_nested
    self.eval_fence(ws, content, start, end)
  File "/home/docs/checkouts/readthedocs.org/user_builds/qownnotes/envs/latest/lib/python3.7/site-packages/pymdownx/superfences.py", line 473, in eval_fence
    self.process_nested_block(ws, content, start, end)
  File "/home/docs/checkouts/readthedocs.org/user_builds/qownnotes/envs/latest/lib/python3.7/site-packages/pymdownx/superfences.py", line 532, in process_nested_block
    code = entry["formatter"](self.rebuild_block(self.code), self.lang, self.options, self.md)
  File "/home/docs/checkouts/readthedocs.org/user_builds/qownnotes/envs/latest/lib/python3.7/site-packages/pymdownx/superfences.py", line 208, in _formatter
    return _fmt(source, language, class_name, options, md, **kwargs)
TypeError: 'NoneType' object is not callable

[masalim2]

I'd like to bump this issue, as I am getting the same error. When I inspect the resulting mkdocs.yml, after it has been parsed on your end, the configuration looks like this:

- pymdownx.superfences:
    custom_fences:
    - class: mermaid
      format: null
      name: mermaid

The format: null is (not surprisingly) the result of dropping any potentially unsafe !! yaml directives. However, this breaks our Mkdocs configuration and the build fails; because pymdownx.superfences expects to receive a callable Python format function there.

[masalim2]

My first thought was that pymdown-extensions should just accept a string for format and fetch the formatter at runtime. Unfortunately, this issue was already raised and closed on their side based on this argument: facelessuser/pymdown-extensions#892 (comment)

Since ReadTheDocs is already running the repository's install scripts, there's probably no additional vulnerability that could be caused by the "unsafe" yaml.load().

Would it be possible to parse mkdocs.yml with ordinary yaml.load after installing the project dependencies? I think that ought to provide the most flexibility for MkDocs builds.

[humitos]

Since ReadTheDocs is already running the repository's install scripts, there's probably no additional vulnerability that could be caused by the "unsafe" yaml.load().

This happens under different environments. The parsing of the mkdocs.yml file is done outside the Docker container. That's why we need to be careful about this.

Please, see the PR linked in this issue (#6897). If you have a proposal to make this work while keeping it safe, let me know.

[masalim2]

Ah okay, that makes sense. So you need to safely load the yaml outside the container and take a "first pass" over the configuration in BaseMkdocs.append_conf() to add some readthedocs-specific settings. Then, the build inside a container happens inside the BaseMkdocs.build() method.

Would it make sense to expand these 2 steps into a process of:

1. Safely load yaml outside container for validation only; do not modify config yet
2. Run append_conf() logic inside the container and dump modified config
3. Run doc builder

Perhaps something like validate_conf() takes the place of append_conf() in step 1, and in step 2, the very same build() method running in the container could call unsafe_append_conf() to modify the config without dropping "!!" directives immediately before running the doc builder.

[masalim2]

Taking a look at projects/tasks.py -- please correct me if I misunderstood, but it looks like the html_builder.append_conf() method is already being called inside the Build environment:

readthedocs.org/readthedocs/projects/tasks.py

Lines 1172 to 1181 in a6b60be

	def build_docs_html(self):
	"""Build HTML docs."""
	html_builder = get_builder_class(self.config.doctype)(
	build_env=self.build_env,
	python_env=self.python_env,
	)
	if self.build_force:
	html_builder.force()
	html_builder.append_conf()
	success = html_builder.build()

This build_docs_html() method is called from build_docs(), which is called inside of the Build Env:

# Environment used for building code, usually with Docker
with self.build_env:
   # ........
   outcomes = self.build_docs()

In that case, I think it would be okay to directly modify the append_conf method to use ordinary unsafe yaml.load, since that is already in the build environment. I could try to submit a PR along these lines if it seems right by you.

Edit: I see that build_env is passed into the Builder, but to actually run a command inside the build environment, it has to be invoked through build_env.run(). Sorry for jumping to conclusions so quickly here. Still, I wonder if joining the yaml modification and build into single container run step would be an appropriate workaround.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stsewd]

An update here, we are not 100% if this change would be safe, that's why we haven't merged that PR yet.

[stsewd]

Closing this in favor of #7461