Add new helper functions and add logic for script type tags in HTML ext

- Add `is_raw`, `is_block`, and `html_escape` helper functions
- HTML extension now has the concept of `html` tags which will store the
  content as non HTML escaped content. `script` and `style` tags will
  be auto detected for this mode, but this mode can be manually set
  now via the `markdown` option as well.
- In the HTML extension, to protect raw and raw HTML content from the
  Treprocessors, content will be placed in the stash on `on_end`.

Author: facelessuser

docs/src/dictionary/en-custom.txt

@@ -114,6 +114,7 @@ Tasklist
 Toc
 Tox
 Treeprocessor
+Treeprocessors
 Twemoji
 Twemoji's
 Twitter's
@@ -156,6 +157,7 @@ emoji
 emojione
 escaper
 eslint
+etree
 facelessuser
 formatter
 formatter's

docs/src/markdown/.snippets/links.md

@@ -43,6 +43,7 @@
 [emojione-index]: https://github.com/facelessuser/pymdown-extensions/blob/master/pymdownx/emoji1_db.py
 [emojione-sprites-svg]: https://github.com/Ranks/emojione/blob/v2.2.7/assets/sprites/emojione.sprites.svg
 [emojione]: https://github.com/Ranks/emojione
+[etree]: https://docs.python.org/3/library/xml.etree.elementtree.html
 [flake8-docstrings]: https://pypi.python.org/pypi/flake8-docstrings
 [flake8]: https://pypi.python.org/pypi/flake8
 [footnotes]: https://python-markdown.github.io/extensions/footnotes/
@@ -73,6 +74,7 @@
 [python-markdown]: https://github.com/Python-Markdown/markdown
 [pyyaml]: https://github.com/yaml/pyyaml
 [requests]: https://pypi.python.org/pypi/requests/
+[stash]: https://python-markdown.github.io/extensions/api/#working_with_raw_html
 [tables]: https://python-markdown.github.io/extensions/tables/
 [toc]: https://python-markdown.github.io/extensions/toc/
 [tox]: https://pypi.python.org/pypi/tox

docs/src/markdown/extensions/blocks/api.md

@@ -176,6 +176,37 @@ developers via `self.options['attrs']`. The result is a dictionary of key/value
 the value is a `#!py3 str` (or `#!py3 list[str]` in the special case of `class`).
 ///
 
+## `is_raw`
+
+```py
+def is_raw(self, tag: Element) -> bool:
+    ...
+```
+
+This method, given a tag will determine if the block should be considered a "raw" tag based on the Blocks extension's
+internal logic.
+
+## `is_block`
+
+```py
+def is_block(self, tag: Element) -> bool:
+    ...
+```
+
+This method, given a tag will determine if the block should be considered a "block" tag based on the Blocks extension's
+internal logic.
+
+
+## `html_escape`
+
+```py
+def html_escape(self, text: str) -> str:
+    ...
+```
+
+Takes a string intended for an HTML tag's content and returns it after applying HTML escaping on it. Escapes `&`, `<`,
+and `>`.
+
 ## `on_init` Event
 
 ```py
@@ -242,7 +273,8 @@ def on_markdown(self) -> str:
 ```
 
 The `on_markdown` event is used to declare how the content of the block should be handled by the Markdown parser. A
-string with one of the following values _must_ be returned. 
+string with one of the following values _must_ be returned. All content is stored under the [etree][etree] element
+returned via the [`on_add` event](#on_add-event), regardless of what mode is returned.
 
 Result\ Value | Description
 ------------- | -----------
@@ -255,10 +287,15 @@ Only during the [`on_end` event](#on_end-event) will all the content be fully ac
 block processors, and only during the [`on_inline_end` event](#on_inline_end-event) will both block and inline
 processing be completed.
 
-When using `raw` mode, all text will be gathered as blocks are processed and will be fully available during the
-[`on_end` event](#on_end-event). Content in a `raw` block should be indented to avoid the HTML parser and will be
-dedented (no more than the current Markdown tab length) in the final result. Content will stored as a Python Markdown
-[`AtomicString`][atomic].
+When using `raw` mode, all text will be accumulated and fully available during the [`on_end` event](#on_end-event).
+Content is accessible under the element returned by the [`on_add` event](#on_add-event) and can be accessed via
+`element.text`. Text content is stored as a Python Markdown [`AtomicString`][atomic]. If desired, the content can be
+stored in the [HTML stash][stash] during the [`on_add` event](#on_add-event)to ensure it makes it through any and all
+Treeprocessors after inline handling. Additionally, when storing in the stash, the developer can HTML escape the content
+to have the text present as literal or store without HTML escaping to present as an altered HTML content.
+
+A `raw` block expects the content to be an indented code block as this is necessary to avoid some Python Markdown's
+internal HTML parser. Content will not have the extra indentation in the final output.
 
 It should be noted that `raw` mode cannot prevent transformations that are applied during Python Markdown's preprocessor
 steps. Blocks will attempt to revert any placeholders within the content that are currently found in the HTML stash.

docs/src/markdown/extensions/blocks/index.md

@@ -105,10 +105,10 @@ Some content.
 ////
 
 Some blocks may take raw content (and should note this in their documentation) which will avoid further Markdown
-processing on the content.  Due to the way Python Markdown works, these content blocks must be indented to avoid having
-the HTML processor from altering content. Raw content blocks will remove the indentation up to the Markdown tab length
-(4 spaces by default). If they are not indented, they will still be processed, but they may be affected by Python
-Markdown's HTML processor.
+processing on the content.  This is done by requiring the content to be an indented code block. Due to the way Python
+Markdown works, these content blocks must be indented to avoid having the HTML processor from altering content. Raw
+blocks cannot shield content from all preprocessor transformations, but by requiring the content to be indented code
+blocks, the content will survive any alterations that a traditional code block would survive.
 
 ```
 /// html | pre

docs/src/markdown/extensions/blocks/plugins/html.md

@@ -56,14 +56,35 @@ some *markdown* content
 
 By default HTML blocks will automatically have the content rendering determined from tag name, so `div` blocks will be
 treated as block elements, `span` will be treated as inline elements, and things like `pre` will treat the content as
-raw text that should not be processed by Markdown further. With that said, there may be cases where an HTML element
-isn't properly recognized yet, or the user simply wants to control how the element processes its content, in these
-cases, the `markdown` option can be used to specify how Markdown content is handled.
+raw text that needs HTML escaping, and things like `script` will be treated as raw content does not need HTML escaping.
+With that said, there may be cases where an HTML element isn't properly recognized yet, or the user simply wants to
+control how the element processes its content, in these cases, the `markdown` option can be used to specify how Markdown
+content is handled.
+
+Markdown\ Modes | Description
+--------------- | -----------
+`block`         | Parsed block content will be handled by the Markdown parser as content under a block element.
+`inline`        | Parsed block content will be handled by the Markdown parser as content under an inline element.
+`raw`           | Parsed block content will be preserved. No additional Markdown parsing will be applied. Content will be HTML escaped to preserve the content as is.
+`auto`          | Depending on whether the wrapping parent is a block element, inline element, or something like a code element, Blocks will choose the best approach for the content. Decision is made based on the element returned by the [`on_add` event](#on_add-event).
+`html`          | Like `raw`, content will be preserved, but the content will _not_ be HTML escaped and will be passed through as unmodified HTML. Any required sanitizing should be provided by the user post Markdown processing.
+
+/// tip | Raw and HTML Mode
+When using _raw_ tags or forcing _raw_ mode with `markdown: raw` (HTML escaped) or `markdown: html` (no HTML escaping),
+code must be indented. This is because Python Markdown will look for and process raw HTML in non indented blocks. The
+only avoid this is to use indented code blocks. If content is not indented, the content may be missing at the end.
+
+Recognized raw block tags: `canvas`, `math`, `option`, `pre`, and `textarea`.
+
+Recognized raw HTML tags: `script` and `style`.
+
+Also, make sure to have a new line before indented content so it is not recognized as an attempt to specify YAML
+options.
+///
 
 In the following example we force `pre` to handle content as Markdown block content instead of the usual raw content
 default.
 
-
 ```text title="Pre as Block"
 /// html | pre
 
@@ -90,20 +111,9 @@ some *markdown* content
 ////
 ///
 
-/// tip | Raw Mode
-When using _raw_ tags or forcing _raw_ mode with `markdown: raw`, it is advised to indent the code. This is because
-Python Markdown will look for and process raw HTML in non indented blocks. The only avoid this is to use indented
-blocks. Content will automatically be dedented by the expected tab length.
-
-Recognized raw block tags: `canvas`, `math`, `option`, `pre`, `script`, `style`, and `textarea`.
-
-Also, make sure to have a new line before indented content so it is not recognized as an attempt to specify YAML
-options.
-///
-
 ## Per Block Options
 
 Options      | Type       | Descriptions
 ------------ | ---------- | ------------
-`markdown`   | string     | String value to control how Markdown content is processed. Valid options are: `auto`, `block`, `inline`, and `raw`.
+`markdown`   | string     | String value to control how Markdown content is processed. Valid options are: `auto`, `block`, `inline`, `html`, and `raw`.
 `attrs`      | string     | A string that defines attributes for the outer, wrapper element.

pymdownx/blocks/__init__.py

@@ -167,7 +167,7 @@ def __init__(self, parser, md):
             ['address', 'dd', 'dt', 'h1', 'h2', 'h3', 'h4', 'h5', 'h6', 'legend', 'li', 'p', 'summary', 'td', 'th']
         )
         # Block-level tags which never get their content parsed.
-        self.raw_tags = set(['canvas', 'math', 'option', 'pre', 'script', 'style', 'textarea'])
+        self.raw_tags = set(['canvas', 'math', 'option', 'pre', 'script', 'style', 'textarea', 'code'])
         # Block-level tags in which the content gets parsed as blocks
         self.block_tags = set(self.block_level_tags) - (self.span_tags | self.raw_tags | self.empty_tags)
         self.span_and_blocks_tags = self.block_tags | self.span_tags
@@ -339,15 +339,15 @@ def get_parent(self, parent):
                 temp = self.lastChild(temp)
         return None
 
-    def is_raw(self, tag, mode):
+    def is_raw(self, tag):
         """Is tag raw."""
 
-        return mode == 'raw' or (mode == 'auto' and tag.tag in self.raw_tags)
+        return tag.tag in self.raw_tags
 
-    def is_block(self, tag, mode):
+    def is_block(self, tag):
         """Is tag block."""
 
-        return mode == 'block' or (mode == 'auto' and tag.tag in self.block_tags)
+        return tag.tag in self.block_tags
 
     def parse_blocks(self, blocks, entry):
         """Parse the blocks."""
@@ -364,8 +364,8 @@ def parse_blocks(self, blocks, entry):
             mode = entry.block.on_markdown()
             if mode not in ('block', 'inline', 'raw'):
                 mode = 'auto'
-            is_block = self.is_block(target, mode)
-            is_atomic = self.is_raw(target, mode)
+            is_block = mode == 'block' or (mode == 'auto' and self.is_block(target))
+            is_atomic = mode == 'raw' or (mode == 'auto' and self.is_raw(target))
 
             # We should revert fenced code in spans or atomic tags.
             # Make sure atomic tags have content wrapped as `AtomicString`.

pymdownx/blocks/block.py

@@ -240,15 +240,23 @@ def __init__(self, length, tracker, block_mgr, config):
         self.config = config
         self.on_init()
 
-    def is_raw(self, tag, mode):
+    def is_raw(self, tag):
         """Is raw element."""
 
-        return self._block_mgr.is_raw(tag, mode)
+        return self._block_mgr.is_raw(tag)
 
-    def is_block(self, tag, mode):  # pragma: no cover
+    def is_block(self, tag):  # pragma: no cover
         """Is block element."""
 
-        return self._block_mgr.is_block(tag, mode)
+        return self._block_mgr.is_block(tag)
+
+    def html_escape(self, text):
+        """Basic html escaping."""
+
+        text = text.replace('&', '&')
+        text = text.replace('<', '&lt;')
+        text = text.replace('>', '&gt;')
+        return text
 
     def dedent(self, text, length=None):
         """Dedent raw text."""
@@ -349,7 +357,7 @@ def _end(self, block):
 
         mode = self.on_markdown()
         add = self.on_add(block)
-        if self.is_raw(add, mode):
+        if mode == 'raw' or (mode == 'auto' and self.is_raw(add)):
             add.text = mutil.AtomicString(self.dedent(add.text))
 
         self.on_end(block)

pymdownx/blocks/html.py

@@ -125,7 +125,7 @@ class HTML(Block):
     NAME = 'html'
     ARGUMENT = True
     OPTIONS = {
-        'markdown': ['auto', type_string_in(['auto', 'inline', 'block', 'raw'])]
+        'markdown': ['auto', type_string_in(['auto', 'inline', 'block', 'raw', 'html'])]
     }
 
     def __init__(self, length, tracker, md, config):
@@ -147,14 +147,31 @@ def on_validate(self, parent):
     def on_markdown(self):
         """Check if this is atomic."""
 
-        return self.options['markdown']
+        mode = self.options['markdown']
+        if mode == 'html':
+            mode = 'raw'
+        return mode
 
     def on_create(self, parent):
         """Create the element."""
 
         # Create element
         return etree.SubElement(parent, self.tag.lower(), self.attr)
 
+    def is_html(self, tag):
+        """Does tag require no processing and no HTML escaping."""
+
+        return tag.tag in ('script', 'style')
+
+    def on_end(self, block):
+        """On end event."""
+
+        mode = self.options['markdown']
+        if (mode == 'auto' and self.is_html(block)) or mode == 'html':
+            block.text = self.md.htmlStash.store(block.text)
+        elif (mode == 'auto' and self.is_raw(block)) or mode == 'raw':
+            block.text = self.md.htmlStash.store(self.html_escape(block.text))
+
 
 class HTMLExtension(BlocksExtension):
     """HTML Blocks Extension."""

tests/test_extensions/test_blocks/test_html.py

@@ -231,8 +231,8 @@ def test_multi_class2(self):
             True
         )
 
-    def test_inline_and_non_empty_parent(self):
-        """Test inline format that utilizes."""
+    def test_inline_and_md_in_html(self):
+        """Test inline format and HTML content."""
 
         self.check_markdown(
             R'''
@@ -258,8 +258,8 @@ def test_inline_and_non_empty_parent(self):
             True
         )
 
-    def test_raw_and_non_empty_parent(self):
-        """Test inline format that utilizes."""
+    def test_raw_and_md_in_html(self):
+        """Test raw format and HTML content."""
 
         self.check_markdown(
             R'''
@@ -282,3 +282,46 @@ def test_raw_and_non_empty_parent(self):
             ''',
             True
         )
+
+    def test_html_and_html(self):
+        """Test HTML mode format with HTML code."""
+
+        self.check_markdown(
+            R'''
+            /// html | div
+                markdown: html
+
+                <div>
+                **content**
+                </div>
+
+                this is <span>raw</span> **content**
+            ///
+            ''',
+            '''
+            <div><div>
+            **content**
+            </div>
+
+            this is <span>raw</span> **content**</div>
+            ''',
+            True
+        )
+
+    def test_html_and_script(self):
+        """Test inline format that script."""
+
+        self.check_markdown(
+            R'''
+            /// html | script
+
+                const el = document.querySelector('div');
+                el.innerHTML = '<span>test</span>
+            ///
+            ''',
+            '''
+            <script>const el = document.querySelector('div');
+            el.innerHTML = '<span>test</span></script>
+            ''',
+            True
+        )