BUG: d3viz html template is not found after installation

[l8l]

Describe the issue:

I followed the example described here: https://pytensor.readthedocs.io/en/latest/library/d3viz/index.html

I did not change any line of the example code provided and everything executes fine (including the installation of pydot-ng) until the line d3v.d3viz(predict, 'examples/mlp.html'), where an error is thrown, even though the import statement import pytensor.d3viz as d3v returns no error.

The error is: No such file or directory: '<HOME>/.local/lib/python3.11/site-packages/pytensor/d3viz/html/template.html'. When checking the directory, I can see that the directory '<HOME>/.local/lib/python3.11/site-packages/pytensor/d3viz/ actually does exist but there is no /html/ folder inside.

I also made sure that graphviz is installed on the system.

What to do about this? Thanks!

Reproducable code example:

import pytensor.d3viz as d3v
d3v.d3viz(predict, 'examples/mlp.html')
# where predict is the function specified in this example: https://pytensor.readthedocs.io/en/latest/library/d3viz/index.html

Error message:

FileNotFoundError                         Traceback (most recent call last)
Cell In[16], line 2
      1 import pytensor.d3viz as d3v
----> 2 d3v.d3viz(predict, 'examples/mlp.html')

File ~/.local/lib/python3.11/site-packages/pytensor/d3viz/d3viz.py:88, in d3viz(fct, outfile, copy_deps, *args, **kwargs)
     86 # Read template HTML file
     87 template_file = os.path.join(__path__, "html", "template.html")
---> 88 with open(template_file) as f:
     89     template = f.read()
     91 # Copy dependencies to output directory

FileNotFoundError: [Errno 2] No such file or directory: '~/.local/lib/python3.11/site-packages/pytensor/d3viz/html/template.html'

PyTensor version information:

Pytensor Version 2.12.1, installed with pip
Python Version 3.11.3
Linux

Context for the issue:

The graph visualization feature would be very important for my work.

[ricardoV94]

The d3viz functionality is not being tested or really used by the developers, so I am not surprised it is broken.

Hopefully someone will be able to take a look to give feedback

[l8l]

Thanks for the post! I hope so too. It seems that one only needs to make sure some template html file is copied to the d3viz directory upon installation to fix the issue.

You're saying that developers don't really use this feature? But isn't the graph visualization quite useful? I think it is a really cool feature, distinguishing it from other computational graph frameworks that is worth maintaining!

[ricardoV94]

Yeah, I think it could be useful, I just haven't seen much use (or any at all) since we forked from Theano/Aesara.

Other libraries usually rely on a condensed type of graph, for instance just showing relationship between named nodes, such as this utility in PyMC:
https://www.pymc.io/projects/docs/en/stable/api/generated/pymc.model_to_graphviz.html

Some example outputs in:
https://www.pymc.io/projects/examples/en/latest/case_studies/multilevel_modeling.html

For pure PyTensor graphs I usually just use dprint, I find a graphical form of all operations too much to digest in practice.

---

Anyway, in the codebase, it seems we have just these tests: https://github.com/pymc-devs/pytensor/blob/main/tests/d3viz/test_d3viz.py

However it doesn't seem the dependency is being installed according to the CI logs: https://github.com/pymc-devs/pytensor/actions/runs/5212573519/jobs/9406421099#step:6:1730

Even though we request it be installed:

pytensor/.github/workflows/test.yml

Line 142 in ec6a315

mamba install --yes -q "python~=${PYTHON_VERSION}=*_cpython" mkl numpy scipy pip mkl-service graphviz cython pytest coverage pytest-cov pytest-benchmark sympy

So a first step, would be to try and run those tests locally and see if they pass or are broken.

[maresb]

I think this is a setuptools issue. One probably needs to add the correct files to MANIFEST.in. Or even better, switch to Hatch so that we don't have to deal with setuptools weirdness.

[ricardoV94]

[Deleted wrong comment]

[ricardoV94]

I think this is a setuptools issue. One probably needs to add the correct files to MANIFEST.in. Or even better, switch to Hatch so that we don't have to deal with setuptools weirdness.

You mean the specific template.html issue?

For the tests it seems we actually have to install pydot and not just graphviz:

pytensor/pytensor/printing.py

Lines 54 to 61 in ec6a315

	if hasattr(pd, "find_graphviz"):
	if pd.find_graphviz():
	pydot_imported = True
	else:
	pydot_imported_msg = "pydot can't find graphviz"
	else:
	pd.Dot.create(pd.Dot())
	pydot_imported = True

However I am not sure this package is still maintaned? Last release was February 2021: https://pypi.org/project/pydot/#history

[maresb]

Ah, I meant the original issue, not the CI tests

[ricardoV94]

Okay, so we should fix the setup to handle those path issues correctly and decide whether we want to test pydot/graphviz support or not.

If not, I guess there's no other reason to install graphviz either

[ricardoV94]

The pydot_ng recommendation should definitely be dropped and the code logic as well, since that project is officially archived in favor of pydot: https://github.com/pydot/pydot-ng

[l8l]

Thanks everyone for trying to solve the problem!
@ricardoV94 : I tried model_to_graphviz() and dprint() that you recommended and they seem nice and useful but the first one has much less information about the computational graph than I need and the second one is not very visual (and only for single tensors). So I strongly vote in favor of maintaining the pydot-d3viz implementation.
Yeah, it sounds reasonable to replace the pydot_ng recommendation by pydot.

[ricardoV94]

Sure thanks for the feedback.

dprint allows multiple tensors. Just pass a list with multiple variables, or you can pass a compiled function as well.

But I agree it's a different use case.

Would you have availability to update the docs regarding the ng library?

Also @maresb do you have more specific recommendations for what the fix could look like?

[l8l]

I committed a pull request to fix the -ng-recommendation: #336

[ricardoV94]

It seems this file used to exist in Theano, and no longer exists in PyTensor: https://github.com/Theano/Theano/blob/master/theano/d3viz/html/template.html

[ricardoV94]

It was lost in the rename to PyTensor: 19e1a98#diff-976e71f8933be25c4bd13d49f8ebf4c8b9f524585a85b1303409ffc44182d577