Improve myst workflow

[twiecki]

Me and many other contributors have had to fight with pre-commit and keeping the myst-file in sync. Hoping there are ways we can improve the workflow to make it less confusing. One thing I by default do is to remove the myst file and have it be re-created by pre-commit. Perhaps we could automatically have it be regenerated by pre-commit?

CC @OriolAbril

[OriolAbril]

I have no idea about possible solutions, but streamlining all this would be very nice. I think one of the issues is I don't understand how pre-commit works when it's not doing run --all. I do understand jupytext though so I often use --no-verify because I am positive the two formats are perfectly aligned but I can't commit if I run pre-commit. And the pre-commit checks on github do pass when I do that.

[twiecki]

At this point I would argue we should remove the myst_nb files and jupytext, it's not worth the hassle and I don't see a reason for us to carry around two formats where one works, especially now that github has better jupyter NB diffs.

I don't think jupytext fits into our workflow. For example, we run all these re-formatters on the NB files which always brings them out of sync, so conflicts are almost guaranteed. And I don't think we should drop those. And jupytext does not seem to have any functionality built for this, or allow you to specify to defer to the ipynb version when in doubt. It's also not trivial to just nuke and recreate everything.

So ultimately it's a little bonus for a few people (not me), but a huge hassle and confusion for everyone.

[twiecki]

#483 is hopefully good enough, I'll close this in the assumption that it will be.