CI add end-of-file-fixer

[MarcoGorelli]

xref #36600 (comment)

ok if you'd fix the EOF issue can merge (and yes good idea to add to the pre-commit hook / separate PR pls)

Note: in addition to adding the end-of-file-fixer pre-commit hook, I applied the following

diff --git a/scripts/generate_pip_deps_from_conda.py b/scripts/generate_pip_deps_from_conda.py
index c417f58f6..c6d00eb58 100755
--- a/scripts/generate_pip_deps_from_conda.py
+++ b/scripts/generate_pip_deps_from_conda.py
@@ -94,7 +94,7 @@ def main(conda_fname, pip_fname, compare=False):
         f"# This file is auto-generated from {fname}, do not modify.\n"
         "# See that file for comments about the need/usage of each dependency.\n\n"
     )
-    pip_content = header + "\n".join(pip_deps)
+    pip_content = header + "\n".join(pip_deps) + "\n"
 
     if compare:
         with open(pip_fname) as pip_fd:

---

I don't know how many of these we want to apply this to, for now I'm just excluding licenses (which I don't think should be touched?), html, txt, svg,and csv files

[dsaxton]

I actually get an error when I try to use this or the yesqa hook (it's the same error), do you know what this might be about:

(pandas-dev) ➜  pandas git:(end-of-file-fixer) git commit --allow-empty
[INFO] Initializing environment for https://github.com/pre-commit/pre-commit-hooks.
[INFO] Installing environment for https://github.com/asottile/yesqa.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
An unexpected error has occurred: CalledProcessError: command: ('/Users/danielsaxton/opt/miniconda3/envs/pandas-dev/bin/python', '-mvirtualenv', '/Users/danielsaxton/.cache/pre-commit/repowwvsgzcw/py_env-python3.8', '-p', '/Users/danielsaxton/opt/miniconda3/envs/pandas-dev/bin/python')
return code: 1
expected return code: 0
stdout:
    ModuleNotFoundError: No module named 'virtualenv.seed.via_app_data'

stderr: (none)
Check the log at /Users/danielsaxton/.cache/pre-commit/pre-commit.log

Full log:

version information

pre-commit version: 2.7.1
sys.version:
    3.8.5 | packaged by conda-forge | (default, Sep 24 2020, 16:37:41) 
    [Clang 10.0.1 ]
sys.executable: /Users/danielsaxton/opt/miniconda3/envs/pandas-dev/bin/python
os.name: posix
sys.platform: darwin

error information

An unexpected error has occurred: CalledProcessError: command: ('/Users/danielsaxton/opt/miniconda3/envs/pandas-dev/bin/python', '-mvirtualenv', '/Users/danielsaxton/.cache/pre-commit/repowwvsgzcw/py_env-python3.8', '-p', '/Users/danielsaxton/opt/miniconda3/envs/pandas-dev/bin/python')
return code: 1
expected return code: 0
stdout:
    ModuleNotFoundError: No module named 'virtualenv.seed.via_app_data'
    
stderr: (none)

Traceback (most recent call last):
  File "/Users/danielsaxton/opt/miniconda3/envs/pandas-dev/lib/python3.8/site-packages/pre_commit/error_handler.py", line 63, in error_handler
    yield
  File "/Users/danielsaxton/opt/miniconda3/envs/pandas-dev/lib/python3.8/site-packages/pre_commit/main.py", line 362, in main
    return hook_impl(
  File "/Users/danielsaxton/opt/miniconda3/envs/pandas-dev/lib/python3.8/site-packages/pre_commit/commands/hook_impl.py", line 217, in hook_impl
    return retv | run(config, store, ns)
  File "/Users/danielsaxton/opt/miniconda3/envs/pandas-dev/lib/python3.8/site-packages/pre_commit/commands/run.py", line 388, in run
    install_hook_envs(hooks, store)
  File "/Users/danielsaxton/opt/miniconda3/envs/pandas-dev/lib/python3.8/site-packages/pre_commit/repository.py", line 206, in install_hook_envs
    _hook_install(hook)
  File "/Users/danielsaxton/opt/miniconda3/envs/pandas-dev/lib/python3.8/site-packages/pre_commit/repository.py", line 82, in _hook_install
    lang.install_environment(
  File "/Users/danielsaxton/opt/miniconda3/envs/pandas-dev/lib/python3.8/site-packages/pre_commit/languages/python.py", line 213, in install_environment
    cmd_output_b(*venv_cmd, cwd='/')
  File "/Users/danielsaxton/opt/miniconda3/envs/pandas-dev/lib/python3.8/site-packages/pre_commit/util.py", line 157, in cmd_output_b
    raise CalledProcessError(returncode, cmd, retcode, stdout_b, stderr_b)
pre_commit.util.CalledProcessError: command: ('/Users/danielsaxton/opt/miniconda3/envs/pandas-dev/bin/python', '-mvirtualenv', '/Users/danielsaxton/.cache/pre-commit/repowwvsgzcw/py_env-python3.8', '-p', '/Users/danielsaxton/opt/miniconda3/envs/pandas-dev/bin/python')
return code: 1
expected return code: 0
stdout:
    ModuleNotFoundError: No module named 'virtualenv.seed.via_app_data'
    
stderr: (none)

[MarcoGorelli]

I've seen that before, yes - I think pip install -U virtualenv?

[dsaxton]

I did that and a different error happens:

(pandas-dev) ➜  pandas git:(dummy-branch) git commit --allow-empty
[INFO] Installing environment for https://github.com/asottile/yesqa.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
An unexpected error has occurred: CalledProcessError: command: ('/Users/danielsaxton/opt/miniconda3/envs/pandas-dev/bin/python', '-mvirtualenv', '/Users/danielsaxton/.cache/pre-commit/repowwvsgzcw/py_env-python3.8', '-p', '/Users/danielsaxton/opt/miniconda3/envs/pandas-dev/bin/python')
return code: 1
expected return code: 0
stdout:
    ImportError: cannot import name 'ensure_file_on_disk' from 'virtualenv.util.zipapp' (/Users/danielsaxton/opt/miniconda3/envs/pandas-dev/lib/python3.8/site-packages/virtualenv/util/zipapp.py)
    
stderr: (none)
Check the log at /Users/danielsaxton/.cache/pre-commit/pre-commit.log

Might be the same issue as here: pypa/virtualenv#1876

[dsaxton]

I think I was able to fix it by uninstalling virtualenv in my base conda environment and then reinstalling virtualenv in my pandas-dev environment, but I wonder if this warrants some type of note somewhere for anyone who might have the same problem?

[dsaxton]

I wonder if we might actually want to exclude .py files and only check others (e.g., .yml), since I think .py files will already fail black if they're missing the EOF newline?