DOC: error 'being used by another process' when removing temp files on Windows

[jorisvandenbossche]

When building the docs on Windows, I get some errors related to the removal of temporary files. For example, for building io.rst:

reading sources... [100%] io
---------------------------------------------------------------------------
WindowsError                              Traceback (most recent call last)
<ipython-input-124-e1699a291992> in <module>()
----> 1 os.remove('tmp.sv')
WindowsError: [Error 32] The process cannot access the file because it is being
used by another process: 'tmp.sv'
---------------------------------------------------------------------------
WindowsError                              Traceback (most recent call last)
<ipython-input-371-8e84b2979f2e> in <module>()
----> 1 os.remove('store.h5')
WindowsError: [Error 32] The process cannot access the file because it is being
used by another process: 'store.h5'

• For temp.sv this is due to:

  reader = pd.read_table('tmp.sv', sep='|', iterator=True)
  reader.get_chunk(5)
  

  where reader is still 'open'. A following os.remove('tmp.sv') will generate this error. Is there a way to close a TextFileReader object?

• For store.h5 this is due to the last os.remove('store.h5') (before 'SQL queries' http://pandas.pydata.org/pandas-docs/dev/io.html#sql-queries). For this, I don't directly see why this fails, as all the other calls to os.remove('store.h5') do work (and there is a store.close() command)

[jreback]

@jorisvandenbossche AFAIKT the final close/remove of store.h5 is fine (when building on linux). I have seen some of these types of errors on windows though in test_pytables.py. I think windows may delay the closing for some reason. I ended up just using a safe close/remove, maybe something like (I know this is just fixing sympton.....)

def safe_remove(store):
    try:
        store.close()
    except:
         pass
    try:
        os.remove(store.filename)
    except:
         pass

TextFileReader does not appear to have a close method, maybe add it?

[jreback]

@jorisvandenbossche is this still appearing?

[jorisvandenbossche]

Moved to 0.14, no blocker. Yes, this is still appearing, but I didn't have the time yet to try your suggestion.

[jreback]

@jorisvandenbossche this still an issue?

[jorisvandenbossche]

The 'tmp.sv' is still generating that error message yes. But maybe not that of a problem ..

[jorisvandenbossche]

The error when removing a hdf file seems to be solved. So it is only the tmp.sv file.

The issue there is that when using the chunksize or iterator keywords, returning a TextFileReader object, this object is not 'closed' after reading the file, also after reading in the full file:

In [48]: df = DataFrame(np.random.randn(10, 4))

In [49]: df.to_csv('tmp.sv', sep='|')

In [50]: reader = pd.read_table('tmp.sv', sep='|', iterator=True)

In [51]: os.remove('tmp.sv')
---------------------------------------------------------------------------
WindowsError                              Traceback (most recent call last)
<ipython-input-51-e1699a291992> in <module>()
----> 1 os.remove('tmp.sv')

WindowsError: [Error 32] The process cannot access the file because it is being
used by another process: 'tmp.sv'

In [52]: reader.read()
Out[52]:
   Unnamed: 0         0         1         2         3
0           0 -0.292821  0.400307  0.915909 -0.264468
1           1 -0.124990  0.169103  0.234643  0.075589
2           2 -0.537386  1.990975 -0.822839  0.629325
...

In [53]: os.remove('tmp.sv')
---------------------------------------------------------------------------
WindowsError                              Traceback (most recent call last)
<ipython-input-53-e1699a291992> in <module>()
----> 1 os.remove('tmp.sv')

WindowsError: [Error 32] The process cannot access the file because it is being
used by another process: 'tmp.sv'

This does not happen with just read_csv/table:

In [57]: df.to_csv('tmp2.sv', sep='|')

In [58]: pd.read_table('tmp2.sv', sep='|')
Out[58]:
   Unnamed: 0         0         1         2         3
0           0 -0.292821  0.400307  0.915909 -0.264468
1           1 -0.124990  0.169103  0.234643  0.075589
2           2 -0.537386  1.990975 -0.822839  0.629325
...

In [59]: os.remove('tmp2.sv')

[jreback]

I think this will fix the doc builds: jreback@4792fe8

the parser is dealloced (and the file closed) by __dealloc__, which I think occurs when the variable loses all references. It doesn't have an explict close handle because if you close it then try to use it again it will simply segfault. So I think that was intentional.

lmk

[bdforbes]

I get this error trying to build docs on Windows in pandas version 1.2.0.dev0+67.gaefae55e1.dirty. Is it a cause for concern?

[jorisvandenbossche]

It's not a big reason for concern if you simply want to build the docs (in which case you can ignore the error). It would still be nice to fixe the error in general though.