BUG: A read-only DataFrame cannot be .diff()'ed

[dycw]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.

Code Sample, a copy-pastable example

import numpy as np
import pandas as pd

data = np.ones(2, dtype=int)
data.flags.writeable = False
df = pd.DataFrame(data)
df.diff()

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-2-df60a870b020> in <module>
      2 data.flags.writeable = False
      3 df = pd.DataFrame(data)
----> 4 df.diff()

~/miniconda3/envs/dts/lib/python3.8/site-packages/pandas/core/frame.py in diff(self, periods, axis)
   7247             return self.T.diff(periods, axis=0).T
   7248 
-> 7249         new_data = self._mgr.diff(n=periods, axis=bm_axis)
   7250         return self._constructor(new_data)
   7251 

~/miniconda3/envs/dts/lib/python3.8/site-packages/pandas/core/internals/managers.py in diff(self, n, axis)
    546 
    547     def diff(self, n: int, axis: int) -> "BlockManager":
--> 548         return self.apply("diff", n=n, axis=axis)
    549 
    550     def interpolate(self, **kwargs) -> "BlockManager":

~/miniconda3/envs/dts/lib/python3.8/site-packages/pandas/core/internals/managers.py in apply(self, f, align_keys, **kwargs)
    394                 applied = b.apply(f, **kwargs)
    395             else:
--> 396                 applied = getattr(b, f)(**kwargs)
    397             result_blocks = _extend_blocks(applied, result_blocks)
    398 

~/miniconda3/envs/dts/lib/python3.8/site-packages/pandas/core/internals/blocks.py in diff(self, n, axis)
   1265     def diff(self, n: int, axis: int = 1) -> List["Block"]:
   1266         """ return block for the diff of the values """
-> 1267         new_values = algos.diff(self.values, n, axis=axis, stacklevel=7)
   1268         return [self.make_block(values=new_values)]
   1269 

~/miniconda3/envs/dts/lib/python3.8/site-packages/pandas/core/algorithms.py in diff(arr, n, axis, stacklevel)
   1914         # TODO: can diff_2d dtype specialization troubles be fixed by defining
   1915         #  out_arr inside diff_2d?
-> 1916         algos.diff_2d(arr, out_arr, n, axis)
   1917     else:
   1918         # To keep mypy happy, _res_indexer is a list while res_indexer is

pandas/_libs/algos.pyx in pandas._libs.algos.diff_2d()

~/miniconda3/envs/dts/lib/python3.8/site-packages/pandas/_libs/algos.cpython-38-x86_64-linux-gnu.so in View.MemoryView.memoryview_cwrapper()

~/miniconda3/envs/dts/lib/python3.8/site-packages/pandas/_libs/algos.cpython-38-x86_64-linux-gnu.so in View.MemoryView.memoryview.__cinit__()

ValueError: buffer source array is read-only

Problem description

df.diff() does not seem like a data-mutating operation, at least not to me. My read is that df.iloc[] was given the same assessment in 2015 (#10043).

Expected Output

        0
0     nan
1 0.00000

Output of pd.show_versions()

``` INSTALLED VERSIONS ------------------ commit : d9fff27 python : 3.8.3.final.0 python-bits : 64 OS : Linux OS-release : 5.4.0-42-generic Version : #46-Ubuntu SMP Fri Jul 10 00:24:02 UTC 2020 machine : x86_64 processor : x86_64 byteorder : little LC_ALL : None LANG : en_HK.UTF-8 LOCALE : en_HK.UTF-8

pandas : 1.1.0
numpy : 1.19.1
pytz : 2020.1
dateutil : 2.8.1
pip : 20.1.1
setuptools : 49.2.0.post20200714
Cython : None
pytest : 6.0.1
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : 2.8.5 (dt dec pq3 ext lo64)
jinja2 : 2.11.2
IPython : 7.16.1
pandas_datareader: None
bs4 : 4.9.1
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : 3.2.2
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pytables : None
pyxlsb : None
s3fs : None
scipy : 1.5.0
sqlalchemy : 1.3.18
tables : None
tabulate : 0.8.3
xarray : None
xlrd : None
xlwt : None
numba : None

</details>

[jbrockmendel]

@WillAyd could use your help here.

The solution should be to go into algos.pyx and change the first argument in diff_2d from diff_t[:, :] arr to either const diff_t[:, :] arr or ndarray[diff_t, ndim=2] arr.

With the first variant, I get a cython compile-time exception that I think is because fused types dont support const (maybe in the upcoming cython release? not sure off the top of my head)

With the ndarray variant, I get clang-compile-time errors:

pandas/_libs/algos.c:81768:3: error: code will never be executed [-Werror,-Wunreachable-code]
  __Pyx_SafeReleaseBuffer(&__pyx_pybuffernd_arr.rcbuffer->pybuffer);

(Note: the ndarray version also requires changing bint f_contig = arr.is_fcontig() to bint f_contig = arr.flags["F_CONTIGUOUS"])

Thoughts on how to handle this?

[WillAyd]

Hmm yea both of those are unfortunate. I do wonder to your point if we can get Cython 3.0 to work with the const declaration, as that would be the cleanest

[jorisvandenbossche]

(this is a regression compared to 1.0, so tagging for 1.1.1)

[simonjayhawkins]

(this is a regression compared to 1.0, so tagging for 1.1.1)

regression in #31935

The solution should be to go into algos.pyx and change the first argument in diff_2d from diff_t[:, :] arr to either const diff_t[:, :] arr or ndarray[diff_t, ndim=2] arr.

see https://github.com/pandas-dev/pandas/pull/31935/files#r378496858

[jorisvandenbossche]

So live with the clang warnings for now? (I don't think that cython 3 is an option for 1.1.1)

[jbrockmendel]

So live with the clang warnings for now? (I don't think that cython 3 is an option for 1.1.1)

@WillAyd Are there build flags we can change in setup.py that will turn these from clang errors into clang warnings?

[WillAyd]

Yea you can just use -Wno-error=<error_name> - we do this already for Py38 here:

pandas/setup.py

Line 458 in 3701a9b

extra_compile_args.append("-Wno-error=deprecated-declarations")

Though if you can create an MRE it's worth reporting upstream. Cython does consider dead code generation to be a bug but doesn't explicitly test for it yet

[jbrockmendel]

Though if you can create an MRE it's worth reporting upstream

Is upstream in this case cython or somewhere else?

[WillAyd]

Yea this would be a Cython issue