DOC: test organization

doc/source/development/index.rst

@@ -16,6 +16,7 @@ Development
     code_style
     maintaining
     internals
+    test_writing
     extending
     developer
     policies

doc/source/development/test_writing.rst

@@ -0,0 +1,152 @@
+.. _test_organization:
+
+Test organization
+=================
+Ideally, there should be one, and only one, obvious place for a test to reside.
+Until we reach that ideal, these are some rules of thumb for where a test should
+be located.
+
+1. Does your test depend only on code in ``pd._libs.tslibs``?
+This test likely belongs in one of:
+
+    - tests.tslibs
+
+        .. note::
+
+             No file in ``tests.tslibs`` should import from any pandas modules outside of ``pd._libs.tslibs``
+
+    - tests.scalar
+    - tests.tseries.offsets
+
+2. Does your test depend only on code in pd._libs?
+This test likely belongs in one of:
+
+    - tests.libs
+    - tests.groupby.test_libgroupby
+
+3. Is your test for an arithmetic or comparison method?
+This test likely belongs in one of:
+
+    - tests.arithmetic
+
+        .. note::
+
+            These are intended for tests that can be shared to test the behavior of DataFrame/Series/Index/ExtensionArray using the ``box_with_array`` fixture.
+
+    - tests.frame.test_arithmetic
+    - tests.series.test_arithmetic
+
+4. Is your test for a reduction method (min, max, sum, prod, ...)?
+This test likely belongs in one of:
+
+    - tests.reductions
+
+        .. note::
+
+            These are intended for tests that can be shared to test the behavior of DataFrame/Series/Index/ExtensionArray.
+
+    - tests.frame.test_reductions
+    - tests.series.test_reductions
+    - tests.test_nanops
+
+5. Is your test for an indexing method?
+    This is the most difficult case for deciding where a test belongs, because
+    there are many of these tests, and many of them test more than one method
+    (e.g. both ``Series.__getitem__`` and ``Series.loc.__getitem__``)
+
+    A) Is the test specifically testing an Index method (e.g. ``Index.get_loc``, ``Index.get_indexer``)?
+    This test likely belongs in one of:
+
+        - tests.indexes.test_indexing
+        - tests.indexes.fooindex.test_indexing
+
+    Within that files there should be a method-specific test class e.g. ``TestGetLoc``.
+
+    In most cases, neither ``Series`` nor ``DataFrame`` objects should be needed in these tests.
+
+    B) Is the test for a Series or DataFrame indexing method *other* than ``__getitem__`` or ``__setitem__``, e.g. ``xs``, ``where``, ``take``, ``mask``, ``lookup``, or ``insert``?
+    This test likely belongs in one of:
+
+        - tests.frame.indexing.test_methodname
+        - tests.series.indexing.test_methodname
+
+    C) Is the test for any of ``loc``, ``iloc``, ``at``, or ``iat``?
+    This test likely belongs in one of:
+
+        - tests.indexing.test_loc
+        - tests.indexing.test_iloc
+        - tests.indexing.test_at
+        - tests.indexing.test_iat
+
+    Within the appropriate file, test classes correspond to either types of indexers (e.g. ``TestLocBooleanMask``) or major use cases (e.g. ``TestLocSetitemWithExpansion``).
+
+    See the note in section D) about tests that test multiple indexing methods.
+
+    D) Is the test for ``Series.__getitem__``, ``Series.__setitem__``, ``DataFrame.__getitem__``, or ``DataFrame.__setitem__``?
+    This test likely belongs in one of:
+
+        - tests.series.test_getitem
+        - tests.series.test_setitem
+        - tests.frame.test_getitem
+        - tests.frame.test_setitem
+
+    If many cases such a test may test multiple similar methods, e.g.
+
+    .. code-block:: python
+
+         import pandas as pd
+         import pandas._testing as tm
+
+         def test_getitem_listlike_of_ints():
+             ser = pd.Series(range(5))
+
+             result = ser[[3, 4]]
+             expected = pd.Series([2, 3])
+             tm.assert_series_equal(result, expected)
+
+             result = ser.loc[[3, 4]]
+             tm.assert_series_equal(result, expected)
+
+    In cases like this, the test location should be based on the *underlying* method being tested.  Or in the case of a test for a bugfix, the location of the actual bug.  So in this example, we know that ``Series.__getitem__`` calls ``Series.loc.__getitem__``, so this is *really* a test for ``loc.__getitem__``.  So this test belongs in ``tests.indexing.test_loc``
+
+6. Is your test for a DataFrame or Series method?
+    A) Is the method a plotting method?
+    This test likely belongs in one of:
+
+        - tests.plotting
+
+    B) Is the method an IO method?
+    This test likely belongs in one of:
+
+        - tests.io
+
+    C) Otherwise
+    This test likely belongs in one of:
+
+        - tests.series.methods.test_mymethod
+        - tests.frame.methods.test_mymethod
+
+            .. note::
+
+                If a test can be shared between DataFrame/Series using the ``frame_or_series`` fixture, by convention it goes in tests.frame file.
+
+        - tests.generic.methods.test_mymethod
+
+            .. note::
+
+                The generic/methods/ directory is only for methods with tests that are fully parametrized over Series/DataFrame
+
+7. Is your test for an Index method, not depending on Series/DataFrame?
+This test likely belongs in one of:
+
+    - tests.indexes
+
+8) Is your test for one of the pandas-provided ExtensionArrays (Categorical, DatetimeArray, TimedeltaArray, PeriodArray, IntervalArray, PandasArray, FloatArray, BoolArray, IntervalArray, StringArray)?
+This test likely belongs in one of:
+
+    - tests.arrays
+
+9) Is your test for *all* ExtensionArray subclasses (the "EA Interface")?
+This test likely belongs in one of:
+
+    - tests.extension