DOC: Add details to DataFrame groupby transform

Add requirements for user function in groupby transform

closes #13543
[skip ci]

Author: bashtage

doc/source/groupby.rst

@@ -580,9 +580,17 @@ Transformation
 --------------
 
 The ``transform`` method returns an object that is indexed the same (same size)
-as the one being grouped. Thus, the passed transform function should return a
-result that is the same size as the group chunk. For example, suppose we wished
-to standardize the data within each group:
+as the one being grouped. The transform function must:
+
+* Return a result that is either the same size as the group chunk or
+  broadcastable to the size of the group chunk.
+* Operate column-by-column on the group chunk.  A fast path is used if the
+  transform function also operates on the entire group chunk as a DataFrame.
+* Does not perform in-place operations on the group chunk. Group chunks should
+  be treated as immutable, and changes to a group chunk may produce unexpected
+  results.
+
+For example, suppose we wished to standardize the data within each group:
 
 .. ipython:: python
 
@@ -620,6 +628,14 @@ We can also visually compare the original and transformed data sets.
    @savefig groupby_transform_plot.png
    compare.plot()
 
+Transformation functions that have lower dimension outputs are broadcast to
+match the shape of the input array.
+
+.. ipython:: python
+
+   extremum = lambda x: x.max() - x.min()
+   transformed = ts.groupby(key).transform(extremum)
+
 Another common data transform is to replace missing data with the group mean.
 
 .. ipython:: python
@@ -664,8 +680,9 @@ and that the transformed data contains no NAs.
 
 .. note::
 
-   Some functions when applied to a groupby object will automatically transform the input, returning
-   an object of the same shape as the original. Passing ``as_index=False`` will not affect these transformation methods.
+   Some functions when applied to a groupby object will automatically transform
+   the input, returning an object of the same shape as the original. Passing
+   ``as_index=False`` will not affect these transformation methods.
 
    For example: ``fillna, ffill, bfill, shift``.
 

pandas/core/groupby.py

@@ -3548,10 +3548,25 @@ def transform(self, func, *args, **kwargs):
         Each subframe is endowed the attribute 'name' in case you need to know
         which group you are working on.
 
+        The current implementation imposes three requirements on f:
+
+        * f must return a value that either has the same shape as the input
+          subframe or can be broadcast to the shape of the input subframe.
+          For example, f returns a scalar it will be broadcast to have the
+          same shape as the input subframe.
+        * f must support application column-by-column in the subframe. An
+          optional fast path is used if f also supports application to the
+          entire subframe.
+        * f must not mutate subframes. Mutation is not supported and may
+          produce unexpected results.
+
         Examples
         --------
         >>> grouped = df.groupby(lambda x: mapping[x])
+        # Same shape
         >>> grouped.transform(lambda x: (x - x.mean()) / x.std())
+        # Broadcastable
+        >>> grouped.transform(lambda x: x.max() - x.min())
         """
 
         # optimized transforms