Merge commit 'v0.15.1-108-ge9ee47c' into debian

* commit 'v0.15.1-108-ge9ee47c': (122 commits)
  BUG: Option context applies on __enter__
  CLN: move import to top of file
  DOC: specify return type in to_datetime
  Doc change for Issue pandas-dev#8805
  Fix unrecognized 'Z' UTC designator
  BUG CSV: fix problem with trailing whitespace in skipped rows, issues 8661, 8679 ENH CSV: Reduce memory usage when skiprows is an integer in read_csv, issue 8681
  DOC: v0.15.2.txt fixups
  BUG/ENH: cleanup for Timedelta arithmetic
  COMPAT: windows compat for tests for dtype inference in parser xref (GH8833)
  DOC: Add Nano to offsets table
  DOC: Add where and mask to API doc
  TST: provide a clear argument to assert_produces_warning to guarantee that we are clearing      specific warnings before trying to catch them (in case they have already happened)      requires the user to specificy the exact class where they are coming
  BUG: missing nose import for skip test
  index into multi-index past the lexsort depth
  BUG: type change breaks BlockManager integrity
  DOC: Clean up Stata documentation
  Clarify encoding kwarg on to_csv
  BUG: fix negative step support for label-based slices
  BUG: Correct importing behavior for Categoricals in StataReader
  DOC: api updates for .size
  ...

Author: yarikoptic

.travis.yml

@@ -54,7 +54,7 @@ matrix:
       - JOB_NAME: "34_nslow"
     - python: 3.2
       env:
-      - NOSE_ARGS="not slow and not disabled"
+      - NOSE_ARGS="not slow and not network and not disabled"
       - FULL_DEPS=true
       - CLIPBOARD_GUI=qt4
       - BUILD_TYPE=pydata
@@ -71,7 +71,7 @@ matrix:
     allow_failures:
       - python: 3.2
         env:
-        - NOSE_ARGS="not slow and not disabled"
+        - NOSE_ARGS="not slow and not network and not disabled"
         - FULL_DEPS=true
         - CLIPBOARD_GUI=qt4
         - BUILD_TYPE=pydata

ci/requirements-3.2.txt

@@ -9,6 +9,7 @@ tables==3.0.0
 matplotlib==1.2.1
 patsy==0.1.0
 lxml==3.2.1
+html5lib
 scipy==0.12.0
 beautifulsoup4==4.2.1
 statsmodels==0.5.0

doc/README.rst

@@ -88,13 +88,12 @@ Furthermore, it is recommended to have all `optional dependencies
 installed. This is not needed, but be aware that you will see some error
 messages. Because all the code in the documentation is executed during the doc
 build, the examples using this optional dependencies will generate errors.
-Run ``pd.show_version()`` to get an overview of the installed version of all
+Run ``pd.show_versions()`` to get an overview of the installed version of all
 dependencies.
 
 .. warning::
 
-   Building the docs with Sphinx version 1.2 is broken. Use the
-   latest stable version (1.2.1) or the older 1.1.3.
+   Sphinx version >= 1.2.2 or the older 1.1.3 is required.
 
 Building pandas
 ^^^^^^^^^^^^^^^
@@ -133,7 +132,7 @@ If you want to do a full clean build, do::
     python make.py build
 
 
-Staring with 0.13.1 you can tell ``make.py`` to compile only a single section
+Starting with 0.13.1 you can tell ``make.py`` to compile only a single section
 of the docs, greatly reducing the turn-around time for checking your changes.
 You will be prompted to delete `.rst` files that aren't required, since the
 last committed version can always be restored from git.

doc/make.py

@@ -31,48 +31,48 @@
 SPHINX_BUILD = 'sphinxbuild'
 
 
-def upload_dev():
+def upload_dev(user='pandas'):
     'push a copy to the pydata dev directory'
-    if os.system('cd build/html; rsync -avz . pandas@pandas.pydata.org'
-                 ':/usr/share/nginx/pandas/pandas-docs/dev/ -essh'):
+    if os.system('cd build/html; rsync -avz . {0}@pandas.pydata.org'
+                 ':/usr/share/nginx/pandas/pandas-docs/dev/ -essh'.format(user)):
         raise SystemExit('Upload to Pydata Dev failed')
 
 
-def upload_dev_pdf():
+def upload_dev_pdf(user='pandas'):
     'push a copy to the pydata dev directory'
-    if os.system('cd build/latex; scp pandas.pdf pandas@pandas.pydata.org'
-                 ':/usr/share/nginx/pandas/pandas-docs/dev/'):
+    if os.system('cd build/latex; scp pandas.pdf {0}@pandas.pydata.org'
+                 ':/usr/share/nginx/pandas/pandas-docs/dev/'.format(user)):
         raise SystemExit('PDF upload to Pydata Dev failed')
 
 
-def upload_stable():
+def upload_stable(user='pandas'):
     'push a copy to the pydata stable directory'
-    if os.system('cd build/html; rsync -avz . pandas@pandas.pydata.org'
-                 ':/usr/share/nginx/pandas/pandas-docs/stable/ -essh'):
+    if os.system('cd build/html; rsync -avz . {0}@pandas.pydata.org'
+                 ':/usr/share/nginx/pandas/pandas-docs/stable/ -essh'.format(user)):
         raise SystemExit('Upload to stable failed')
 
 
-def upload_stable_pdf():
+def upload_stable_pdf(user='pandas'):
     'push a copy to the pydata dev directory'
-    if os.system('cd build/latex; scp pandas.pdf pandas@pandas.pydata.org'
-                 ':/usr/share/nginx/pandas/pandas-docs/stable/'):
+    if os.system('cd build/latex; scp pandas.pdf {0}@pandas.pydata.org'
+                 ':/usr/share/nginx/pandas/pandas-docs/stable/'.format(user)):
         raise SystemExit('PDF upload to stable failed')
 
 
-def upload_prev(ver, doc_root='./'):
+def upload_prev(ver, doc_root='./', user='pandas'):
     'push a copy of older release to appropriate version directory'
     local_dir = doc_root + 'build/html'
     remote_dir = '/usr/share/nginx/pandas/pandas-docs/version/%s/' % ver
-    cmd = 'cd %s; rsync -avz . pandas@pandas.pydata.org:%s -essh'
-    cmd = cmd % (local_dir, remote_dir)
+    cmd = 'cd %s; rsync -avz . %s@pandas.pydata.org:%s -essh'
+    cmd = cmd % (local_dir, user, remote_dir)
     print(cmd)
     if os.system(cmd):
         raise SystemExit(
             'Upload to %s from %s failed' % (remote_dir, local_dir))
 
     local_dir = doc_root + 'build/latex'
-    pdf_cmd = 'cd %s; scp pandas.pdf pandas@pandas.pydata.org:%s'
-    pdf_cmd = pdf_cmd % (local_dir, remote_dir)
+    pdf_cmd = 'cd %s; scp pandas.pdf %s@pandas.pydata.org:%s'
+    pdf_cmd = pdf_cmd % (local_dir, user, remote_dir)
     if os.system(pdf_cmd):
         raise SystemExit('Upload PDF to %s from %s failed' % (ver, doc_root))
 
@@ -337,6 +337,10 @@ def generate_index(api=True, single=False, **kwds):
                    type=str,
                    default=False,
                    help='filename of section to compile, e.g. "indexing"')
+argparser.add_argument('--user',
+                   type=str,
+                   default=False,
+                   help='Username to connect to the pydata server')
 
 def main():
     args, unknown = argparser.parse_known_args()
@@ -354,16 +358,19 @@ def main():
         ver = sys.argv[2]
 
         if ftype == 'build_previous':
-            build_prev(ver)
+            build_prev(ver, user=args.user)
         if ftype == 'upload_previous':
-            upload_prev(ver)
+            upload_prev(ver, user=args.user)
     elif len(sys.argv) == 2:
         for arg in sys.argv[1:]:
             func = funcd.get(arg)
             if func is None:
                 raise SystemExit('Do not know how to handle %s; valid args are %s' % (
                     arg, list(funcd.keys())))
-            func()
+            if args.user:
+                func(user=args.user)
+            else:
+                func()
     else:
         small_docs = False
         all()

doc/source/api.rst

@@ -190,6 +190,8 @@ Standard moving window functions
    rolling_quantile
    rolling_window
 
+.. _api.functions_expanding:
+
 Standard expanding window functions
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
@@ -248,9 +250,9 @@ Attributes
    Series.dtype
    Series.ftype
    Series.shape
-   Series.size
    Series.nbytes
    Series.ndim
+   Series.size
    Series.strides
    Series.itemsize
    Series.base
@@ -392,6 +394,8 @@ Reindexing / Selection / Label manipulation
    Series.take
    Series.tail
    Series.truncate
+   Series.where
+   Series.mask
 
 Missing data handling
 ~~~~~~~~~~~~~~~~~~~~~
@@ -652,6 +656,7 @@ Attributes and underlying data
    DataFrame.values
    DataFrame.axes
    DataFrame.ndim
+   DataFrame.size
    DataFrame.shape
 
 Conversion
@@ -686,6 +691,8 @@ Indexing, iteration
    DataFrame.tail
    DataFrame.xs
    DataFrame.isin
+   DataFrame.where
+   DataFrame.mask
    DataFrame.query
 
 For more information on ``.at``, ``.iat``, ``.ix``, ``.loc``, and
@@ -919,6 +926,7 @@ Attributes and underlying data
    Panel.values
    Panel.axes
    Panel.ndim
+   Panel.size
    Panel.shape
    Panel.dtypes
    Panel.ftypes
@@ -1126,6 +1134,7 @@ Attributes and underlying data
    Panel4D.values
    Panel4D.axes
    Panel4D.ndim
+   Panel4D.size
    Panel4D.shape
    Panel4D.dtypes
    Panel4D.ftypes
@@ -1164,14 +1173,16 @@ Attributes
 
    Index.values
    Index.is_monotonic
+   Index.is_monotonic_increasing
+   Index.is_monotonic_decreasing
    Index.is_unique
    Index.dtype
    Index.inferred_type
    Index.is_all_dates
    Index.shape
-   Index.size
    Index.nbytes
    Index.ndim
+   Index.size
    Index.strides
    Index.itemsize
    Index.base

doc/source/categorical.rst

@@ -236,7 +236,7 @@ which are removed are replaced by ``np.nan``.:
     s = s.cat.remove_categories([4])
     s
 
-Renaming unused categories
+Removing unused categories
 ~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 Removing unused categories can also be done:
@@ -541,8 +541,13 @@ The same applies to ``df.append(df_different)``.
 Getting Data In/Out
 -------------------
 
-Writing data (`Series`, `Frames`) to a HDF store that contains a ``category`` dtype will currently
-raise ``NotImplementedError``.
+.. versionadded:: 0.15.2
+
+Writing data (`Series`, `Frames`) to a HDF store that contains a ``category`` dtype was implemented
+in 0.15.2. See :ref:`here <io.hdf5-categorical>` for an example and caveats.
+
+Writing data to and reading data from *Stata* format files was implemented in
+0.15.2. See :ref:`here <io.stata-categorical>` for an example and caveats.
 
 Writing to a CSV file will convert the data, effectively removing any information about the
 categorical (categories and ordering). So if you read back the CSV file you have to convert the
@@ -573,6 +578,7 @@ relevant columns back to `category` and assign the right categories and categori
     df2.dtypes
     df2["cats"]
 
+The same holds for writing to a SQL database with ``to_sql``.
 
 Missing Data
 ------------
@@ -804,4 +810,3 @@ Use ``copy=True`` to prevent such a behaviour or simply don't reuse `Categorical
     This also happens in some cases when you supply a `numpy` array instead of a `Categorical`:
     using an int array (e.g. ``np.array([1,2,3,4])``) will exhibit the same behaviour, while using
     a string array (e.g. ``np.array(["a","b","c","a"])``) will not.
-

doc/source/conf.py

@@ -44,12 +44,9 @@
               'ipython_sphinxext.ipython_directive',
               'ipython_sphinxext.ipython_console_highlighting',
               'sphinx.ext.intersphinx',
-              'sphinx.ext.todo',
               'sphinx.ext.coverage',
               'sphinx.ext.pngmath',
               'sphinx.ext.ifconfig',
-              'matplotlib.sphinxext.only_directives',
-              'matplotlib.sphinxext.plot_directive',
               ]
 
 

doc/source/cookbook.rst

@@ -588,6 +588,19 @@ Unlike agg, apply's callable is passed a sub-DataFrame which gives you access to
    df['beyer_shifted'] = df.groupby(level=0)['beyer'].shift(1)
    df
 
+`Select row with maximum value from each group
+<http://stackoverflow.com/q/26701849/190597>`__
+
+.. ipython:: python
+
+   df = pd.DataFrame({'host':['other','other','that','this','this'],
+                      'service':['mail','web','mail','mail','web'],
+                      'no':[1, 2, 1, 2, 1]}).set_index(['host', 'service'])
+   mask = df.groupby(level=0).agg('idxmax')
+   df_count = df.loc[mask['no']].reset_index()
+   df_count
+
+
 Expanding Data
 **************
 
@@ -1202,4 +1215,4 @@ of the data values:
       {'height': [60, 70],
        'weight': [100, 140, 180],
        'sex': ['Male', 'Female']})
-   df
+   df

doc/source/faq.rst

@@ -50,6 +50,10 @@ when calling ``df.info()``:
 
     df.info()
 
+The ``+`` symbol indicates that the true memory usage could be higher, because
+pandas does not count the memory used by values in columns with
+``dtype=object``.
+
 By default the display option is set to ``True`` but can be explicitly
 overridden by passing the ``memory_usage`` argument when invoking ``df.info()``.
 

doc/source/groupby.rst

@@ -338,6 +338,21 @@ In the case of grouping by multiple keys, the group name will be a tuple:
 It's standard Python-fu but remember you can unpack the tuple in the for loop
 statement if you wish: ``for (k1, k2), group in grouped:``.
 
+Selecting a group
+-----------------
+
+A single group can be selected using ``GroupBy.get_group()``:
+
+.. ipython:: python
+
+   grouped.get_group('bar')
+   
+Or for an object grouped on multiple columns:
+
+.. ipython:: python
+
+   df.groupby(['A', 'B']).get_group(('bar', 'one'))
+
 .. _groupby.aggregate:
 
 Aggregation