Merge commit 'v0.8.1-203-g67121af' into debian

* commit 'v0.8.1-203-g67121af': (193 commits)
  BUG: DataFrame column formatting issue in length-truncated column close pandas-dev#1906
  BUG: override min/max in DatetimeIndex to function as expected close pandas-dev#1895
  BUG: DataFrame mixed-type arithmetic column-wise, fix DataFrame.diff upcasting->object bug close pandas-dev#1896
  BUG: treat nobs=1 >= min_periods case in rolling_std/variance as 0 trivially. close pandas-dev#1884
  TST: skip to_file test if URLError occurs on some systems
  VB: resolve test name conflict and update make script
  DOC: minor change to build script to help auto build process
  DOC: fixed extlinks in sphinx conf
  TST: oops import in wrong place
  TST: skip test_console_encode if sys.stdin.encoding is None
  TST: unit test for pandas-dev#1902 and default to csv.QUOTE_MINIMAL
  Make it possible to set quoting for to_csv
  ENH: clean up pandas-dev#1691 changes, rls note
  ENH: add more possible bool values to read_csv pandas-dev#1295
  BUG: fix rolling_max/min for small inputs and large windows. Add a check that the min_period <= window size. Fixes pandas-dev#1897.
  Mention Ubuntu for NeuroDebian repository
  BUG: don't clobber color keyword in Series.plot, close pandas-dev#1890
  DOC: add intersphinx mapping for python library, close pandas-dev#1556
  BUG: fix mixed-integer .ix indexing bugs. close#1799
  BUG: unicode sheet name in to_excel pandas-dev#1828
  ...

Author: yarikoptic

RELEASE.rst

@@ -22,6 +22,146 @@ Where to get it
 * Binary installers on PyPI: http://pypi.python.org/pypi/pandas
 * Documentation: http://pandas.pydata.org
 
+pandas 0.9.0
+============
+
+**Release date:** NOT YET RELEASED
+
+**New features**
+
+  - Add ``str.encode`` and ``str.decode`` to Series (#1706)
+  - Add `to_latex` method to DataFrame (#1735)
+  - Add convenient expanding window equivalents of all rolling_* ops (#1785)
+  - Add Options class to pandas.io.data for fetching options data from Yahoo!
+    Finance (#1748, #1739)
+  - Recognize and convert more boolean values in file parsing (Yes, No, TRUE,
+    FALSE, variants thereof) (#1691)
+
+**Improvements to existing features**
+
+  - Add ``flags`` option for ``re.compile`` in some Series.str methods (#1659)
+  - Parsing of UTC date strings in read_* functions (#1693)
+  - Handle generator input to Series (#1679)
+  - Add `na_action='ignore'` to Series.map to quietly propagate NAs (#1661)
+  - Add args/kwds options to Series.apply (#1829)
+  - Add inplace option to Series/DataFrame.reset_index (#1797)
+  - Add quoting option for DataFrame.to_csv (#1902)
+
+**API Changes**
+
+  - Deprecated ``day_of_year`` API removed from PeriodIndex, use ``dayofyear``
+    (#1723)
+  - Don't modify NumPy suppress printoption at import time
+  - The internal HDF5 data arrangement for DataFrames has been
+    transposed. Legacy files will still be readable by HDFStore (#1834, #1824)
+  - Legacy cruft removed: pandas.stats.misc.quantileTS
+  - Use ISO8601 format for Period repr: monthly, daily, and on down (#1776)
+
+**Bug fixes**
+
+  - Perform arithmetic column-by-column in mixed-type DataFrame to avoid type
+    upcasting issues. Caused downstream DataFrame.diff bug (#1896)
+  - Fix matplotlib auto-color assignment when no custom spectrum passed. Also
+    respect passed color keyword argument (#1711)
+  - Fix resampling logical error with closed='left' (#1726)
+  - Fix critical DatetimeIndex.union bugs (#1730, #1719, #1745, #1702)
+  - Fix critical DatetimeIndex.intersection bug with unanchored offsets (#1708)
+  - Fix MM-YYYY time series indexing case (#1672)
+  - Fix case where Categorical group key was not being passed into index in
+    GroupBy result (#1701)
+  - Handle Ellipsis in Series.__getitem__/__setitem__ (#1721)
+  - Fix some bugs with handling datetime64 scalars of other units in NumPy 1.6
+    and 1.7 (#1717)
+  - Fix performance issue in MultiIndex.format (#1746)
+  - Fixed GroupBy bugs interacting with DatetimeIndex asof / map methods (#1677)
+  - Handle factors with NAs in pandas.rpy (#1615)
+  - Fix statsmodels import in pandas.stats.var (#1734)
+  - Fix DataFrame repr/info summary with non-unique columns (#1700)
+  - Fix Series.iget_value for non-unique indexes (#1694)
+  - Don't lose tzinfo when passing DatetimeIndex as DataFrame column (#1682)
+  - Fix tz conversion with time zones that haven't had any DST transitions since
+    first date in the array (#1673)
+  - Fix field access with  UTC->local conversion on unsorted arrays (#1756)
+  - Fix isnull handling of array-like (list) inputs (#1755)
+  - Fix regression in handling of Series in Series constructor (#1671)
+  - Fix comparison of Int64Index with DatetimeIndex (#1681)
+  - Fix min_periods handling in new rolling_max/min at array start (#1695)
+  - Fix errors with how='median' and generic NumPy resampling in some cases
+    caused by SeriesBinGrouper (#1648, #1688)
+  - When grouping by level, exclude unobserved levels (#1697)
+  - Don't lose tzinfo in DatetimeIndex when shifting by different offset (#1683)
+  - Hack to support storing data with a zero-length axis in HDFStore (#1707)
+  - Fix DatetimeIndex tz-aware range generation issue (#1674)
+  - Fix method='time' interpolation with intraday data (#1698)
+  - Don't plot all-NA DataFrame columns as zeros (#1696)
+  - Fix bug in scatter_plot with by option (#1716)
+  - Fix performance problem in infer_freq with lots of non-unique stamps (#1686)
+  - Fix handling of PeriodIndex as argument to create MultiIndex (#1705)
+  - Fix re: unicode MultiIndex level names in Series/DataFrame repr (#1736)
+  - Handle PeriodIndex in to_datetime instance method (#1703)
+  - Support StaticTzInfo in DatetimeIndex infrastructure (#1692)
+  - Allow MultiIndex setops with length-0 other type indexes (#1727)
+  - Fix handling of DatetimeIndex in DataFrame.to_records (#1720)
+  - Fix handling of general objects in isnull on which bool(...) fails (#1749)
+  - Fix .ix indexing with MultiIndex ambiguity (#1678)
+  - Fix .ix setting logic error with non-unique MultiIndex (#1750)
+  - Basic indexing now works on MultiIndex with > 1000000 elements, regression
+    from earlier version of pandas (#1757)
+  - Handle non-float64 dtypes in fast DataFrame.corr/cov code paths (#1761)
+  - Fix DatetimeIndex.isin to function properly (#1763)
+  - Fix conversion of array of tz-aware datetime.datetime to DatetimeIndex with
+    right time zone (#1777)
+  - Fix DST issues with generating ancxhored date ranges (#1778)
+  - Fix issue calling sort on result of Series.unique (#1807)
+  - Fix numerical issue leading to square root of negative number in
+    rolling_std (#1840)
+  - Let Series.str.split accept no arguments (like str.split) (#1859)
+  - Allow user to have dateutil 2.1 installed on a Python 2 system (#1851)
+  - Catch ImportError less aggressively in pandas/__init__.py (#1845)
+  - Fix pip source installation bug when installing from GitHub (#1805)
+  - Fix error when window size > array size in rolling_apply (#1850)
+  - Fix pip source installation issues via SSH from GitHub
+  - Fix OLS.summary when column is a tuple (#1837)
+  - Fix bug in __doc__ patching when -OO passed to interpreter (#1792, #1741)
+  - Fix unicode console encoding issue in IPython notebook (#1782, #1768)
+  - Fix unicode formatting issue with Series.name (#1782)
+  - Fix bug in DataFrame.duplicated with datetime64 columns (#1833)
+  - Fix bug in Panel internals resulting in error when doing fillna after
+    truncate not changing size of panel (#1823)
+  - Prevent segfault due to MultiIndex not being supported in HDFStore table
+    format (#1848)
+  - Fix UnboundLocalError in Panel.__setitem__ and add better error (#1826)
+  - Fix to_csv issues with list of string entries. Isnull works on list of
+    strings now too (#1791)
+  - Fix Timestamp comparisons with datetime values outside the nanosecond range
+    (1677-2262)
+  - Revert to prior behavior of normalize_date with datetime.date objects
+    (return datetime)
+  - Fix broken interaction between np.nansum and Series.any/all
+  - Fix bug with multiple column date parsers (#1866)
+  - DatetimeIndex.union(Int64Index) was broken
+  - Make plot x vs y interface consistent with integer indexing (#1842)
+  - set_index inplace modified data even if unique check fails (#1831)
+  - Only use Q-OCT/NOV/DEC in quarterly frequency inference (#1789)
+  - Upcast to dtype=object when unstacking boolean DataFrame (#1820)
+  - Fix float64/float32 merging bug (#1849)
+  - Fixes to Period.start_time for non-daily frequencies (#1857)
+  - Fix failure when converter used on index_col in read_csv (#1835)
+  - Implement PeriodIndex.append so that pandas.concat works correctly (#1815)
+  - Avoid Cython out-of-bounds access causing segfault sometimes in pad_2d,
+    backfill_2d
+  - Fix resampling error with intraday times and anchored target time (like
+    AS-DEC) (#1772)
+  - Fix .ix indexing bugs with mixed-integer indexes (#1799)
+  - Respect passed color keyword argument in Series.plot (#1890)
+  - Fix rolling_min/max when the window is larger than the size of the input
+    array. Check other malformed inputs (#1899, #1897)
+  - Rolling variance / standard deviation with only a single observation in
+    window (#1884)
+  - Fix unicode sheet name failure in to_excel (#1828)
+  - Override DatetimeIndex.min/max to return Timestamp objects (#1895)
+  - Fix column name formatting issue in length-truncated column (#1906)
+
 pandas 0.8.1
 ============
 

doc/make.py

@@ -150,16 +150,18 @@ def sendmail(step=None, err_msg=None):
     finally:
         server.close()
 
-def _get_dir():
+def _get_dir(subdir=None):
     import getpass
     USERNAME = getpass.getuser()
     if sys.platform == 'darwin':
         HOME = '/Users/%s' % USERNAME
     else:
         HOME = '/home/%s' % USERNAME
 
-    tmp_dir = '%s/tmp' % HOME
-    return tmp_dir
+    if subdir is None:
+        subdir = '/code/scripts/config'
+    conf_dir = '%s/%s' % (HOME, subdir)
+    return conf_dir
 
 def _get_credentials():
     tmp_dir = _get_dir()
@@ -177,7 +179,7 @@ def _get_credentials():
 
 def _get_config():
     tmp_dir = _get_dir()
-    with open('%s/config' % tmp_dir, 'r') as fh:
+    with open('%s/addresses' % tmp_dir, 'r') as fh:
         from_name, to_name = fh.read().split(',')
     return from_name, to_name
 

doc/source/api.rst

@@ -81,6 +81,27 @@ Standard moving window functions
    rolling_apply
    rolling_quantile
 
+Standard expanding window functions
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. currentmodule:: pandas.stats.moments
+
+.. autosummary::
+   :toctree: generated/
+
+   expanding_count
+   expanding_sum
+   expanding_mean
+   expanding_median
+   expanding_var
+   expanding_std
+   expanding_corr
+   expanding_cov
+   expanding_skew
+   expanding_kurt
+   expanding_apply
+   expanding_quantile
+
 Exponentially-weighted moving window functions
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 

doc/source/basics.rst

@@ -876,6 +876,14 @@ Methods like ``replace`` and ``findall`` take regular expressions, too:
    s3
    s3.str.replace('^.a|dog', 'XX-XX ', case=False)
 
+Methods like ``contains``, ``startswith``, and ``endswith`` takes an extra
+``na`` arguement so missing values can be considered True or False:
+
+.. ipython:: python
+
+   s4 = Series(['A', 'B', 'C', 'Aaba', 'Baca', np.nan, 'CABA', 'dog', 'cat'])
+   s4.str.contains('A', na=False)
+
 .. csv-table::
     :header: "Method", "Description"
     :widths: 20, 80

doc/source/computation.rst

@@ -192,7 +192,7 @@ accept the following arguments:
   - ``window``: size of moving window
   - ``min_periods``: threshold of non-null data points to require (otherwise
     result is NA)
-  - ``freq``: optionally specify a :ref: `frequency string <timeseries.alias>`
+  - ``freq``: optionally specify a :ref:`frequency string <timeseries.alias>`
     or :ref:`DateOffset <timeseries.offsets>` to pre-conform the data to.
     Note that prior to pandas v0.8.0, a keyword argument ``time_rule`` was used
     instead of ``freq`` that referred to the legacy time rule constants
@@ -288,6 +288,79 @@ columns using ``ix`` indexing:
    @savefig rolling_corr_pairwise_ex.png width=4.5in
    correls.ix[:, 'A', 'C'].plot()
 
+Expanding window moment functions
+---------------------------------
+A common alternative to rolling statistics is to use an *expanding* window, 
+which yields the value of the statistic with all the data available up to that 
+point in time. As these calculations are a special case of rolling statistics, 
+they are implemented in pandas such that the following two calls are equivalent:
+
+.. ipython:: python
+
+   rolling_mean(df, window=len(df), min_periods=1)[:5]
+
+   expanding_mean(df)[:5]
+
+Like the ``rolling_`` functions, the following methods are included in the 
+``pandas`` namespace or can be located in ``pandas.stats.moments``.
+
+.. csv-table::
+    :header: "Function", "Description"
+    :widths: 20, 80
+
+    ``expanding_count``, Number of non-null observations
+    ``expanding_sum``, Sum of values
+    ``expanding_mean``, Mean of values
+    ``expanding_median``, Arithmetic median of values
+    ``expanding_min``, Minimum
+    ``expanding_max``, Maximum
+    ``expanding_std``, Unbiased standard deviation
+    ``expanding_var``, Unbiased variance
+    ``expanding_skew``, Unbiased skewness (3rd moment)
+    ``expanding_kurt``, Unbiased kurtosis (4th moment)
+    ``expanding_quantile``, Sample quantile (value at %)
+    ``expanding_apply``, Generic apply
+    ``expanding_cov``, Unbiased covariance (binary)
+    ``expanding_corr``, Correlation (binary)
+    ``expanding_corr_pairwise``, Pairwise correlation of DataFrame columns
+
+Aside from not having a ``window`` parameter, these functions have the same 
+interfaces as their ``rolling_`` counterpart. Like above, the parameters they 
+all accept are:
+
+  - ``min_periods``: threshold of non-null data points to require. Defaults to 
+    minimum needed to compute statistic. No ``NaNs`` will be output once 
+    ``min_periods`` non-null data points have been seen.
+  - ``freq``: optionally specify a :ref:`frequency string <timeseries.alias>`
+    or :ref:`DateOffset <timeseries.offsets>` to pre-conform the data to.
+    Note that prior to pandas v0.8.0, a keyword argument ``time_rule`` was used
+    instead of ``freq`` that referred to the legacy time rule constants
+
+.. note::
+
+   The output of the ``rolling_`` and ``expanding_`` functions do not return a 
+   ``NaN`` if there are at least ``min_periods`` non-null values in the current 
+   window. This differs from ``cumsum``, ``cumprod``, ``cummax``, and 
+   ``cummin``, which return ``NaN`` in the output wherever a ``NaN`` is 
+   encountered in the input.
+
+An expanding window statistic will be more stable (and less responsive) than 
+its rolling window counterpart as the increasing window size decreases the 
+relative impact of an individual data point. As an example, here is the 
+``expanding_mean`` output for the previous time series dataset:
+
+.. ipython:: python
+   :suppress:
+
+   plt.close('all')
+
+.. ipython:: python
+
+   ts.plot(style='k--')
+
+   @savefig expanding_mean_frame.png width=4.5in
+   expanding_mean(ts).plot(style='k')
+
 Exponentially weighted moment functions
 ---------------------------------------
 

doc/source/conf.py

@@ -233,16 +233,17 @@
 
 
 # Example configuration for intersphinx: refer to the Python standard library.
-# intersphinx_mapping = {'http://docs.scipy.org/': None}
+intersphinx_mapping = {
+        'statsmodels' : ('http://statsmodels.sourceforge.net/devel/', None),
+        'python': ('http://docs.python.org/', None)
+                      }
 import glob
 autosummary_generate = glob.glob("*.rst")
 
 # extlinks alias
 extlinks = {'issue': ('https://github.com/pydata/pandas/issues/%s',
-                      'issue ')}
-
-extlinks = {'pull request': ('https://github.com/pydata/pandas/pulls/%s',
-                      'pull request ')}
-                      
-extlinks = {'wiki': ('https://github.com/pydata/pandas/pulls/%s',
+                      'issue '),
+            'pull request': ('https://github.com/pydata/pandas/pulls/%s',
+                      'pull request '),
+            'wiki': ('https://github.com/pydata/pandas/pulls/%s',
                       'wiki ')}

doc/source/dsintro.rst

@@ -140,15 +140,20 @@ label:
     'e' in s
     'f' in s
 
-If a label is not contained, an exception
+If a label is not contained, an exception is raised:
 
 .. code-block:: python
 
     >>> s['f']
     KeyError: 'f'
 
-    >>> s.get('f')
-    nan
+Using the ``get`` method, a missing label will return None or specified default:
+
+.. ipython:: python
+
+   s.get('f')
+
+   s.get('f', np.nan)
 
 Vectorized operations and label alignment with Series
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

doc/source/index.rst

@@ -89,7 +89,7 @@ Some other notes
    on one feature for your application you may be able to create a faster
    specialized tool.
 
- - pandas will soon become a dependency of `statsmodels
+ - pandas is a dependency of `statsmodels
    <http://statsmodels.sourceforge.net>`__, making it an important part of the
    statistical computing ecosystem in Python.
 

doc/source/install.rst

@@ -48,7 +48,7 @@ ___________
     Windows, all, stable, :ref:`all-platforms`,  ``pip install pandas``
     Mac, all, stable, :ref:`all-platforms`,  ``pip install pandas``
     Linux, Debian, stable, `official Debian repository <http://packages.debian.org/search?keywords=pandas&searchon=names&suite=all&section=all>`_ , ``sudo apt-get install python-pandas``
-    Linux, Debian, unstable (latest packages), `NeuroDebian <http://neuro.debian.net/index.html#how-to-use-this-repository>`_ , ``sudo apt-get install python-pandas``
+    Linux, Debian & Ubuntu, unstable (latest packages), `NeuroDebian <http://neuro.debian.net/index.html#how-to-use-this-repository>`_ , ``sudo apt-get install python-pandas``
     Linux, Ubuntu, stable, `official Ubuntu repository <http://packages.ubuntu.com/search?keywords=pandas&searchon=names&suite=all&section=all>`_ , ``sudo apt-get install python-pandas``
     Linux, Ubuntu, unstable (daily builds), `PythonXY PPA  <https://code.launchpad.net/~pythonxy/+archive/pythonxy-devel>`_; activate by: ``sudo add-apt-repository ppa:pythonxy/pythonxy-devel && sudo apt-get update``, ``sudo apt-get install python-pandas``
 	Linux, OpenSuse & Fedora, stable, `OpenSuse Repository  <http://software.opensuse.org/package/python-pandas?search_term=pandas>`_ , ``zypper in  python-pandas``
@@ -74,7 +74,7 @@ Optional dependencies
   * `SciPy <http://www.scipy.org>`__: miscellaneous statistical functions
   * `PyTables <http://www.pytables.org>`__: necessary for HDF5-based storage
   * `matplotlib <http://matplotlib.sourceforge.net/>`__: for plotting
-  * `scikits.statsmodels <http://statsmodels.sourceforge.net/>`__
+  * `statsmodels <http://statsmodels.sourceforge.net/>`__: 0.4.0 or higher
      * Needed for parts of :mod:`pandas.stats`
   * `pytz <http://pytz.sourceforge.net/>`__
      * Needed for time zone support with ``date_range``