Merge branch 'master' of https://github.com/pydata/pandas

Author: aflaxman

RELEASE.rst

@@ -36,6 +36,7 @@ pandas 0.9.0
     Finance (#1748, #1739)
   - Recognize and convert more boolean values in file parsing (Yes, No, TRUE,
     FALSE, variants thereof) (#1691, #1295)
+  - Add Panel.update method, analogous to DataFrame.update (#1999, #1988)
 
 **Improvements to existing features**
 
@@ -63,6 +64,8 @@ pandas 0.9.0
 
 **API Changes**
 
+  - Change default header names in read_* functions to more Pythonic X0, X1,
+    etc. instead of X.1, X.2. (#2000)
   - Deprecated ``day_of_year`` API removed from PeriodIndex, use ``dayofyear``
     (#1723)
   - Don't modify NumPy suppress printoption at import time
@@ -243,6 +246,7 @@ pandas 0.9.0
   - Fix reset_index bug if both drop and level are specified (#1957)
   - Work around unsafe NumPy object->int casting with Cython function (#1987)
   - Fix datetime64 formatting bug in DataFrame.to_csv (#1993)
+  - Default start date in pandas.io.data to 1/1/2000 as the docs say (#2011)
 
 
 pandas 0.8.1

doc/source/computation.rst

@@ -397,8 +397,8 @@ available:
     :widths: 20, 80
 
     ``ewma``, EW moving average
-    ``ewvar``, EW moving variance
-    ``ewstd``, EW moving standard deviation
+    ``ewmvar``, EW moving variance
+    ``ewmstd``, EW moving standard deviation
     ``ewmcorr``, EW moving correlation
     ``ewmcov``, EW moving covariance
 

doc/source/v0.9.0.txt

@@ -1,7 +1,7 @@
 .. _whatsnew_0900:
 
-v0.9.0 (September 25, 2012)
----------------------------
+v0.9.0 (October 2, 2012)
+------------------------
 
 This is a major release from 0.8.1 and includes several new features and
 enhancements along with a large number of bug fixes. New features include
@@ -30,31 +30,62 @@ New features
 API changes
 ~~~~~~~~~~~
 
-  - Creating a Series from another Series, passing an index, will cause
-    reindexing to happen inside rather than treating the Series like an
-    ndarray. Technically improper usages like Series(df[col1], index=df[col2])
-    that worked before "by accident" (this was never intended) will lead to all
-    NA Series in some cases.
-  - Deprecated ``day_of_year`` API removed from PeriodIndex, use ``dayofyear``
-    (GH1723_)
-  - Don't modify NumPy suppress printoption to True at import time
-  - The internal HDF5 data arrangement for DataFrames has been transposed.
-    Legacy files will still be readable by HDFStore (GH1834_, GH1824_)
-  - Legacy cruft removed: pandas.stats.misc.quantileTS
-  - Use ISO8601 format for Period repr: monthly, daily, and on down (GH1776_)
-  - Empty DataFrame columns are now created as object dtype. This will prevent
-    a class of TypeErrors that was occurring in code where the dtype of a
-    column would depend on the presence of data or not (e.g. a SQL query having
-    results) (GH1783_)
-  - Setting parts of DataFrame/Panel using ix now aligns input Series/DataFrame
-    (GH1630_)
-  - ``first`` and ``last`` methods in ``GroupBy`` no longer drop non-numeric
-    columns (GH1809_)
-  - Resolved inconsistencies in specifying custom NA values in text parser.
-    ``na_values`` of type dict no longer override default NAs unless
-    ``keep_default_na`` is set to false explicitly (GH1657_)
-  - ``DataFrame.dot`` will not do data alignment, and also work with Series
-    (GH1915_)
+  - The default column names when ``header=None`` and no columns names passed to
+    functions like ``read_csv`` has changed to be more Pythonic and amenable to
+    attribute access:
+
+.. ipython:: python
+
+   from StringIO import StringIO
+
+   data = '0,0,1\n1,1,0\n0,1,0'
+   df = read_csv(StringIO(data), header=None)
+   df
+
+
+- Creating a Series from another Series, passing an index, will cause reindexing
+  to happen inside rather than treating the Series like an ndarray. Technically
+  improper usages like ``Series(df[col1], index=df[col2])11 that worked before
+  "by accident" (this was never intended) will lead to all NA Series in some
+  cases. To be perfectly clear:
+
+.. ipython:: python
+
+   s1 = Series([1, 2, 3])
+   s1
+
+   s2 = Series(s1, index=['foo', 'bar', 'baz'])
+   s2
+
+- Deprecated ``day_of_year`` API removed from PeriodIndex, use ``dayofyear``
+  (GH1723_)
+
+- Don't modify NumPy suppress printoption to True at import time
+
+- The internal HDF5 data arrangement for DataFrames has been transposed.  Legacy
+  files will still be readable by HDFStore (GH1834_, GH1824_)
+
+- Legacy cruft removed: pandas.stats.misc.quantileTS
+
+- Use ISO8601 format for Period repr: monthly, daily, and on down (GH1776_)
+
+- Empty DataFrame columns are now created as object dtype. This will prevent a
+  class of TypeErrors that was occurring in code where the dtype of a column
+  would depend on the presence of data or not (e.g. a SQL query having results)
+  (GH1783_)
+
+- Setting parts of DataFrame/Panel using ix now aligns input Series/DataFrame
+  (GH1630_)
+
+- ``first`` and ``last`` methods in ``GroupBy`` no longer drop non-numeric
+  columns (GH1809_)
+
+- Resolved inconsistencies in specifying custom NA values in text parser.
+  ``na_values`` of type dict no longer override default NAs unless
+  ``keep_default_na`` is set to false explicitly (GH1657_)
+
+- ``DataFrame.dot`` will not do data alignment, and also work with Series
+  (GH1915_)
 
 
 See the `full release notes

pandas/core/frame.py

@@ -2461,7 +2461,8 @@ def set_index(self, keys, drop=True, append=False, inplace=False,
         frame.index = index
         return frame
 
-    def reset_index(self, level=None, drop=False, inplace=False):
+    def reset_index(self, level=None, drop=False, inplace=False, col_level=0,
+                    col_fill=''):
         """
         For DataFrame with multi-level index, return new DataFrame with
         labeling information in the columns under the index names, defaulting
@@ -2479,6 +2480,13 @@ def reset_index(self, level=None, drop=False, inplace=False):
             the index to the default integer index.
         inplace : boolean, default False
             Modify the DataFrame in place (do not create a new object)
+        col_level : int or str, default 0
+            If the columns have multiple levels, determines which level the
+            labels are inserted into. By default it is inserted into the first
+            level.
+        col_fill : object, default ''
+            If the columns have multiple levels, determines how the other levels
+            are named. If None then the index name is repeated.
 
         Returns
         -------
@@ -2507,11 +2515,22 @@ def _maybe_cast(values):
                 names = self.index.names
                 zipped = zip(self.index.levels, self.index.labels)
 
+                multi_col = isinstance(self.columns, MultiIndex)
                 for i, (lev, lab) in reversed(list(enumerate(zipped))):
                     col_name = names[i]
                     if col_name is None:
                         col_name = 'level_%d' % i
 
+                    if multi_col:
+                        if col_fill is None:
+                            col_name = tuple([col_name] *
+                                             self.columns.nlevels)
+                        else:
+                            name_lst = [col_fill] * self.columns.nlevels
+                            lev_num = self.columns._get_level_number(col_level)
+                            name_lst[lev_num] = col_name
+                            col_name = tuple(name_lst)
+
                     # to ndarray and maybe infer different dtype
                     level_values = _maybe_cast(lev.values)
                     if level is None or i in level:
@@ -2521,6 +2540,14 @@ def _maybe_cast(values):
             name = self.index.name
             if name is None or name == 'index':
                 name = 'index' if 'index' not in self else 'level_0'
+            if isinstance(self.columns, MultiIndex):
+                if col_fill is None:
+                    name = tuple([name] * self.columns.nlevels)
+                else:
+                    name_lst = [col_fill] * self.columns.nlevels
+                    lev_num = self.columns._get_level_number(col_level)
+                    name_lst[lev_num] = name
+                    name = tuple(name_lst)
             new_obj.insert(0, name, _maybe_cast(self.index.values))
 
         new_obj.index = new_index
@@ -3368,7 +3395,7 @@ def update(self, other, join='left', overwrite=True, filter_func=None,
 
         Parameters
         ----------
-        other : DataFrame
+        other : DataFrame, or object coercible into a DataFrame
         join : {'left', 'right', 'outer', 'inner'}, default 'left'
         overwrite : boolean, default True
             If True then overwrite values for common keys in the calling frame
@@ -3382,7 +3409,11 @@ def update(self, other, join='left', overwrite=True, filter_func=None,
         if join != 'left':
             raise NotImplementedError
 
+        if not isinstance(other, DataFrame):
+            other = DataFrame(other)
+
         other = other.reindex_like(self)
+
         for col in self.columns:
             this = self[col].values
             that = other[col].values

pandas/core/groupby.py

@@ -303,7 +303,7 @@ def mean(self):
 
     def median(self):
         """
-        Compute mean of groups, excluding missing values
+        Compute median of groups, excluding missing values
 
         For multiple groupings, the result index will be a MultiIndex
         """

pandas/core/panel.py

@@ -1318,6 +1318,36 @@ def join(self, other, how='left', lsuffix='', rsuffix=''):
             return concat([self] + list(other), axis=0, join=how,
                           join_axes=join_axes, verify_integrity=True)
 
+    def update(self, other, join='left', overwrite=True, filter_func=None,
+                     raise_conflict=False):
+        """
+        Modify Panel in place using non-NA values from passed
+        Panel, or object coercible to Panel. Aligns on items
+
+        Parameters
+        ----------
+        other : Panel, or object coercible to Panel
+        join : How to join individual DataFrames
+            {'left', 'right', 'outer', 'inner'}, default 'left'
+        overwrite : boolean, default True
+            If True then overwrite values for common keys in the calling panel
+        filter_func : callable(1d-array) -> 1d-array<boolean>, default None
+            Can choose to replace values other than NA. Return True for values
+            that should be updated
+        raise_conflict : bool
+            If True, will raise an error if a DataFrame and other both
+            contain data in the same place.
+        """
+
+        if not isinstance(other, Panel):
+            other = Panel(other)
+
+        other = other.reindex(items=self.items)
+
+        for frame in self.items:
+            self[frame].update(other[frame], join, overwrite, filter_func,
+                               raise_conflict)
+
     def _get_join_index(self, other, how):
         if how == 'left':
             join_major, join_minor = self.major_axis, self.minor_axis

pandas/io/data.py

@@ -67,7 +67,7 @@ def _sanitize_dates(start, end):
     start = to_datetime(start)
     end = to_datetime(end)
     if start is None:
-        start = dt.datetime.today() - dt.timedelta(365)
+        start = dt.datetime(2010, 1, 1)
     if end is None:
         end = dt.datetime.today()
     return start, end
@@ -178,7 +178,8 @@ def get_data_fred(name=None, start=dt.datetime(2010, 1, 1),
 
     url = fred_URL + '%s' % name + \
       '/downloaddata/%s' % name + '.csv'
-    data = read_csv(urllib.urlopen(url), index_col=0, parse_dates=True)
+    data = read_csv(urllib.urlopen(url), index_col=0, parse_dates=True, header=None,
+                    skiprows=1, names=["DATE", name])
     return data.truncate(start, end)
 
 

pandas/io/parsers.py

@@ -43,15 +43,17 @@ class DateConversionError(Exception):
     If None defaults to Excel dialect. Ignored if sep longer than 1 char
     See csv.Dialect documentation for more details
 header : int, default 0
-    Row to use for the column labels of the parsed DataFrame
+    Row to use for the column labels of the parsed DataFrame. Specify None if
+    there is no header row.
 skiprows : list-like or integer
     Row numbers to skip (0-indexed) or number of rows to skip (int)
     at the start of the file
 index_col : int or sequence, default None
     Column to use as the row labels of the DataFrame. If a sequence is
     given, a MultiIndex is used.
 names : array-like
-    List of column names
+    List of column names to use. If passed, header will be implicitly set to
+    None.
 na_values : list-like or dict, default None
     Additional strings to recognize as NA/NaN. If dict passed, specific
     per-column NA values
@@ -613,7 +615,7 @@ def _infer_columns(self):
 
             ncols = len(line)
             if not names:
-                columns = ['X.%d' % (i + 1) for i in range(ncols)]
+                columns = ['X%d' % i for i in range(ncols)]
             else:
                 columns = names
 
@@ -747,7 +749,7 @@ def _explicit_index_names(self, columns):
             else:
                 index_name = columns[self.index_col]
 
-            if index_name is not None and 'Unnamed' in index_name:
+            if index_name is not None and 'Unnamed' in str(index_name):
                 index_name = None
 
         elif self.index_col is not None:
@@ -833,6 +835,9 @@ def get_chunk(self, rows=None):
         alldata = self._rows_to_cols(content)
         data = self._exclude_implicit_index(alldata)
 
+        if self.parse_dates is not None:
+            data, columns = self._process_date_conversion(data)
+
         # apply converters
         for col, f in self.converters.iteritems():
             if isinstance(col, int) and col not in self.orig_columns:
@@ -841,9 +846,6 @@ def get_chunk(self, rows=None):
 
         data = _convert_to_ndarrays(data, self.na_values, self.verbose)
 
-        if self.parse_dates is not None:
-            data, columns = self._process_date_conversion(data)
-
         if self.index_col is None:
             numrows = len(content)
             index = Index(np.arange(numrows))
@@ -1160,19 +1162,9 @@ def _convert_types(values, na_values):
 
     return result, na_count
 
-def _get_col_names(colspec, columns):
-    colset = set(columns)
-    colnames = []
-    for c in colspec:
-        if c in colset:
-            colnames.append(str(c))
-        elif isinstance(c, int):
-            colnames.append(str(columns[c]))
-    return colnames
-
 def _try_convert_dates(parser, colspec, data_dict, columns):
     colspec = _get_col_names(colspec, columns)
-    new_name = '_'.join(colspec)
+    new_name = '_'.join([str(x) for x in colspec])
 
     to_parse = [data_dict[c] for c in colspec if c in data_dict]
     try:
@@ -1181,6 +1173,17 @@ def _try_convert_dates(parser, colspec, data_dict, columns):
         new_col = parser(_concat_date_cols(to_parse))
     return new_name, new_col, colspec
 
+def _get_col_names(colspec, columns):
+    colset = set(columns)
+    colnames = []
+    for c in colspec:
+        if c in colset:
+            colnames.append(c)
+        elif isinstance(c, int):
+            colnames.append(columns[c])
+    return colnames
+
+
 def _concat_date_cols(date_cols):
     if len(date_cols) == 1:
         return np.array([str(x) for x in date_cols[0]], dtype=object)