v0.13.0.txt

.. _whatsnew_0130:

v0.13.0 (August ??, 2013)
-------------------------

This is a major release from 0.12.0 and includes several new features and
enhancements along with a large number of bug fixes.

.. warning::

   In 0.13.0 ``Series`` has internaly been refactored to no longer sub-class ``ndarray``
   but instead subclass ``NDFrame``, similarly to the rest of the pandas containers. This should be
   a transparent change with only very limited API implications. See :ref:`Internal Refactoring<whatsnew_0130.refactoring>`

API changes
~~~~~~~~~~~

  - ``read_excel`` now supports an integer in its ``sheetname`` argument giving
    the index of the sheet to read in (:issue:`4301`).
  - Text parser now treats anything that reads like inf ("inf", "Inf", "-Inf",
    "iNf", etc.) as infinity. (:issue:`4220`, :issue:`4219`), affecting
    ``read_table``, ``read_csv``, etc.
  - ``pandas`` now is Python 2/3 compatible without the need for 2to3 thanks to
    @jtratner. As a result, pandas now uses iterators more extensively. This
    also led to the introduction of substantive parts of the Benjamin
    Peterson's ``six`` library into compat. (:issue:`4384`, :issue:`4375`,
    :issue:`4372`)
  - ``pandas.util.compat`` and ``pandas.util.py3compat`` have been merged into
    ``pandas.compat``. ``pandas.compat`` now includes many functions allowing
    2/3 compatibility. It contains both list and iterator versions of range,
    filter, map and zip, plus other necessary elements for Python 3
    compatibility. ``lmap``, ``lzip``, ``lrange`` and ``lfilter`` all produce
    lists instead of iterators, for compatibility with ``numpy``, subscripting
    and ``pandas`` constructors.(:issue:`4384`, :issue:`4375`, :issue:`4372`)
  - deprecated ``iterkv``, which will be removed in a future release (was just
    an alias of iteritems used to get around ``2to3``'s changes).
    (:issue:`4384`, :issue:`4375`, :issue:`4372`)
  - ``Series.get`` with negative indexers now returns the same as ``[]`` (:issue:`4390`)
  - ``HDFStore``

    - Significant table writing performance improvements
    - handle a passed ``Series`` in table format (:issue:`4330`)
    - added an ``is_open`` property to indicate if the underlying file handle is_open;
      a closed store will now report 'CLOSED' when viewing the store (rather than raising an error)
      (:issue:`4409`)
    - a close of a ``HDFStore`` now will close that instance of the ``HDFStore``
      but will only close the actual file if the ref count (by ``PyTables``) w.r.t. all of the open handles
      are 0. Essentially you have a local instance of ``HDFStore`` referenced by a variable. Once you
      close it, it will report closed. Other references (to the same file) will continue to operate
      until they themselves are closed. Performing an action on a closed file will raise
      ``ClosedFileError``

      .. ipython:: python

         path = 'test.h5'
         df = DataFrame(randn(10,2))
         store1 = HDFStore(path)
         store2 = HDFStore(path)
         store1.append('df',df)
         store2.append('df2',df)

         store1
         store2
         store1.close()
         store2
         store2.close()
         store2

      .. ipython:: python
         :suppress:

         import os
         os.remove(path)

    - removed the ``_quiet`` attribute, replace by a ``DuplicateWarning`` if retrieving
      duplicate rows from a table (:issue:`4367`)
    - removed the ``warn`` argument from ``open``. Instead a ``PossibleDataLossError`` exception will
      be raised if you try to use ``mode='w'`` with an OPEN file handle (:issue:`4367`)
    - allow a passed locations array or mask as a ``where`` condition (:issue:`4467`).
      See :ref:`here<io.hdf5-where_mask>` for an example.

    - the ``format`` keyword now replaces the ``table`` keyword; allowed values are ``fixed(f)`` or ``table(t)``
      the same defaults as prior < 0.13.0 remain, e.g. ``put`` implies 'fixed` or 'f' (Fixed) format
      and ``append`` imples 'table' or 't' (Table) format

      .. ipython:: python

         path = 'test.h5'
         df = DataFrame(randn(10,2))
         df.to_hdf(path,'df_table',format='table')
         df.to_hdf(path,'df_table2',append=True)
         df.to_hdf(path,'df_fixed')
         with get_store(path) as store:
            print store

      .. ipython:: python
         :suppress:

         import os
         os.remove(path)
    - add the keyword ``dropna=True`` to ``append`` to change whether ALL nan rows are not written
      to the store (default is ``True``, ALL nan rows are NOT written), also settable
      via the option ``io.hdf.dropna_table`` (:issue:`4625`)

  - Changes to how ``Index`` and ``MultiIndex`` handle metadata (``levels``,
    ``labels``, and ``names``) (:issue:`4039`):

    ..code-block ::

        # previously, you would have set levels or labels directly
        index.levels = [[1, 2, 3, 4], [1, 2, 4, 4]]

        # now, you use the set_levels or set_labels methods
        index = index.set_levels([[1, 2, 3, 4], [1, 2, 4, 4]])

        # similarly, for names, you can rename the object
        # but setting names is not deprecated.
        index = index.set_names(["bob", "cranberry"])

        # and all methods take an inplace kwarg
        index.set_names(["bob", "cranberry"], inplace=True)

  - Infer and downcast dtype if ``downcast='infer'`` is passed to ``fillna/ffill/bfill`` (:issue:`4604`)
  - ``__nonzero__`` for all NDFrame objects, will now raise a ``ValueError``, this reverts back to (:issue:`1073`, :issue:`4633`)
    behavior.

    This prevent behaviors like (which will now all raise ``ValueError``)

    ..code-block ::

        if df:
           ....

        df1 and df2
        s1 and s2

Indexing API Changes
~~~~~~~~~~~~~~~~~~~~

    Prior to 0.13, it was impossible to use an indexer (``.loc/.iloc/.ix``) to set a value that
    was not contained in the index of a particular axis. (:issue:`2578`). See more at :ref:`here<indexing.basics.partial_setting>`

    In the ``Series`` case this is effectively an appending operation

    .. ipython:: python

       s = Series([1,2,3])
       s
       s[5] = 5.
       s

    .. ipython:: python

       dfi = DataFrame(np.arange(6).reshape(3,2),
                       columns=['A','B'])
       dfi

    This would previously ``KeyError``

    .. ipython:: python

       dfi.loc[:,'C'] = dfi.loc[:,'A']
       dfi

    This is like an ``append`` operation.

    .. ipython:: python

       dfi.loc[3] = 5
       dfi

    A Panel setting operation on an arbitrary axis aligns the input to the Panel

    .. ipython:: python

       p = pd.Panel(np.arange(16).reshape(2,4,2),
                   items=['Item1','Item2'],
                   major_axis=pd.date_range('2001/1/12',periods=4),
                   minor_axis=['A','B'],dtype='float64')
       p
       p.loc[:,:,'C'] = Series([30,32],index=p.items)
       p
       p.loc[:,:,'C']

Enhancements
~~~~~~~~~~~~

  - ``read_html`` now raises a ``URLError`` instead of catching and raising a
    ``ValueError`` (:issue:`4303`, :issue:`4305`)
  - Added a test for ``read_clipboard()`` and ``to_clipboard()`` (:issue:`4282`)
  - Clipboard functionality now works with PySide (:issue:`4282`)
  - Added a more informative error message when plot arguments contain
    overlapping color and style arguments (:issue:`4402`)

  - NaN handing in get_dummies (:issue:`4446`) with `dummy_na`

    .. ipython:: python
       # previously, nan was erroneously counted as 2 here
       # now it is not counted at all
       get_dummies([1, 2, np.nan])

       # unless requested
       get_dummies([1, 2, np.nan], dummy_na=True)


  - ``timedelta64[ns]`` operations

    - A Series of dtype ``timedelta64[ns]`` can now be divided by another
      ``timedelta64[ns]`` object to yield a ``float64`` dtyped Series. This
      is frequency conversion. See :ref:`here<timeseries.timedeltas_convert>` for the docs.

      .. ipython:: python

         from datetime import timedelta
         td = Series(date_range('20130101',periods=4))-Series(date_range('20121201',periods=4))
         td[2] += np.timedelta64(timedelta(minutes=5,seconds=3))
         td[3] = np.nan
         td

         # to days
         td / np.timedelta64(1,'D')

         # to seconds
         td / np.timedelta64(1,'s')

    - Dividing or multiplying a ``timedelta64[ns]`` Series by an integer or integer Series

      .. ipython:: python

         td * -1
         td * Series([1,2,3,4])

    - Absolute ``DateOffset`` objects can act equivalenty to ``timedeltas``

      .. ipython:: python

         from pandas import offsets
         td + offsets.Minute(5) + offsets.Milli(5)

    - ``plot(kind='kde')`` now accepts the optional parameters ``bw_method`` and
      ``ind``, passed to scipy.stats.gaussian_kde() (for scipy >= 0.11.0) to set
      the bandwidth, and to gkde.evaluate() to specify the indicies at which it
      is evaluated, respecttively. See scipy docs.

.. _whatsnew_0130.refactoring:

Internal Refactoring
~~~~~~~~~~~~~~~~~~~~

In 0.13.0 there is a major refactor primarily to subclass ``Series`` from ``NDFrame``,
which is the base class currently for ``DataFrame`` and ``Panel``, to unify methods
and behaviors. Series formerly subclassed directly from ``ndarray``. (:issue:`4080`, :issue:`3862`, :issue:`816`)

.. warning::

   There are two potential incompatibilities from < 0.13.0

   - Using certain numpy functions would previously return a ``Series`` if passed a ``Series``
     as an argument. This seems only to affect ``np.ones_like``, ``np.empty_like``,
     ``np.diff`` and ``np.where``. These now return ``ndarrays``.

     .. ipython:: python

        s = Series([1,2,3,4])

     Numpy Usage

     .. ipython:: python

        np.ones_like(s)
        np.diff(s)
        np.where(s>1,s,np.nan)

     Pandonic Usage

     .. ipython:: python

        Series(1,index=s.index)
        s.diff()
        s.where(s>1)

   - Passing a ``Series`` directly to a cython function expecting an ``ndarray`` type will no
     long work directly, you must pass ``Series.values``, See :ref:`Enhancing Performance<enhancingperf.ndarray>`

   - ``Series(0.5)`` would previously return the scalar ``0.5``, instead this will return a 1-element ``Series``

- Refactor of series.py/frame.py/panel.py to move common code to generic.py

  - added ``_setup_axes`` to created generic NDFrame structures
  - moved methods

    - ``from_axes,_wrap_array,axes,ix,loc,iloc,shape,empty,swapaxes,transpose,pop``
    - ``__iter__,keys,__contains__,__len__,__neg__,__invert__``
    - ``convert_objects,as_blocks,as_matrix,values``
    - ``__getstate__,__setstate__`` (compat remains in frame/panel)
    - ``__getattr__,__setattr__``
    - ``_indexed_same,reindex_like,align,where,mask``
    - ``fillna,replace`` (``Series`` replace is now consistent with ``DataFrame``)
    - ``filter`` (also added axis argument to selectively filter on a different axis)
    - ``reindex,reindex_axis,take``
    - ``truncate`` (moved to become part of ``NDFrame``)

- These are API changes which make ``Panel`` more consistent with ``DataFrame``

  - ``swapaxes`` on a ``Panel`` with the same axes specified now return a copy
  - support attribute access for setting
  - filter supports same api as original ``DataFrame`` filter

- Reindex called with no arguments will now return a copy of the input object

- Series now inherits from ``NDFrame`` rather than directly from ``ndarray``.
  There are several minor changes that affect the API.

  - numpy functions that do not support the array interface will now
    return ``ndarrays`` rather than series, e.g. ``np.diff`` and ``np.ones_like``
  - ``Series(0.5)`` would previously return the scalar ``0.5``, this is no
    longer supported
  - ``TimeSeries`` is now an alias for ``Series``. the property ``is_time_series``
    can be used to distinguish (if desired)

- Refactor of Sparse objects to use BlockManager

  - Created a new block type in internals, ``SparseBlock``, which can hold multi-dtypes
    and is non-consolidatable. ``SparseSeries`` and ``SparseDataFrame`` now inherit
    more methods from there hierarchy (Series/DataFrame), and no longer inherit
    from ``SparseArray`` (which instead is the object of the ``SparseBlock``)
  - Sparse suite now supports integration with non-sparse data. Non-float sparse
    data is supportable (partially implemented)
  - Operations on sparse structures within DataFrames should preserve sparseness,
    merging type operations will convert to dense (and back to sparse), so might
    be somewhat inefficient
  - enable setitem on ``SparseSeries`` for boolean/integer/slices
  - ``SparsePanels`` implementation is unchanged (e.g. not using BlockManager, needs work)

- added ``ftypes`` method to Series/DataFame, similar to ``dtypes``, but indicates
  if the underlying is sparse/dense (as well as the dtype)
- All ``NDFrame`` objects now have a ``_prop_attributes``, which can be used to indcated various
  values to propogate to a new object from an existing (e.g. name in ``Series`` will follow
  more automatically now)
- Internal type checking is now done via a suite of generated classes, allowing ``isinstance(value, klass)``
  without having to directly import the klass, courtesy of @jtratner
- Bug in Series update where the parent frame is not updating its cache based on
  changes (:issue:`4080`) or types (:issue:`3217`), fillna (:issue:`3386`)
- Indexing with dtype conversions fixed (:issue:`4463`, :issue:`4204`)
- Refactor ``Series.reindex`` to core/generic.py (:issue:`4604`, :issue:`4618`), allow ``method=`` in reindexing
  on a Series to work
- ``Series.copy`` no longer accepts the ``order`` parameter and is now consistent with ``NDFrame`` copy
- Refactor ``rename`` methods to core/generic.py; fixes ``Series.rename`` for (:issue`4605`), and adds ``rename``
  with the same signature for ``Panel``
- Refactor of ``_get_numeric_data/_get_bool_data`` to core/generic.py, allowing Series/Panel functionaility
- ``Series`` (for index) / ``Panel`` (for items) now allow attribute access to its elements  (:issue:`1903`)

  .. ipython:: python

     s = Series([1,2,3],index=list('abc'))
     s.b
     s.a = 5
     s

Bug Fixes
~~~~~~~~~

  - ``HDFStore`` raising an invalid ``TypeError`` rather than ``ValueError`` when appending
    with a different block ordering (:issue:`4096`)

  - The ``by`` argument now works correctly with the ``layout`` argument
    (:issue:`4102`, :issue:`4014`) in ``*.hist`` plotting methods

  - Fixed bug in ``PeriodIndex.map`` where using ``str`` would return the str
    representation of the index (:issue:`4136`)

  - Fixed (:issue:`3334`) in pivot_table. Margins did not compute if values is the index.

  - Fixed test failure ``test_time_series_plot_color_with_empty_kwargs`` when
    using custom matplotlib default colors (:issue:`4345`)

  - Fix running of stata IO tests. Now uses temporary files to write
    (:issue:`4353`)

  - Fixed an issue where ``DataFrame.sum`` was slower than ``DataFrame.mean``
    for integer valued frames (:issue:`4365`)

  - ``read_html`` tests now work with Python 2.6 (:issue:`4351`)

  - Fixed bug where ``network`` testing was throwing ``NameError`` because a
    local variable was undefined (:issue:`4381`)

  - Suppressed DeprecationWarning associated with internal calls issued by repr() (:issue:`4391`)

  - Fixed bug where thousands operator was not handled correctly for floating point numbers
    in csv_import (:issue:`4322`)

See the :ref:`full release notes
<release>` or issue tracker
on GitHub for a complete list.