Skip to content

Pandas 0.12 is much faster than Pandas 0.18 #14549

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
giulianoamantini opened this issue Nov 1, 2016 · 4 comments
Closed

Pandas 0.12 is much faster than Pandas 0.18 #14549

giulianoamantini opened this issue Nov 1, 2016 · 4 comments
Labels
Indexing Related to indexing on series/frames, not to indexes themselves Performance Memory or execution speed performance Usage Question

Comments

@giulianoamantini
Copy link

Hello,

I am running the code below in pandas 0.18 and benchmarking it against pandas 0.12 and the profiling shows that 0.18 performs much slower on this. I put the output in the details section. Could you please advise?

Thank you so very much for your help,
Giuliano

from datetime import date, timedelta
from time import time
import pandas as pd

sizes = [500]

base_date = date(2016,10,31)

for n in sizes:
    dates = [base_date - timedelta(days = x) for x in range(1, n, 1)]
    dates_df = pd.DataFrame({'DATE' : dates, 'key' : 1})
    identifiers = range(1, 5000)
    identifiers_df = pd.DataFrame({'IDENTIFIER' : identifiers, 'key' : 1})

    df = pd.merge(dates_df, identifiers_df, on='key')
    df = df.set_index(['DATE', 'IDENTIFIER'])
    df = df.sort_index(axis = 0, level = ['DATE', 'IDENTIFIER'], ascending=False)

    start_time = time()
    for d in dates:
        temp = df.loc[d]

    end_time = time()

    print ('%s %s' % (n, end_time - start_time))
# Output from 0.18 /opt/anaconda/base/envs/pandas0.18/bin/python2.7 /home/gamantini/QA/avoitenok/pandas_migration/testDatesPerformance.py 500 5.26671504974 337459 function calls (331370 primitive calls) in 14.176 seconds

Ordered by: standard name

ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 14.176 14.176 :1()
1519 0.002 0.000 0.002 0.000 init.py:157(iteritems)
1 0.000 0.000 0.000 0.000 init.py:22(find_module)
2 0.000 0.000 0.000 0.000 _methods.py:34(_prod)
1027 0.001 0.000 0.023 0.000 _methods.py:37(_any)
7 0.000 0.000 0.000 0.000 _methods.py:40(_all)
6 0.020 0.003 2.945 0.491 algorithms.py:145(factorize)
6 0.000 0.000 0.007 0.001 algorithms.py:616(_get_data_algo)
11 0.000 0.000 0.000 0.000 algorithms.py:807(_get_take_nd_function)
11 0.012 0.001 0.527 0.048 algorithms.py:840(take_nd)
499 0.002 0.000 0.004 0.000 base.py:1016(_convert_slice_indexer)
2002 0.001 0.000 0.004 0.000 base.py:106(_reset_cache)
1 0.000 0.000 0.000 0.000 base.py:1136(_cleanup)
14 0.000 0.000 0.000 0.000 base.py:1143(_engine)
15 0.000 0.000 0.000 0.000 base.py:1146()
1 0.000 0.000 0.000 0.000 base.py:1172(inferred_type)
6 0.000 0.000 0.000 0.000 base.py:1233(contains)
16/14 0.001 0.000 0.003 0.000 base.py:124(new)
10 0.000 0.000 0.000 0.000 base.py:1247(getitem)
1 0.000 0.000 0.000 0.000 base.py:1281(_ensure_compat_append)
1 0.000 0.000 0.000 0.000 base.py:1308(append)
1 0.000 0.000 0.000 0.000 base.py:1326(_ensure_compat_concat)
499 0.005 0.000 0.047 0.000 base.py:1360(take)
4 0.000 0.000 0.000 0.000 base.py:1415(hasnans)
5 0.000 0.000 0.000 0.000 base.py:1507(equals)
1 0.000 0.000 0.000 0.000 base.py:1520(identical)
1 0.000 0.000 0.003 0.003 base.py:169(_box_values)
1 0.000 0.000 0.000 0.000 base.py:1770(_wrap_union_result)
1 0.000 0.000 0.000 0.000 base.py:1774(intersection)
2007 0.013 0.000 0.043 0.000 base.py:1915(get_loc)
2 0.000 0.000 0.000 0.000 base.py:2028(get_indexer)
998 0.002 0.000 0.002 0.000 base.py:218(freqstr)
2 0.000 0.000 0.000 0.000 base.py:2201(_possibly_promote)
1 0.000 0.000 0.000 0.000 base.py:227(inferred_freq)
1 0.000 0.000 0.000 0.000 base.py:2276(_can_reindex)
1 0.000 0.000 0.000 0.000 base.py:2295(reindex)
998 0.001 0.000 0.002 0.000 base.py:2795(_maybe_cast_indexer)
1497 0.001 0.000 0.002 0.000 base.py:2810(_validate_indexer)
1 0.000 0.000 0.000 0.000 base.py:3001(delete)
1 0.000 0.000 0.000 0.000 base.py:3031(drop)
1517 0.006 0.000 0.011 0.000 base.py:309(_simple_new)
2 0.000 0.000 0.000 0.000 base.py:321(_isnan)
2 0.000 0.000 0.000 0.000 base.py:326(hasnans)
3527 0.002 0.000 0.004 0.000 base.py:3381(_ensure_index)
1 0.000 0.000 0.000 0.000 base.py:3412(_get_na_value)
2002 0.003 0.000 0.012 0.000 base.py:3417(_ensure_frozen)
1 0.000 0.000 0.000 0.000 base.py:3425(_ensure_has_len)
2506 0.011 0.000 0.070 0.000 base.py:346(_shallow_copy)
1 0.000 0.000 0.000 0.000 base.py:354(shallow_copy_with_infer)
5 0.000 0.000 0.000 0.000 base.py:383(is)
3523 0.003 0.000 0.003 0.000 base.py:403(_reset_identity)
3579 0.002 0.000 0.003 0.000 base.py:409(len)
1 0.000 0.000 0.000 0.000 base.py:415(array)
4 0.000 0.000 0.000 0.000 base.py:430(dtype)
2542 0.003 0.000 0.006 0.000 base.py:440(values)
10 0.000 0.000 0.000 0.000 base.py:445(get_values)
5 0.000 0.000 0.000 0.000 base.py:490(_coerce_to_ndarray)
2508 0.011 0.000 0.014 0.000 base.py:506(_get_attributes_dict)
2 0.000 0.000 0.000 0.000 base.py:510(view)
6001 0.011 0.000 0.015 0.000 base.py:711(getitem)
2002 0.001 0.000 0.002 0.000 base.py:774(_shallow_copy)
1 0.000 0.000 0.000 0.000 base.py:778(_assert_can_do_setop)
2 0.000 0.000 0.000 0.000 base.py:791(nlevels)
1 0.000 0.000 0.000 0.000 base.py:795(_get_names)
2003 0.001 0.000 0.002 0.000 base.py:798(_set_names)
2003 0.004 0.000 0.013 0.000 base.py:806(set_names)
2002 0.002 0.000 0.015 0.000 base.py:865(rename)
8 0.000 0.000 0.000 0.000 base.py:870(_values)
2 0.000 0.000 0.000 0.000 base.py:914(is_monotonic)
1 0.000 0.000 0.000 0.000 base.py:919(is_monotonic_increasing)
10 0.000 0.000 0.000 0.000 base.py:938(is_unique)
499 0.001 0.000 0.005 0.000 base.py:972(_convert_scalar_indexer)
6 0.003 0.001 3.767 0.628 categorical.py:222(init)
4 0.010 0.003 2.967 0.742 categorical.py:374(from_array)
8 0.000 0.000 0.000 0.000 categorical.py:436(_get_codes)
6 0.000 0.000 0.003 0.000 categorical.py:470(_validate_categories)
6 0.000 0.000 0.000 0.000 categorical.py:534(_get_categories)
6 0.000 0.000 0.000 0.000 categorical.py:565(set_ordered)
1500 0.002 0.000 0.002 0.000 collections.py:78(iter)
4 0.000 0.000 0.000 0.000 common.py:1011(_possibly_cast_to_datetime)
10 0.000 0.000 0.802 0.080 common.py:1111(_possibly_infer_to_datetimelike)
2 0.000 0.000 0.801 0.401 common.py:1147(_try_datetime)
2497 0.005 0.000 0.017 0.000 common.py:1203(is_bool_indexer)
2 0.000 0.000 0.000 0.000 common.py:1226(_default_index)
2 0.000 0.000 0.000 0.000 common.py:1268(_try_sort)
1 0.000 0.000 0.000 0.000 common.py:1276(_count_not_none)
3 0.000 0.000 0.000 0.000 common.py:1277()
11 0.000 0.000 0.000 0.000 common.py:1380(_asarray_tuplesafe)
1 0.000 0.000 0.000 0.000 common.py:1412(_index_labels_to_array)
3 0.000 0.000 0.000 0.000 common.py:1427(_maybe_make_list)
1 0.000 0.000 0.000 0.000 common.py:1456(is_period_arraylike)
12 0.000 0.000 0.000 0.000 common.py:1491(_get_dtype)
2055 0.006 0.000 0.009 0.000 common.py:1511(_get_dtype_type)
5 0.000 0.000 0.000 0.000 common.py:1532(is_dtype_equal)
4 0.000 0.000 0.000 0.000 common.py:1550(is_integer_dtype)
4 0.000 0.000 0.000 0.000 common.py:1561(is_int_or_datetime_dtype)
1011 0.002 0.000 0.005 0.000 common.py:1567(is_datetime64_dtype)
64 0.000 0.000 0.000 0.000 common.py:1575(is_datetime64tz_dtype)
8 0.000 0.000 0.000 0.000 common.py:1592(is_timedelta64_dtype)
6 0.000 0.000 0.000 0.000 common.py:1602(is_datetime_or_timedelta_dtype)
1 0.000 0.000 0.000 0.000 common.py:165(_isnull_ndarraylike)
6 0.000 0.000 0.000 0.000 common.py:1664(needs_i8_conversion)
1 0.000 0.000 0.000 0.000 common.py:1675(is_string_dtype)
1 0.000 0.000 0.000 0.000 common.py:1680(is_string_like_dtype)
6 0.000 0.000 0.000 0.000 common.py:1686(is_float_dtype)
8 0.000 0.000 0.000 0.000 common.py:1696(is_bool_dtype)
6 0.000 0.000 0.000 0.000 common.py:1705(is_sparse)
58 0.000 0.000 0.001 0.000 common.py:1710(is_datetimetz)
2 0.000 0.000 0.000 0.000 common.py:1717(is_extension_type)
16 0.000 0.000 0.000 0.000 common.py:1731(is_categorical)
70 0.000 0.000 0.001 0.000 common.py:1736(is_categorical_dtype)
1008 0.002 0.000 0.009 0.000 common.py:1745(is_object_dtype)
5019 0.005 0.000 0.008 0.000 common.py:1763(is_list_like)
4 0.000 0.000 0.000 0.000 common.py:1788(is_hashable)
1002 0.001 0.000 0.001 0.000 common.py:1846(_apply_if_callable)
5 0.000 0.000 0.000 0.000 common.py:272(array_equivalent)
2008 0.002 0.000 0.021 0.000 common.py:379(_coerce_indexer_dtype)
2 0.000 0.000 0.000 0.000 common.py:446(_infer_dtype_from_scalar)
8 0.000 0.000 0.001 0.000 common.py:527(_maybe_promote)
5 0.000 0.000 0.000 0.000 common.py:73(isnull)
5 0.000 0.000 0.000 0.000 common.py:94(_isnull_new)
2 0.000 0.000 0.001 0.000 common.py:998(_possibly_convert_platform)
2 0.000 0.000 0.000 0.000 decorators.py:65(wrapper)
27 0.000 0.000 0.000 0.000 dtypes.py:122(construct_from_string)
9 0.000 0.000 0.000 0.000 dtypes.py:159(init)
9 0.000 0.000 0.000 0.000 dtypes.py:191(construct_from_string)
134 0.000 0.000 0.001 0.000 dtypes.py:74(is_dtype)
4 0.000 0.000 0.002 0.000 frame.py:1973(getitem)
503 0.000 0.000 0.000 0.000 frame.py:199(_constructor)
4 0.000 0.000 0.001 0.000 frame.py:1999(_getitem_column)
505 0.004 0.000 0.011 0.000 frame.py:210(init)
4 0.000 0.000 0.001 0.000 frame.py:2331(_box_item_values)
4 0.000 0.000 0.001 0.000 frame.py:2338(_box_col_values)
1 0.000 0.000 0.001 0.001 frame.py:2675(_reindex_axes)
1 0.000 0.000 0.001 0.001 frame.py:2700(_reindex_columns)
1 0.000 0.000 0.001 0.001 frame.py:2738(reindex)
1 0.000 0.000 2.401 2.401 frame.py:2769(set_index)
2 0.000 0.000 0.003 0.001 frame.py:307(_init_dict)
1 0.022 0.022 5.554 5.554 frame.py:3227(sort_index)
1 0.000 0.000 0.000 0.000 frame.py:434(axes)
2 0.000 0.000 0.002 0.001 frame.py:5224(_arrays_to_mgr)
2 0.000 0.000 0.000 0.000 frame.py:5244(extract_index)
2 0.000 0.000 0.001 0.001 frame.py:5521(_homogenize)
2 0.000 0.000 0.000 0.000 frame.py:726(len)
1 0.000 0.000 0.000 0.000 frequencies.py:1017(mdiffs)
1 0.000 0.000 0.000 0.000 frequencies.py:1022(ydiffs)
1 0.000 0.000 0.000 0.000 frequencies.py:1026(_infer_daily_rule)
1 0.000 0.000 0.000 0.000 frequencies.py:1062(_get_annual_rule)
1 0.000 0.000 0.000 0.000 frequencies.py:1073(_get_quarterly_rule)
1 0.000 0.000 0.000 0.000 frequencies.py:1084(_get_monthly_rule)
1 0.000 0.000 0.000 0.000 frequencies.py:1127(_maybe_add_count)
1 0.000 0.000 0.000 0.000 frequencies.py:1300(_is_multiple)
2 0.000 0.000 0.000 0.000 frequencies.py:420(to_offset)
1 0.000 0.000 0.000 0.000 frequencies.py:826(infer_freq)
1 0.000 0.000 0.000 0.000 frequencies.py:890(init)
1 0.000 0.000 0.000 0.000 frequencies.py:908(deltas)
1 0.000 0.000 0.000 0.000 frequencies.py:916(is_unique)
1 0.000 0.000 0.000 0.000 frequencies.py:924(get_freq)
1 0.000 0.000 0.000 0.000 frequencies.py:968(fields)
2 0.000 0.000 0.000 0.000 fromnumeric.py:2395(prod)
1 0.000 0.000 0.000 0.000 fromnumeric.py:829(argsort)
500 0.002 0.000 0.019 0.000 function.py:36(call)
3 0.107 0.036 0.107 0.036 function_base.py:4110(delete)
503 0.001 0.000 0.001 0.000 generic.py:124(_init_mgr)
998 0.001 0.000 0.002 0.000 generic.py:1312(_indexer)
4 0.000 0.000 0.001 0.000 generic.py:1345(_get_item_cache)
4 0.000 0.000 0.000 0.000 generic.py:1359(_set_as_cached)
499 0.001 0.000 0.003 0.000 generic.py:1402(_is_view)
500 0.001 0.000 0.001 0.000 generic.py:1441(_clear_item_cache)
499 0.006 0.000 0.167 0.000 generic.py:1447(_slice)
998 0.003 0.000 0.005 0.000 generic.py:1467(_set_is_copy)
2 0.000 0.000 0.108 0.054 generic.py:1580(delitem)
499 0.014 0.000 2.877 0.006 generic.py:1643(xs)
1 0.000 0.000 0.001 0.001 generic.py:1846(drop)
1 0.000 0.000 0.001 0.001 generic.py:2194(reindex)
1 0.000 0.000 0.000 0.000 generic.py:2251(_needs_reindex_multi)
1 0.000 0.000 0.000 0.000 generic.py:2320(_reindex_with_indexers)
1 0.000 0.000 0.000 0.000 generic.py:260(_construct_axes_from_arguments)
504 0.001 0.000 0.001 0.000 generic.py:2641(finalize)
4 0.000 0.000 0.000 0.000 generic.py:2658(getattr)
2012 0.007 0.000 0.016 0.000 generic.py:2674(setattr)
500 0.002 0.000 0.010 0.000 generic.py:2713(_protect_consolidate)
500 0.002 0.000 0.012 0.000 generic.py:2723(_consolidate_inplace)
500 0.003 0.000 0.008 0.000 generic.py:2726(f)
1 0.000 0.000 0.877 0.877 generic.py:2953(copy)
1002 0.004 0.000 0.005 0.000 generic.py:307(_get_axis_number)
2499 0.005 0.000 0.006 0.000 generic.py:320(_get_axis_name)
2498 0.004 0.000 0.014 0.000 generic.py:333(_get_axis)
501 0.001 0.000 0.003 0.000 generic.py:337(_get_block_manager_axis)
2 0.000 0.000 0.000 0.000 generic.py:381(_info_axis)
2 0.000 0.000 0.000 0.000 generic.py:401(ndim)
500 0.002 0.000 0.009 0.000 generic.py:427(_set_axis)
3151 0.003 0.000 0.007 0.000 generic.py:7(_check)
2 0.000 0.000 0.000 0.000 generic.py:844(contains)
509 0.002 0.000 0.002 0.000 generic.py:94(init)
1 0.115 0.115 0.339 0.339 groupby.py:3978(get_group_index)
1 0.000 0.000 0.000 0.000 groupby.py:4005(_int64_cut_off)
1 0.218 0.218 0.218 0.218 groupby.py:4013(loop)
2 0.005 0.003 0.006 0.003 groupby.py:4043(maybe_lift)
1 0.000 0.000 0.000 0.000 groupby.py:4056(_int64_overflow_possible)
1 0.004 0.004 1.720 1.720 groupby.py:4110(_indexer_from_factorized)
1 0.010 0.010 2.549 2.549 groupby.py:4122(_lexsort_indexer)
1 0.000 0.000 0.030 0.030 groupby.py:4244(_get_group_index_sorter)
1 0.001 0.001 1.348 1.348 groupby.py:4269(_compress_group_index)
1 0.004 0.004 0.237 0.237 groupby.py:4290(_reorder_by_uniques)
998 0.035 0.000 0.102 0.000 index.py:1406(get_loc)
998 0.004 0.000 0.024 0.000 index.py:1477(_get_string_slice)
1999 0.001 0.000 0.001 0.000 index.py:1538(_get_freq)
1 0.000 0.000 0.000 0.000 index.py:1657(inferred_type)
1 0.000 0.000 0.000 0.000 index.py:1663(dtype)
2 0.000 0.000 0.000 0.000 index.py:219(new)
1 0.000 0.000 0.000 0.000 index.py:545(_box_func)
499 0.003 0.000 0.003 0.000 index.py:547()
1003 0.009 0.000 0.028 0.000 index.py:570(_simple_new)
2 0.000 0.000 0.000 0.000 index_tricks.py:231(_retval)
2 0.000 0.000 0.000 0.000 index_tricks.py:251(getitem)
998/499 0.005 0.000 5.262 0.011 indexing.py:1286(getitem)
499 0.006 0.000 2.364 0.005 indexing.py:1355(_has_valid_type)
499 0.001 0.000 0.001 0.000 indexing.py:1407(_get_partial_string_timestamp_match_key)
499 0.006 0.000 5.259 0.011 indexing.py:1431(_getitem_axis)
499 0.001 0.000 0.003 0.000 indexing.py:1498(_has_valid_type)
499 0.003 0.000 0.183 0.000 indexing.py:1571(_get_slice_axis)
499 0.002 0.000 0.188 0.000 indexing.py:1583(_getitem_axis)
499 0.002 0.000 0.009 0.000 indexing.py:183(_convert_scalar_indexer)
499 0.003 0.000 0.012 0.000 indexing.py:189(_convert_slice_indexer)
998 0.001 0.000 0.002 0.000 indexing.py:1890(is_list_like_indexer)
499 0.001 0.000 0.001 0.000 indexing.py:1901(need_slice)
2 0.000 0.000 0.000 0.000 indexing.py:50(init)
499 0.004 0.000 2.882 0.006 indexing.py:80(_get_label)
499 0.001 0.000 0.168 0.000 indexing.py:98(_slice)
499 0.001 0.000 0.001 0.000 internals.py:100(is_view)
4 0.000 0.000 0.000 0.000 internals.py:129(internal_values)
3 0.000 0.000 0.000 0.000 internals.py:135(get_values)
2 0.000 0.000 0.000 0.000 internals.py:156(fill_value)
1555 0.001 0.000 0.001 0.000 internals.py:160(mgr_locs)
4 0.000 0.000 0.000 0.000 internals.py:1657(init)
1 0.000 0.000 0.000 0.000 internals.py:1665(is_bool)
507 0.002 0.000 0.008 0.000 internals.py:183(make_block_same_class)
517 0.001 0.000 0.001 0.000 internals.py:191(mgr_locs)
499 0.001 0.000 0.001 0.000 internals.py:225(_slice)
499 0.006 0.000 0.016 0.000 internals.py:244(getitem_block)
514 0.002 0.000 0.006 0.000 internals.py:2482(make_block)
505 0.004 0.000 0.053 0.000 internals.py:2578(init)
1027 0.004 0.000 0.012 0.000 internals.py:2619(shape)
3081 0.001 0.000 0.008 0.000 internals.py:2621()
1016 0.001 0.000 0.001 0.000 internals.py:2623(ndim)
500 0.002 0.000 0.007 0.000 internals.py:2627(set_axis)
8 0.000 0.000 0.000 0.000 internals.py:264(shape)
3 0.000 0.000 0.000 0.000 internals.py:2662(_is_single_block)
508 0.015 0.000 0.041 0.000 internals.py:2674(_rebuild_blknos_and_blklocs)
21 0.000 0.000 0.000 0.000 internals.py:2695(_get_items)
520 0.000 0.000 0.000 0.000 internals.py:272(dtype)
509 0.003 0.000 0.004 0.000 internals.py:276(ftype)
2 0.000 0.000 0.000 0.000 internals.py:2784(len)
5 0.000 0.000 0.000 0.000 internals.py:2799(_verify_integrity)
13 0.000 0.000 0.000 0.000 internals.py:2801()
1 0.000 0.000 0.877 0.877 internals.py:2811(apply)
1 0.000 0.000 0.000 0.000 internals.py:2871()
1006 0.001 0.000 0.001 0.000 internals.py:2996(is_consolidated)
505 0.003 0.000 0.006 0.000 internals.py:3004(_consolidate_check)
4 0.000 0.000 0.000 0.000 internals.py:301(iget)
499 0.001 0.000 0.002 0.000 internals.py:3027(is_view)
499 0.009 0.000 0.145 0.000 internals.py:3084(get_slice)
1 0.000 0.000 0.877 0.877 internals.py:3111(copy)
2 0.000 0.000 0.000 0.000 internals.py:3131()
1 0.000 0.000 0.107 0.107 internals.py:314(delete)
500 0.001 0.000 0.002 0.000 internals.py:3260(consolidate)
506 0.004 0.000 0.498 0.001 internals.py:3276(_consolidate_inplace)
4 0.000 0.000 0.000 0.000 internals.py:3283(get)
4 0.000 0.000 0.000 0.000 internals.py:3312(iget)
2 0.000 0.000 0.108 0.054 internals.py:3345(delete)
4 0.000 0.000 0.000 0.000 internals.py:3378()
2 0.000 0.000 0.012 0.006 internals.py:3560(reindex_indexer)
1 0.000 0.000 0.000 0.000 internals.py:3603(_slice_take_blocks_ax0)
1 0.000 0.000 0.030 0.030 internals.py:3690(take)
4 0.000 0.000 0.000 0.000 internals.py:3778(init)
4 0.000 0.000 0.000 0.000 internals.py:3824(_block)
4 0.000 0.000 0.000 0.000 internals.py:3911(internal_values)
2 0.000 0.000 0.001 0.001 internals.py:3996(create_block_manager_from_arrays)
2 0.000 0.000 0.001 0.000 internals.py:4007(form_blocks)
1 0.000 0.000 0.000 0.000 internals.py:4122(_simple_blockify)
2 0.000 0.000 0.000 0.000 internals.py:4136(_multi_blockify)
3 0.000 0.000 0.000 0.000 internals.py:4140()
3 0.000 0.000 0.000 0.000 internals.py:4168(_stack_arrays)
4 0.000 0.000 0.000 0.000 internals.py:4171(_asarray_compat)
3 0.000 0.000 0.000 0.000 internals.py:4177(_shape_compat)
1 0.000 0.000 0.493 0.493 internals.py:4257(_consolidate)
6 0.000 0.000 0.000 0.000 internals.py:4263()
2 0.250 0.125 0.493 0.247 internals.py:4274(_merge_blocks)
4 0.000 0.000 0.000 0.000 internals.py:4301(_extend_blocks)
1 0.000 0.000 0.243 0.243 internals.py:4327(_vstack)
5 0.000 0.000 0.000 0.000 internals.py:4409(_get_blkno_placements)
1 0.000 0.000 0.000 0.000 internals.py:4432(items_overlap_with_suffix)
1 0.000 0.000 0.372 0.372 internals.py:4526(concatenate_block_managers)
3 0.000 0.000 0.012 0.004 internals.py:4550(get_empty_dtype_and_na)
3 0.000 0.000 0.372 0.124 internals.py:4630(concatenate_join_units)
2 0.000 0.000 0.000 0.000 internals.py:4655(get_mgr_concatenation_plan)
4 0.000 0.000 0.000 0.000 internals.py:4734(combine_concat_plans)
3 0.000 0.000 0.000 0.000 internals.py:4827(init)
3 0.011 0.004 0.011 0.004 internals.py:4839(needs_filling)
3 0.000 0.000 0.011 0.004 internals.py:4848(dtype)
3 0.000 0.000 0.000 0.000 internals.py:4861(is_null)
3 0.000 0.000 0.360 0.120 internals.py:4890(get_reindexed_values)
2 0.000 0.000 0.000 0.000 internals.py:4947(_fast_count_smallints)
1 0.000 0.000 0.000 0.000 internals.py:4958(_preprocess_slice_or_indexer)
2 0.000 0.000 0.379 0.190 internals.py:581(copy)
514 0.002 0.000 0.004 0.000 internals.py:77(init)
6 0.000 0.000 0.000 0.000 internals.py:92(_consolidate_key)
2 0.000 0.000 0.011 0.006 internals.py:975(take_nd)
1 0.000 0.000 0.000 0.000 merge.py:1252(_should_fill)
2 0.000 0.000 0.000 0.000 merge.py:1259(_any)
1 0.000 0.000 0.002 0.002 merge.py:161(init)
1 0.000 0.000 0.929 0.929 merge.py:212(get_result)
1 0.003 0.003 0.003 0.003 merge.py:282(_maybe_add_join_keys)
1 0.003 0.003 0.933 0.933 merge.py:30(merge)
1 0.000 0.000 0.553 0.553 merge.py:333(_get_join_info)
1 0.000 0.000 0.002 0.002 merge.py:393(_get_merge_keys)
1 0.000 0.000 0.000 0.000 merge.py:416()
1 0.000 0.000 0.000 0.000 merge.py:418()
1 0.000 0.000 0.000 0.000 merge.py:482(_validate_specification)
1 0.000 0.000 0.458 0.458 merge.py:527(_get_join_indexers)
2 0.000 0.000 0.000 0.000 merge.py:698(_factorize_keys)
1 0.000 0.000 0.000 0.000 merge.py:757(_get_join_keys)
1 0.000 0.000 0.000 0.000 merge.py:761()
3 0.000 0.000 0.000 0.000 missing.py:559(clean_reindex_fill_method)
3 0.000 0.000 0.000 0.000 missing.py:61(clean_fill_method)
1 0.000 0.000 0.019 0.019 multi.py:1010(take)
1 0.000 0.000 0.018 0.018 multi.py:1022(_assert_take_fillable)
499 0.033 0.000 0.111 0.000 multi.py:1155(droplevel)
998 0.001 0.000 0.009 0.000 multi.py:1180()
14495 0.004 0.000 0.004 0.000 multi.py:131(_get_levels)
1001 0.008 0.000 0.086 0.000 multi.py:134(_set_levels)
3003 0.005 0.000 0.071 0.000 multi.py:148()
499 0.004 0.000 2.342 0.005 multi.py:1515(get_loc)
499 0.003 0.000 0.005 0.000 multi.py:1534(_maybe_to_slice)
499 0.009 0.000 2.633 0.005 multi.py:1601(get_loc_level)
499 0.008 0.000 0.274 0.001 multi.py:1615(maybe_droplevels)
998 0.029 0.000 4.658 0.005 multi.py:1734(_get_level_indexer)
4503 0.002 0.000 0.002 0.000 multi.py:240(_get_labels)
1001 0.008 0.000 0.030 0.000 multi.py:243(_set_labels)
3003 0.005 0.000 0.019 0.000 multi.py:253()
2002 0.003 0.000 0.009 0.000 multi.py:433(len)
5493 0.028 0.000 0.035 0.000 multi.py:436(_get_names)
14477 0.005 0.000 0.005 0.000 multi.py:437()
1001 0.010 0.000 0.034 0.000 multi.py:439(_set_names)
1497 0.016 0.000 0.046 0.000 multi.py:510(_get_level_number)
1 0.000 0.000 1.315 1.315 multi.py:535(values)
1001 0.009 0.000 0.160 0.000 multi.py:69(new)
1 0.000 0.000 0.009 0.009 multi.py:781(is_lexsorted)
2 0.000 0.000 0.023 0.011 multi.py:793(lexsort_depth)
2 0.000 0.000 2.968 1.484 multi.py:808(from_arrays)
1 0.000 0.000 1.628 1.628 multi.py:852(from_tuples)
3502 0.003 0.000 0.004 0.000 multi.py:941(nlevels)
499 0.002 0.000 2.345 0.005 multi.py:949(contains)
998 0.021 0.000 0.206 0.000 multi.py:986(getitem)
2 0.000 0.000 0.000 0.000 numeric.py:121(asi8)
4 0.000 0.000 0.000 0.000 numeric.py:126(is_all_dates)
3 0.000 0.000 0.000 0.000 numeric.py:148(ones)
53 0.000 0.000 0.000 0.000 numeric.py:414(asarray)
4 0.000 0.000 0.000 0.000 numeric.py:484(asanyarray)
5 0.000 0.000 0.000 0.000 numeric.py:607(require)
10 0.000 0.000 0.000 0.000 numeric.py:676()
5 0.000 0.000 0.000 0.000 numeric.py:93(new)
4 0.000 0.000 0.000 0.000 numerictypes.py:942(_can_coerce_all)
2 0.000 0.000 0.000 0.000 numerictypes.py:964(find_common_type)
2 0.000 0.000 0.000 0.000 range.py:117(_simple_new)
2 0.000 0.000 0.000 0.000 range.py:139(_validate_dtype)
2 0.000 0.000 0.000 0.000 range.py:42(new)
18 0.000 0.000 0.000 0.000 range.py:435(len)
4 0.000 0.000 0.000 0.000 range.py:58(_ensure_int)
4 0.000 0.000 0.000 0.000 series.py:120(init)
4 0.000 0.000 0.000 0.000 series.py:236(from_array)
4 0.000 0.000 0.000 0.000 series.py:270(_set_axis)
4 0.000 0.000 0.001 0.000 series.py:2787(_sanitize_array)
2 0.000 0.000 0.000 0.000 series.py:2804(_try_cast)
2 0.000 0.000 0.000 0.000 series.py:2875(create_from_value)
4 0.000 0.000 0.000 0.000 series.py:292(_set_subtyp)
4 0.000 0.000 0.000 0.000 series.py:302(name)
4 0.000 0.000 0.000 0.000 series.py:306(name)
4 0.000 0.000 0.000 0.000 series.py:366(_values)
1 0.000 0.000 0.243 0.243 shape_base.py:180(vstack)
2 0.000 0.000 0.000 0.000 shape_base.py:61(atleast_2d)
1 0.000 0.000 0.000 0.000 six.py:180(find_module)
3 0.000 0.000 0.000 0.000 six.py:184(find_module)
1 0.023 0.023 14.176 14.176 testDatesPerformance.py:6(testPerformance)
998 0.013 0.000 0.014 0.000 tools.py:603(parse_time_string)
500 0.006 0.000 0.007 0.000 validators.py:104(_check_for_invalid_keys)
500 0.002 0.000 0.017 0.000 validators.py:120(validate_kwargs)
500 0.007 0.000 0.007 0.000 validators.py:29(_check_for_default_values)
1 0.000 0.000 0.000 0.000 {all}
1004 0.001 0.000 0.001 0.000 {any}
3523 0.002 0.000 0.002 0.000 {built-in method new of type object at 0x7f63038a7f20}
1002 0.000 0.000 0.000 0.000 {callable}
6001 0.002 0.000 0.002 0.000 {function getitem at 0x7f62f7ed9230}
16175/16172 0.017 0.000 0.020 0.000 {getattr}
6711 0.003 0.000 0.003 0.000 {hasattr}
509 0.001 0.000 0.001 0.000 {hash}
50016 0.025 0.000 0.032 0.000 {isinstance}
2230 0.001 0.000 0.001 0.000 {issubclass}
27470/21885 0.010 0.000 0.021 0.000 {len}
21 0.000 0.000 0.000 0.000 {max}
5 0.000 0.000 0.000 0.000 {method 'add' of 'pandas.lib.BlockPlacement' objects}
7 0.000 0.000 0.000 0.000 {method 'all' of 'numpy.ndarray' objects}
1025 0.002 0.000 0.025 0.000 {method 'any' of 'numpy.ndarray' objects}
49 0.000 0.000 0.000 0.000 {method 'append' of 'list' objects}
8 0.055 0.007 0.055 0.007 {method 'argsort' of 'numpy.ndarray' objects}
12 0.000 0.000 0.000 0.000 {method 'astype' of 'numpy.ndarray' objects}
500 0.000 0.000 0.000 0.000 {method 'clear' of 'dict' objects}
1 0.000 0.000 0.000 0.000 {method 'clear_mapping' of 'pandas.index.IndexEngine' objects}
502 0.000 0.000 0.000 0.000 {method 'copy' of 'dict' objects}
4 0.381 0.095 0.381 0.095 {method 'copy' of 'numpy.ndarray' objects}
1497 0.002 0.000 0.002 0.000 {method 'count' of 'list' objects}
2 0.000 0.000 0.000 0.000 {method 'cumsum' of 'numpy.ndarray' objects}
1 0.000 0.000 0.000 0.000 {method 'delete' of 'pandas.lib.BlockPlacement' objects}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
3 0.000 0.000 0.000 0.000 {method 'extend' of 'list' objects}
4 0.000 0.000 0.000 0.000 {method 'factorize' of 'pandas.hashtable.Int64Factorizer' objects}
1018 0.002 0.000 0.002 0.000 {method 'fill' of 'numpy.ndarray' objects}
3519 0.001 0.000 0.001 0.000 {method 'get' of 'dict' objects}
2 0.000 0.000 0.000 0.000 {method 'get_count' of 'pandas.hashtable.Int64Factorizer' objects}
2 0.000 0.000 0.000 0.000 {method 'get_indexer' of 'pandas.index.IndexEngine' objects}
5 1.934 0.387 1.934 0.387 {method 'get_labels' of 'pandas.hashtable.Int64HashTable' objects}
1 0.264 0.264 0.264 0.264 {method 'get_labels' of 'pandas.hashtable.PyObjectHashTable' objects}
1 1.110 1.110 1.110 1.110 {method 'get_labels_groupby' of 'pandas.hashtable.Int64HashTable' objects}
2994 0.023 0.000 0.023 0.000 {method 'get_loc' of 'pandas.index.DatetimeEngine' objects}
11 0.000 0.000 0.000 0.000 {method 'get_loc' of 'pandas.index.IndexEngine' objects}
1497 0.006 0.000 0.006 0.000 {method 'index' of 'list' objects}
509 0.000 0.000 0.000 0.000 {method 'items' of 'dict' objects}
1519 0.001 0.000 0.001 0.000 {method 'iteritems' of 'dict' objects}
3 0.000 0.000 0.000 0.000 {method 'keys' of 'dict' objects}
6 0.000 0.000 0.000 0.000 {method 'nonzero' of 'numpy.ndarray' objects}
1 0.000 0.000 0.000 0.000 {method 'partition' of 'str' objects}
16 0.000 0.000 0.000 0.000 {method 'pop' of 'dict' objects}
1497 0.001 0.000 0.001 0.000 {method 'pop' of 'list' objects}
7 0.039 0.006 0.039 0.006 {method 'put' of 'numpy.ndarray' objects}
7 0.000 0.000 0.000 0.000 {method 'ravel' of 'numpy.ndarray' objects}
1036 0.021 0.000 0.021 0.000 {method 'reduce' of 'numpy.ufunc' objects}
3 0.000 0.000 0.000 0.000 {method 'reshape' of 'numpy.ndarray' objects}
9 0.000 0.000 0.000 0.000 {method 'search' of '_sre.SRE_Pattern' objects}
1996 4.507 0.002 4.507 0.002 {method 'searchsorted' of 'numpy.ndarray' objects}
2 0.000 0.000 0.000 0.000 {method 'startswith' of 'str' objects}
513 0.733 0.001 0.733 0.001 {method 'take' of 'numpy.ndarray' objects}
5 0.000 0.000 0.000 0.000 {method 'to_array' of 'pandas.hashtable.Int64Vector' objects}
1 0.000 0.000 0.000 0.000 {method 'to_array' of 'pandas.hashtable.ObjectVector' objects}
4 0.000 0.000 0.000 0.000 {method 'transpose' of 'numpy.ndarray' objects}
2507 0.001 0.000 0.001 0.000 {method 'update' of 'dict' objects}
5 0.000 0.000 0.000 0.000 {method 'upper' of 'str' objects}
4 0.000 0.000 0.000 0.000 {method 'values' of 'dict' objects}
6560 0.006 0.000 0.006 0.000 {method 'view' of 'numpy.ndarray' objects}
1009 0.001 0.000 0.001 0.000 {min}
1 0.000 0.000 0.000 0.000 {next}
523 0.115 0.000 0.115 0.000 {numpy.core.multiarray.arange}
1090 0.001 0.000 0.001 0.000 {numpy.core.multiarray.array}
2 0.000 0.000 0.000 0.000 {numpy.core.multiarray.bincount}
3 0.000 0.000 0.000 0.000 {numpy.core.multiarray.can_cast}
5 0.243 0.049 0.243 0.049 {numpy.core.multiarray.concatenate}
3 0.000 0.000 0.000 0.000 {numpy.core.multiarray.copyto}
1043 0.121 0.000 0.121 0.000 {numpy.core.multiarray.empty}
7 0.012 0.002 0.012 0.002 {numpy.core.multiarray.putmask}
2 0.006 0.003 0.006 0.003 {numpy.core.multiarray.where}
2 0.000 0.000 0.000 0.000 {numpy.core.multiarray.zeros}
2008 0.015 0.000 0.015 0.000 {pandas.algos.ensure_int16}
38 0.028 0.001 0.028 0.001 {pandas.algos.ensure_int64}
5 0.000 0.000 0.000 0.000 {pandas.algos.ensure_object}
509 0.007 0.000 0.007 0.000 {pandas.algos.ensure_platform_int}
1 0.030 0.030 0.030 0.030 {pandas.algos.groupsort_indexer}
1 0.000 0.000 0.000 0.000 {pandas.algos.inner_join_indexer_object}
1 0.457 0.457 0.457 0.457 {pandas.algos.inner_join}
5 0.126 0.025 0.126 0.025 {pandas.algos.take_1d_int64_int64}
1 0.013 0.013 0.013 0.013 {pandas.algos.take_1d_object_object}
1 0.000 0.000 0.000 0.000 {pandas.algos.take_2d_axis0_int64_int64}
3 0.239 0.080 0.239 0.080 {pandas.algos.take_2d_axis1_int64_int64}
1 0.014 0.014 0.014 0.014 {pandas.algos.take_2d_axis1_object_object}
2 0.000 0.000 0.000 0.000 {pandas.lib.array_equivalent_object}
4 0.000 0.000 0.000 0.000 {pandas.lib.checknull}
1 1.278 1.278 1.278 1.278 {pandas.lib.fast_zip}
2 0.000 0.000 0.000 0.000 {pandas.lib.get_blkno_indexers}
19 0.000 0.000 0.000 0.000 {pandas.lib.infer_dtype}
1 0.000 0.000 0.000 0.000 {pandas.lib.is_bool_array}
11 0.000 0.000 0.000 0.000 {pandas.lib.is_bool}
1 0.000 0.000 0.000 0.000 {pandas.lib.is_complex}
1006 0.000 0.000 0.000 0.000 {pandas.lib.is_float}
2007 0.001 0.000 0.001 0.000 {pandas.lib.is_integer}
3 0.006 0.002 0.006 0.002 {pandas.lib.is_lexsorted}
1 0.000 0.000 0.000 0.000 {pandas.lib.is_timedelta_array}
1 0.000 0.000 0.000 0.000 {pandas.lib.isnullobj}
1516 0.008 0.000 0.008 0.000 {pandas.lib.isscalar}
4 0.000 0.000 0.000 0.000 {pandas.lib.list_to_object_array}
1 0.000 0.000 0.003 0.003 {pandas.lib.map_infer}
2 0.001 0.000 0.001 0.000 {pandas.lib.maybe_convert_objects}
1 0.073 0.073 0.073 0.073 {pandas.lib.tuples_to_object_array}
2023 0.004 0.000 0.004 0.000 {pandas.lib.values_from_object}
2 0.801 0.400 0.801 0.400 {pandas.tslib.array_to_datetime}
1 0.000 0.000 0.000 0.000 {pandas.tslib.build_field_sarray}
1 0.000 0.000 0.000 0.000 {pandas.tslib.is_timestamp_array}
1003 0.002 0.000 0.002 0.000 {pandas.tslib.maybe_get_tz}
3 0.000 0.000 0.000 0.000 {pandas.tslib.unique_deltas}
8 0.000 0.000 0.000 0.000 {range}
2 0.000 0.000 0.000 0.000 {setattr}
1002 0.004 0.000 0.013 0.000 {sorted}
6 0.000 0.000 0.000 0.000 {sum}
2 0.000 0.000 0.000 0.000 {time.time}

Process finished with exit code 0

Output from 0.12

/opt/epd/7.3-2_pandas0.12/bin/python2.7 /home/gamantini/QA/avoitenok/pandas_migration/testDatesPerformance.py
500 0.138274908066
91488 function calls (90977 primitive calls) in 2.079 seconds

Ordered by: standard name

ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 2.079 2.079 :1()
4 0.000 0.000 0.000 0.000 _methods.py:20(_prod)
11 0.000 0.000 0.011 0.001 _methods.py:24(_any)
1 0.000 0.000 0.000 0.000 _methods.py:28(_all)
2 0.008 0.004 0.309 0.154 algorithms.py:105(factorize)
2 0.000 0.000 0.000 0.000 algorithms.py:298(_get_data_algo)
1 0.000 0.000 0.000 0.000 arraysetops.py:281(in1d)
1 0.000 0.000 0.000 0.000 arraysetops.py:388(setdiff1d)
2 0.000 0.000 0.000 0.000 arraysetops.py:90(unique)
1000 0.000 0.000 0.000 0.000 base.py:45(_constructor)
2 0.000 0.000 0.000 0.000 categorical.py:111(_set_levels)
2 0.000 0.000 0.000 0.000 categorical.py:119(_get_levels)
2 0.000 0.000 0.000 0.000 categorical.py:71(init)
2 0.001 0.000 0.310 0.155 categorical.py:87(from_array)
2 0.000 0.000 0.001 0.000 common.py:1138(_possibly_convert_platform)
4 0.000 0.000 0.000 0.000 common.py:1167(_possibly_cast_to_datetime)
1504 0.001 0.000 0.003 0.000 common.py:1232(_is_bool_indexer)
2 0.000 0.000 0.000 0.000 common.py:1292(_try_sort)
3 0.000 0.000 0.000 0.000 common.py:1369(split_ranges)
16 0.000 0.000 0.000 0.000 common.py:1468(_asarray_tuplesafe)
1 0.000 0.000 0.000 0.000 common.py:1496(_index_labels_to_array)
3 0.000 0.000 0.000 0.000 common.py:1511(_maybe_make_list)
2 0.000 0.000 0.000 0.000 common.py:1517(is_bool)
1502 0.001 0.000 0.001 0.000 common.py:1521(is_integer)
5 0.000 0.000 0.000 0.000 common.py:1525(is_float)
2 0.000 0.000 0.000 0.000 common.py:1542(is_integer_dtype)
4 0.000 0.000 0.000 0.000 common.py:1552(_is_int_or_datetime_dtype)
4 0.000 0.000 0.000 0.000 common.py:1561(is_datetime64_dtype)
2 0.000 0.000 0.000 0.000 common.py:1583(is_float_dtype)
4 0.000 0.000 0.000 0.000 common.py:1611(is_list_like)
8 0.000 0.000 0.000 0.000 common.py:431(_get_take_nd_function)
8 0.013 0.002 0.204 0.025 common.py:462(take_nd)
2 0.000 0.000 0.000 0.000 common.py:677(_infer_dtype_from_scalar)
5 0.000 0.000 0.000 0.000 common.py:728(_maybe_promote)
1 0.000 0.000 0.000 0.000 frame.py:1722(as_matrix)
504 0.001 0.000 0.039 0.000 frame.py:1986(getitem)
499 0.000 0.000 0.033 0.000 frame.py:2008(_getitem_slice)
499 0.002 0.000 0.033 0.000 frame.py:2063(_slice)
5 0.000 0.000 0.000 0.000 frame.py:2075(_box_item_values)
1501 0.004 0.000 0.007 0.000 frame.py:2090(setattr)
499 0.004 0.000 0.127 0.000 frame.py:2264(xs)
1 0.000 0.000 0.001 0.001 frame.py:2573(reindex)
1 0.000 0.000 0.000 0.000 frame.py:2709(_reindex_columns)
1 0.000 0.000 0.000 0.000 frame.py:2717(_reindex_with_indexers)
1 0.000 0.000 0.342 0.342 frame.py:2771(set_index)
1 0.000 0.000 0.053 0.053 frame.py:2963(take)
1 0.024 0.024 1.380 1.380 frame.py:3207(sort_index)
505 0.001 0.000 0.006 0.000 frame.py:386(init)
502 0.000 0.000 0.000 0.000 frame.py:466(_init_mgr)
2 0.000 0.000 0.003 0.001 frame.py:480(_init_dict)
1 0.000 0.000 0.000 0.000 frame.py:530(_init_ndarray)
2 0.000 0.000 0.002 0.001 frame.py:5663(_arrays_to_mgr)
2 0.000 0.000 0.000 0.000 frame.py:5682(extract_index)
1 0.000 0.000 0.000 0.000 frame.py:5734(_prep_ndarray)
1 0.000 0.000 0.000 0.000 frame.py:583(axes)
2 0.000 0.000 0.001 0.001 frame.py:5908(_homogenize)
1 0.000 0.000 0.000 0.000 frame.py:795(len)
2 0.000 0.000 0.000 0.000 frame.py:799(contains)
4 0.000 0.000 0.001 0.000 fromnumeric.py:2022(prod)
3 0.000 0.000 0.000 0.000 function_base.py:3384(delete)
499 0.001 0.000 0.002 0.000 function_base.py:35(iterable)
499 0.000 0.000 0.001 0.000 generic.py:102(_indexer)
1 0.000 0.000 0.001 0.001 generic.py:380(drop)
505 0.001 0.000 0.001 0.000 generic.py:605(init)
1 0.000 0.000 0.000 0.000 generic.py:651(ndim)
500 0.001 0.000 0.003 0.000 generic.py:655(_set_axis)
500 0.001 0.000 0.001 0.000 generic.py:66(_get_axis_number)
5 0.000 0.000 0.000 0.000 generic.py:662(_get_item_cache)
500 0.000 0.000 0.000 0.000 generic.py:675(_clear_item_cache)
2 0.000 0.000 0.001 0.000 generic.py:682(delitem)
1000 0.002 0.000 0.007 0.000 generic.py:750(_consolidate_inplace)
1000 0.001 0.000 0.002 0.000 generic.py:751()
1 0.000 0.000 0.000 0.000 generic.py:778(_is_mixed_type)
1001 0.001 0.000 0.002 0.000 generic.py:78(_get_axis_name)
1 0.000 0.000 0.000 0.000 generic.py:780()
1001 0.001 0.000 0.004 0.000 generic.py:788(_protect_consolidate)
1000 0.001 0.000 0.003 0.000 generic.py:90(_get_axis)
1 0.000 0.000 0.031 0.031 generic.py:950(copy)
3 0.036 0.012 0.041 0.014 groupby.py:2427(get_group_index)
1 0.000 0.000 0.000 0.000 groupby.py:2450(_int64_overflow_possible)
1 0.001 0.001 0.781 0.781 groupby.py:2473(_indexer_from_factorized)
1 0.016 0.016 1.302 1.302 groupby.py:2497(_lexsort_indexer)
1 0.000 0.000 0.658 0.658 groupby.py:2567(_compress_group_index)
1 0.004 0.004 0.337 0.337 groupby.py:2587(_reorder_by_uniques)
3 0.000 0.000 0.000 0.000 index.py:1261(delete)
1 0.000 0.000 0.000 0.000 index.py:1291(drop)
503 0.002 0.000 0.005 0.000 index.py:1341(new)
520 0.000 0.000 0.001 0.000 index.py:135(array_finalize)
5 0.000 0.000 0.000 0.000 index.py:1382(is_all_dates)
2 0.000 0.000 0.000 0.000 index.py:1430(new)
1000 0.001 0.000 0.001 0.000 index.py:1473(array_finalize)
502 0.000 0.000 0.000 0.000 index.py:1509(len)
1 0.000 0.000 0.000 0.000 index.py:1519(inferred_type)
1497 0.006 0.000 0.010 0.000 index.py:1533(_get_level_number)
1 0.000 0.000 0.007 0.007 index.py:1730(lexsort_depth)
1 0.000 0.000 0.310 0.310 index.py:1745(from_arrays)
1 0.000 0.000 0.000 0.000 index.py:180(_assert_can_do_setop)
1500 0.001 0.000 0.001 0.000 index.py:1808(nlevels)
998 0.008 0.000 0.019 0.000 index.py:1843(getitem)
1 0.000 0.000 0.037 0.037 index.py:1872(take)
6 0.000 0.000 0.000 0.000 index.py:189(dtype)
499 0.003 0.000 0.035 0.000 index.py:1957(droplevel)
998 0.001 0.000 0.004 0.000 index.py:1982()
1 0.000 0.000 0.000 0.000 index.py:199(_get_names)
1 0.000 0.000 0.000 0.000 index.py:202(_set_names)
499 0.002 0.000 0.072 0.000 index.py:2287(get_loc_level)
499 0.003 0.000 0.052 0.000 index.py:2299(_drop_levels)
36 0.000 0.000 0.000 0.000 index.py:236(values)
499 0.002 0.000 0.014 0.000 index.py:2385(_get_level_indexer)
2 0.000 0.000 0.000 0.000 index.py:240(is_monotonic)
9 0.000 0.000 0.000 0.000 index.py:247(is_unique)
1 0.000 0.000 0.000 0.000 index.py:266(_cleanup)
1546 0.001 0.000 0.002 0.000 index.py:2665(_ensure_index)
13 0.000 0.000 0.000 0.000 index.py:269(_engine)
14 0.000 0.000 0.000 0.000 index.py:272()
2 0.000 0.000 0.000 0.000 index.py:282(inferred_type)
4 0.000 0.000 0.000 0.000 index.py:293(iter)
24 0.000 0.000 0.000 0.000 index.py:318(contains)
8 0.000 0.000 0.000 0.000 index.py:332(getitem)
2 0.000 0.000 0.000 0.000 index.py:347(append)
2 0.000 0.000 0.000 0.000 index.py:378(_ensure_compat_concat)
500 0.003 0.000 0.025 0.000 index.py:390(take)
2 0.000 0.000 0.000 0.000 index.py:450(equals)
1 0.000 0.000 0.000 0.000 index.py:622(_wrap_union_result)
1 0.000 0.000 0.000 0.000 index.py:626(intersection)
513 0.001 0.000 0.001 0.000 index.py:708(get_loc)
3 0.000 0.000 0.000 0.000 index.py:770(get_indexer)
3 0.000 0.000 0.000 0.000 index.py:847(_possibly_promote)
19 0.000 0.000 0.001 0.000 index.py:86(new)
3 0.000 0.000 0.000 0.000 index.py:860(isin)
3 0.000 0.000 0.000 0.000 index.py:876(_array_values)
3 0.000 0.000 0.000 0.000 index.py:879(_get_method)
1 0.000 0.000 0.000 0.000 index.py:889(reindex)
499 0.001 0.000 0.002 0.000 indexing.py:1077(_is_list_like)
1 0.000 0.000 0.000 0.000 indexing.py:32(init)
499 0.001 0.000 0.128 0.000 indexing.py:51(_get_label)
499 0.001 0.000 0.136 0.000 indexing.py:695(getitem)
499 0.001 0.000 0.135 0.000 indexing.py:771(_getitem_axis)
504 0.002 0.000 0.005 0.000 indexing.py:910(_convert_to_index_sliceable)
499 0.001 0.000 0.003 0.000 indexing.py:938(_is_index_slice)
998 0.001 0.000 0.002 0.000 indexing.py:939(_is_valid_index)
998 0.001 0.000 0.002 0.000 indexing.py:943(_crit)
500 0.000 0.000 0.000 0.000 internals.py:1016(ndim)
9 0.000 0.000 0.000 0.000 internals.py:102(contains)
502 0.001 0.000 0.002 0.000 internals.py:1020(set_axis)
8 0.000 0.000 0.000 0.000 internals.py:105(len)
6 0.000 0.000 0.000 0.000 internals.py:1072(_set_ref_locs)
553 0.000 0.000 0.000 0.000 internals.py:1177(_get_items)
5 0.000 0.000 0.000 0.000 internals.py:1227(shape)
15 0.000 0.000 0.000 0.000 internals.py:1229()
5 0.000 0.000 0.000 0.000 internals.py:1231(_verify_integrity)
12 0.000 0.000 0.000 0.000 internals.py:1233()
513 0.000 0.000 0.000 0.000 internals.py:128(dtype)
2 0.000 0.000 0.031 0.016 internals.py:132(copy)
1505 0.001 0.000 0.001 0.000 internals.py:1350(is_consolidated)
506 0.002 0.000 0.002 0.000 internals.py:1358(_consolidate_check)
1 0.000 0.000 0.000 0.000 internals.py:1363(is_mixed_type)
499 0.002 0.000 0.024 0.000 internals.py:1431(get_slice)
499 0.004 0.000 0.007 0.000 internals.py:1459(_slice_blocks)
1 0.000 0.000 0.046 0.046 internals.py:147(reindex_axis)
1 0.000 0.000 0.031 0.031 internals.py:1487(copy)
1 0.000 0.000 0.000 0.000 internals.py:1505(as_matrix)
1000 0.001 0.000 0.001 0.000 internals.py:1626(consolidate)
503 0.000 0.000 0.001 0.000 internals.py:1640(_consolidate_inplace)
5 0.000 0.000 0.000 0.000 internals.py:1653(get)
2 0.000 0.000 0.001 0.000 internals.py:1704(delete)
5 0.000 0.000 0.000 0.000 internals.py:181(get)
2 0.000 0.000 0.000 0.000 internals.py:1816(set_items_norename)
2 0.000 0.000 0.000 0.000 internals.py:1823(_delete_from_all_blocks)
2 0.000 0.000 0.000 0.000 internals.py:1841(_delete_from_block)
7 0.000 0.000 0.000 0.000 internals.py:1934(_find_block)
7 0.000 0.000 0.000 0.000 internals.py:1940(_check_have)
1 0.000 0.000 0.000 0.000 internals.py:1971(reindex_indexer)
1 0.000 0.000 0.000 0.000 internals.py:1987(_reindex_indexer_items)
1 0.000 0.000 0.000 0.000 internals.py:2108(_maybe_rename_join)
1 0.000 0.000 0.000 0.000 internals.py:2219(create_block_manager_from_blocks)
2 0.000 0.000 0.001 0.000 internals.py:2236(create_block_manager_from_arrays)
2 0.000 0.000 0.001 0.000 internals.py:2254(form_blocks)
1 0.000 0.000 0.000 0.000 internals.py:2338(_simple_blockify)
2 0.000 0.000 0.000 0.000 internals.py:2351(_multi_blockify)
3 0.000 0.000 0.000 0.000 internals.py:2355()
3 0.000 0.000 0.000 0.000 internals.py:2368(_stack_arrays)
4 0.000 0.000 0.000 0.000 internals.py:2372(_asarray_compat)
3 0.000 0.000 0.000 0.000 internals.py:2379(_shape_compat)
4 0.000 0.000 0.000 0.000 internals.py:2531(_possibly_convert_to_indexer)
509 0.001 0.000 0.001 0.000 internals.py:37(init)
3 0.000 0.000 0.000 0.000 internals.py:688(init)
509 0.000 0.000 0.000 0.000 internals.py:76(set_ref_locs)
2 0.000 0.000 0.000 0.000 internals.py:83(set_ref_items)
509 0.001 0.000 0.017 0.000 internals.py:932(make_block)
505 0.002 0.000 0.005 0.000 internals.py:989(init)
2 0.000 0.000 0.000 0.000 merge.py:1195(_concat_indexes)
1 0.000 0.000 0.000 0.000 merge.py:1299(_should_fill)
2 0.000 0.000 0.000 0.000 merge.py:1305(_any)
1 0.000 0.000 0.001 0.001 merge.py:157(init)
1 0.000 0.000 0.197 0.197 merge.py:182(get_result)
1 0.004 0.004 0.017 0.017 merge.py:201(_maybe_add_join_keys)
1 0.000 0.000 0.083 0.083 merge.py:244(_get_join_info)
1 0.000 0.000 0.000 0.000 merge.py:274(_get_merge_data)
1 0.001 0.001 0.199 0.199 merge.py:28(merge)
1 0.000 0.000 0.001 0.001 merge.py:284(_get_merge_keys)
1 0.000 0.000 0.000 0.000 merge.py:307()
1 0.000 0.000 0.000 0.000 merge.py:308()
1 0.000 0.000 0.000 0.000 merge.py:371(_validate_specification)
1 0.000 0.000 0.077 0.077 merge.py:420(_get_join_indexers)
2 0.000 0.000 0.000 0.000 merge.py:578(_factorize_keys)
1 0.000 0.000 0.000 0.000 merge.py:641(init)
1 0.000 0.000 0.014 0.014 merge.py:664(_prepare_blocks)
1 0.000 0.000 0.098 0.098 merge.py:676(get_result)
6 0.000 0.000 0.000 0.000 merge.py:692()
2 0.000 0.000 0.083 0.042 merge.py:698(_get_merged_block)
1 0.000 0.000 0.037 0.037 merge.py:706(_merge_blocks)
3 0.000 0.000 0.000 0.000 merge.py:717()
2 0.000 0.000 0.000 0.000 merge.py:740(init)
2 0.009 0.005 0.014 0.007 merge.py:744(mask_info)
2 0.000 0.000 0.014 0.007 merge.py:753(get_upcasted_blocks)
1 0.000 0.000 0.046 0.046 merge.py:759(reindex_block)
2 0.000 0.000 0.000 0.000 merge.py:769(_may_need_upcasting)
1 0.000 0.000 0.000 0.000 merge.py:803(_get_merge_block_kinds)
1 0.000 0.000 0.000 0.000 merge.py:810(_get_block_dtype)
1507 0.002 0.000 0.003 0.000 numeric.py:1666(isscalar)
2 0.000 0.000 0.000 0.000 numeric.py:2080(array_equal)
75 0.000 0.000 0.000 0.000 numeric.py:252(asarray)
4 0.000 0.000 0.000 0.000 series.py:1297(values)
4 0.000 0.000 0.001 0.000 series.py:3299(_sanitize_array)
2 0.000 0.000 0.000 0.000 series.py:3310(_try_cast)
5 0.000 0.000 0.000 0.000 series.py:501(from_array)
5 0.000 0.000 0.000 0.000 series.py:532(array_finalize)
1 0.018 0.018 2.079 2.079 testDatesPerformance.py:6(testPerformance)
4 0.000 0.000 0.000 0.000 {any}
2517 0.001 0.000 0.001 0.000 {getattr}
11 0.000 0.000 0.000 0.000 {hasattr}
24 0.000 0.000 0.000 0.000 {hash}
19344 0.006 0.000 0.006 0.000 {isinstance}
1159 0.000 0.000 0.000 0.000 {issubclass}
503 0.000 0.000 0.000 0.000 {iter}
8704/8193 0.002 0.000 0.002 0.000 {len}
2 0.000 0.000 0.000 0.000 {max}
1 0.000 0.000 0.000 0.000 {method 'all' of 'numpy.ndarray' objects}
11 0.000 0.000 0.011 0.001 {method 'any' of 'numpy.ndarray' objects}
539 0.000 0.000 0.000 0.000 {method 'append' of 'list' objects}
4 0.148 0.037 0.148 0.037 {method 'argsort' of 'numpy.ndarray' objects}
2 0.064 0.032 0.064 0.032 {method 'astype' of 'numpy.ndarray' objects}
500 0.000 0.000 0.000 0.000 {method 'clear' of 'dict' objects}
1 0.000 0.000 0.000 0.000 {method 'clear_mapping' of 'pandas.index.IndexEngine' objects}
3 0.031 0.010 0.031 0.010 {method 'copy' of 'numpy.ndarray' objects}
1497 0.001 0.000 0.001 0.000 {method 'count' of 'list' objects}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
6 0.000 0.000 0.000 0.000 {method 'extend' of 'list' objects}
2 0.440 0.220 0.440 0.220 {method 'factorize' of 'pandas.hashtable.Factorizer' objects}
4 0.000 0.000 0.000 0.000 {method 'factorize' of 'pandas.hashtable.Int64Factorizer' objects}
2 0.000 0.000 0.000 0.000 {method 'fill' of 'numpy.ndarray' objects}
2 0.000 0.000 0.000 0.000 {method 'flatten' of 'numpy.ndarray' objects}
1512 0.000 0.000 0.000 0.000 {method 'get' of 'dict' objects}
2 0.000 0.000 0.000 0.000 {method 'get_count' of 'pandas.hashtable.Int64Factorizer' objects}
3 0.000 0.000 0.000 0.000 {method 'get_indexer' of 'pandas.index.IndexEngine' objects}
1 0.062 0.062 0.062 0.062 {method 'get_labels' of 'pandas.hashtable.Int64HashTable' objects}
1 0.190 0.190 0.190 0.190 {method 'get_labels' of 'pandas.hashtable.PyObjectHashTable' objects}
1 0.320 0.320 0.320 0.320 {method 'get_labels_groupby' of 'pandas.hashtable.Int64HashTable' objects}
513 0.001 0.000 0.001 0.000 {method 'get_loc' of 'pandas.index.IndexEngine' objects}
1497 0.002 0.000 0.002 0.000 {method 'index' of 'list' objects}
4 0.000 0.000 0.000 0.000 {method 'keys' of 'dict' objects}
2 0.013 0.006 0.013 0.006 {method 'nonzero' of 'numpy.ndarray' objects}
1501 0.001 0.000 0.001 0.000 {method 'pop' of 'list' objects}
3 0.139 0.046 0.139 0.046 {method 'put' of 'numpy.ndarray' objects}
4 0.000 0.000 0.000 0.000 {method 'ravel' of 'numpy.ndarray' objects}
16 0.011 0.001 0.011 0.001 {method 'reduce' of 'numpy.ufunc' objects}
998 0.004 0.000 0.004 0.000 {method 'searchsorted' of 'numpy.ndarray' objects}
3 0.000 0.000 0.000 0.000 {method 'setdefault' of 'dict' objects}
2 0.000 0.000 0.000 0.000 {method 'sort' of 'numpy.ndarray' objects}
2 0.000 0.000 0.000 0.000 {method 'startswith' of 'str' objects}
506 0.097 0.000 0.097 0.000 {method 'take' of 'numpy.ndarray' objects}
1 0.000 0.000 0.000 0.000 {method 'to_array' of 'pandas.hashtable.Int64Vector' objects}
1 0.000 0.000 0.000 0.000 {method 'to_array' of 'pandas.hashtable.ObjectVector' objects}
2035 0.003 0.000 0.005 0.000 {method 'view' of 'numpy.ndarray' objects}
1 0.000 0.000 0.000 0.000 {min}
7 0.012 0.002 0.012 0.002 {numpy.core.multiarray.arange}
583 0.001 0.000 0.001 0.000 {numpy.core.multiarray.array}
4 0.000 0.000 0.000 0.000 {numpy.core.multiarray.concatenate}
1017 0.012 0.000 0.012 0.000 {numpy.core.multiarray.empty}
4 0.008 0.002 0.008 0.002 {numpy.core.multiarray.putmask}
4 0.003 0.001 0.003 0.001 {numpy.core.multiarray.zeros}
20 0.000 0.000 0.000 0.000 {pandas.algos.ensure_int64}
1 0.000 0.000 0.000 0.000 {pandas.algos.ensure_object}
507 0.000 0.000 0.000 0.000 {pandas.algos.ensure_platform_int}
1 0.000 0.000 0.000 0.000 {pandas.algos.inner_join_indexer_object}
1 0.077 0.077 0.077 0.077 {pandas.algos.inner_join}
2 0.120 0.060 0.120 0.060 {pandas.algos.take_1d_int64_int64}
1 0.000 0.000 0.000 0.000 {pandas.algos.take_1d_object_object}
2 0.009 0.005 0.009 0.005 {pandas.algos.take_2d_axis0_int64_int64}
2 0.023 0.012 0.023 0.012 {pandas.algos.take_2d_axis1_int64_int64}
1 0.024 0.024 0.024 0.024 {pandas.algos.take_2d_axis1_object_object}
23 0.014 0.001 0.014 0.001 {pandas.lib.infer_dtype}
1 0.007 0.007 0.007 0.007 {pandas.lib.is_lexsorted}
3 0.000 0.000 0.000 0.000 {pandas.lib.ismember}
4 0.000 0.000 0.000 0.000 {pandas.lib.list_to_object_array}
2 0.001 0.000 0.001 0.000 {pandas.lib.maybe_convert_objects}
2 0.000 0.000 0.000 0.000 {pandas.tslib.is_timestamp_array}
502 0.000 0.000 0.000 0.000 {range}
1 0.000 0.000 0.000 0.000 {setattr}
1000 0.002 0.000 0.006 0.000 {sorted}
6 0.000 0.000 0.000 0.000 {sum}
2 0.000 0.000 0.000 0.000 {time.time}
9 0.000 0.000 0.000 0.000 {zip}

Process finished with exit code 0
@jreback
Copy link
Contributor

jreback commented Nov 1, 2016

Well this is completely non-idiomatic code. You shouldl never iterate over something and do a single selection.

simply do this

In [34]: %timeit df.reindex(dates, level='DATE')
10 loops, best of 3: 118 ms per loop

furthermore using date is not supported (though sort of works, but slowly) as it is not a first class type (here they are converted to the correct datetime64[ns] dtype).

In addition please read the slicing documentation on how to properly interact with multi-indexes. Which generally are quite performant when lexsorted.

@jreback jreback closed this as completed Nov 1, 2016
@jreback jreback added Indexing Related to indexing on series/frames, not to indexes themselves Performance Memory or execution speed performance Usage Question labels Nov 1, 2016
@jreback jreback added this to the No action milestone Nov 1, 2016
@jorisvandenbossche
Copy link
Member

It is known that loc indexing has slowed down somewhat over time, but at the same time many bugs have been fixed.

You are always welcome to look into optimizing the loc indexing call. But regarding your application, if the execution time of df.loc[dt] is a problem, you should look into whether the for loop is needed. From your example it is not clear what you want to do with temp in the for loop, and depending on that, it is quite possible that you can eliminate the for loop.

@jorisvandenbossche jorisvandenbossche mentioned this issue Nov 1, 2016
Merged
@wesm
Copy link
Member

wesm commented Nov 1, 2016

See https://mail.python.org/pipermail/pandas-dev/2016-March/000499.html

Microperformance in .loc and .iloc indexing in theory will be vastly improved in pandas 2.0 (see https://github.com/pandas-dev/pandas2). Even though single item selection is not very idiomatic, people will do it anyway, so we attempt to make it faster again.

@jorisvandenbossche
Copy link
Member

@wesm @giulianoamantini See also #14551, one-line change brings loc (for this specific case) almost back to performance of 0.12

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Indexing Related to indexing on series/frames, not to indexes themselves Performance Memory or execution speed performance Usage Question
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@wesm @jreback @jorisvandenbossche @giulianoamantini