CLN: some code cleanups in pandas/_libs/

[ShaharNaveh]

• closes #xxxx
• tests added / passed
• passes black pandas
• passes git diff upstream/master -u -- "*.py" | flake8 --diff
• whatsnew entry

[ShaharNaveh]

The only "big" changes I have done here, is taking this pattern:

arr = ["a", "b", "c"]

for i in range(len(arr)):
    print(arr[i])

and making it to look like this:

arr = ["a", "b", "c"]

for value in arr:
    print(value)

---

When it comes to python, I can't see why would one use the first pattern, but when it comes to Cython, I don't know if there's some sort of optimization, that the first pattern gives that the second one don't

[jbrockmendel]

are there no cases where int rounds up?

[ShaharNaveh]

No, in order for int() to round up it needs to be done with math.ceil or with a "hack" like:

number // divider + (number % divider > 0)

---

For example:

number = 42.01
divider = 8

number / divider # 5.25125
number // divider # 5.0
int(number / divider) # 5
number // divider + (number % divider > 0) # 6.0

[jorisvandenbossche]

but when it comes to Cython, I don't know if there's some sort of optimization, that the first pattern gives that the second one don't

Then you will need to show that it gives similar results (eg show some timings of the changed code, or look at the generated C code).

For example, with a simple test (but I don't know if that is fully equivalent with the changes you did, this was just a quick test), I see a big overhead when iterating directly through the values of an array. Eg check

%%cython -a
import cython

import numpy as np
cimport numpy as np


@cython.wraparound(False)
@cython.boundscheck(False)
def sum1():
    cdef:
        np.int64_t[:] arr
        int n
        int val, total
    
    arr = np.random.randint(10, size=10000)
    
    n = len(arr)
    total = 0
    
    for i in range(n):
        val = arr[i]
        total += val
    
    return total

vs

%%cython -a
import cython

import numpy as np
cimport numpy as np


@cython.wraparound(False)
@cython.boundscheck(False)
def sum2():
    cdef:
        np.int64_t[:] arr
        np.int64_t val, total
    
    arr = np.random.randint(10, size=10000)
    
    n = len(arr)
    total = 0
    
    for val in arr:
        total += val
    
    return total

the second version is much slower (and with the annotation in the notebook you clearly see a lot of python interaction). But I am no cython expert, so I might also have done something wrong.

[jbrockmendel]

missing trailing commas in some of these

[jbrockmendel]

+1 for avoiding 1-character variable name

[jbrockmendel]

@MomIsBestFriend i'd advocate reverting the non-easy parts of this and getting the clear improvements in

[ShaharNaveh]

@jbrockmendel NP, I will do it over the weekend as soon as I have time.

[jorisvandenbossche]

Doesn't this need brackets to be done in the correct order?

[ShaharNaveh]

@jorisvandenbossche I have reverted this, as I was doing it wrong before, I haven't looked at the possibility that row can be let's say 12 and col be 2.

Reverted in 208bc03

[jbrockmendel]

if you wanted to remove the parens here, i wouldnt object

[jbrockmendel]

the in fmt can go on the previous line, then parens removed

[jbrockmendel]

@MomIsBestFriend can you rebase and ping on green

[ShaharNaveh]

ping @jbrockmendel

[jbrockmendel]

thanks @MomIsBestFriend