DOC: pandas.read_csv

[ryanbaekr]

Pandas version checks

• I have checked that the issue still exists on the latest versions of the docs on main here

Location of the documentation

https://pandas.pydata.org/docs/reference/api/pandas.read_csv.html

Documentation problem

There are certain values that read_csv interprets as booleans even if the true_values and false_values parameters are set to none or an empty list. In my opinion, there are two oversights with the documentation here.

1. There is nothing in the documentation that lists out what string values are interpreted as booleans by default (I found true, True, TRUE, false, False, and FALSE but there could be more, it would be nice to know that)
2. There is no direct way to force read_csv to treat a column of data with values that could all be interpreted as booleans as strings.

Suggested fix for documentation

1. The values that are interpreted as true and false no matter what should be documented.
2. I managed to get around the issue by interpreting all columns as strings and then applying the to_numeric function to each column. If that is the only way to prevent those values from being interpreted as booleans, I think it should be noted on the page. I have an example of what I'm talking about here

[AlexKirko]

Confirmed. Looks like true and false (case-insensitive) are treated as True and False by default.

Should we just add to the docs that the default values for true_values and false_values are None, but any case-insensitive variant of true and false is treated as boolean in addition to that?

[ryanbaekr]

Confirmed. Looks like true and false (case-insensitive) are treated as True and False by default.

Should we just add to the docs that the default values for true_values and false_values are None, but any case-insensitive variant of true and false is treated as boolean in addition to that?

I think that's a good start. I guess I'd like a definitive list, I tried to trace the source code but got to a .so and went no further. There could be more than just true and false that are treated this way for all we know.

I also think the second part of my request is important, documenting how to prevent this behavior

[AlexKirko]

Weird that you got stuck on .so. By default we use the python CSV engine, I think, which is coded here.
Do you mind looking there to check what might be causing this behavior? If not, no sweat, I'll take a deeper look tomorrow and try to figure it out.

[ryanbaekr]

Weird that you got stuck on .so. By default we use the python CSV engine, I think, which is coded here. Do you mind looking there to check what might be causing this behavior? If not, no sweat, I'll take a deeper look tomorrow and try to figure it out.

Not sure if this is right, but I was looking here. Which eventually calls this where I see libops.maybe_convert_bool. Sorry if I'm going down the complete wrong path here but that's what I remember looking at when I first ran into this a while back.

[AlexKirko]

Got it. The code you were looking for is in _libs/ops.pyx and it's writtein in Cython. Here is the link and the relevant bit:

    # the defaults
    true_vals = {'True', 'TRUE', 'true'}
    false_vals = {'False', 'FALSE', 'false'}

    if true_values is not None:
        true_vals = true_vals | set(true_values)

    if false_values is not None:
        false_vals = false_vals | set(false_values)

[AlexKirko]

My experiments also show that any case variant, such as TrUe or FAlse also gets treated as one of these, so there is probably a lower or an upper call somewhere, though I can't find it at the moment.

@ryanbaekr , would you like to make a PR with the Docs modification?

[AlexKirko]

No worries, I'll do it then.