BUG: Impossible creation of array with dtype=string

[Manju080]

closes #61155

• Tests added and passed if fixing a bug or adding a new feature
• All code checks passed.

Hello @rhshadrach ,

I’ve created a fix that raises a ValueError when trying to create a StringArray from a list of lists with inconsistent lengths or non-character elements. This aligns the behavior for both consistent and inconsistent input formats and also tested.

I've would like to hear opinion to raise an error when a list of lists is passed for dtype=StringDtype, to avoid ambiguous behavior. If preferred, we could instead join the inner lists into strings automatically — happy to adjust based on guidance.
Example case : pd.array([["t", "e", "s", "t"], ["w", "o", "r", "d"]], dtype="string")
output : <StringArray> ['test', 'word'] Length: 2, dtype: string

Thanks

[rhshadrach]

Instead of this, can you modify ensure_string_array in pandas._libs.lib.pyx as follows. Instead of

elif not util.is_array(arr):
    arr = np.array(arr, dtype="object")

do

elif not util.is_array(arr):
    # GH#61155: Guarantee a 1-d result when array is a list of lists
    arr = np.empty(len(array), dtype="object")
    arr[:] = array

Will has almost no performance impact.

[Manju080]

Thank you very much for the suggestions, I have made the necessary changes as per the guidance.

[rhshadrach]

Thanks for making the update. The changes in this file should now be reverted.

[rhshadrach]

Also, please add a test for this.

[Manju080]

pre-commit.ci autofix

[rhshadrach]

We use pytest for testing, you'll need to add a test using that format. See here:

https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#using-pytest

The general pytest introduction may also be useful:

https://docs.pytest.org/en/7.1.x/getting-started.html

[Manju080]

We use pytest for testing, you'll need to add a test using that format. See here:

https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#using-pytest

The general pytest introduction may also be useful:

https://docs.pytest.org/en/7.1.x/getting-started.html

Thank you for the details, will work on it

[Manju080]

@rhshadrach I’ve been testing the following case in test_lib.py

def test_ensure_string_array_list_of_lists():
# GH#61155: ensure list of lists doesn't get converted to string
arr = [['t', 'e', 's', 't'], ['w', 'o', 'r', 'd']]
result = lib.ensure_string_array(arr)

# Each item in result should still be a list, not a stringified version
assert isinstance(result[0], list)
assert isinstance(result[1], list)
assert result[0] == ['t', 'e', 's', 't']
assert result[1] == ['w', 'o', 'r', 'd']

However, the test fails with
FAILED pandas/tests/libs/test_lib.py::test_ensure_string_array_list_of_lists - AssertionError
DEBUG RESULT: ["['t', 'e', 's', 't']" "['w', 'o', 'r', 'd']"] <class 'numpy.ndarray'> <class 'str'>

So currently, the list of lists gets converted into a 1D NumPy array of strings.
With the current implementation, arr becomes a 1D object array of lists (as intended), but it seems that downstream processing stringifies each list.
Do you want me to guard against this case inside ensure_string_array to preserve the list structure? Or is the stringification expected behavior in this context?

Thanks!