Performance of ExtensionArray display in DataFrame/Series

[samgd]

When printing a DataFrame/Series that is backed by an ExtensionArray the values are first converted to a numpy array:

pandas/pandas/io/formats/format.py

Lines 1561 to 1574 in c03ee85

	def _format_strings(self) -> list[str]:
	values = extract_array(self.values, extract_numpy=True)
	formatter = self.formatter
	if formatter is None:
	# error: Item "ndarray" of "Union[Any, Union[ExtensionArray, ndarray]]" has
	# no attribute "_formatter"
	formatter = values._formatter(boxed=True) # type: ignore[union-attr]
	if isinstance(values, Categorical):
	# Categorical is special for now, so that we can preserve tzinfo
	array = values._internal_get_values()
	else:
	array = np.asarray(values)

This is problematic for ExtensionArrays that are very expensive to convert to numpy arrays. i.e. converting an extremely large ExtensionArray to a numpy array to ultimately print ~10 values.

Can this be overcome or worked around? There is a already a special case for categorical here.

[samgd]

Tagging @jreback as you may be able to help given the git blame?

[jbrockmendel]

PR to address this would be welcome

[TomAugspurger]

I believe this is a duplicate of / covered by #26837.

#26837 (comment) has a proposed addition to the ExtensionArray interface to allow the EA to control the conversion from array -> List[str]