[ENH] pandas.read_html argument to interpret hyperlinks as links (not merely text)

[zeluspudding]

For starters I'd just like to share that pandas is AWESOME! Thank you to all who contribute and make this lean, mean number crunching machine available to others like me.

pandas.read_html is super convenient for making dataframes from the web! The power of the web is that it's just that... a web of many highly interconnected data chunks. Being able to get the hyperlinks in an HTML table instead of the plain text that is hyperlinked would be a great productivity boost for web scraping.

Take the example of getting links to SEC forms. Here, one is probably more interested in getting the link to an insider trading disclosure form rather then merely the type of form (typically form 4).

Baking this functionality into read_html also averts the need to double parse HTML as in this StackOverflow post - http://stackoverflow.com/questions/31771619/html-table-to-pandas-table-info-inside-html-tags

[nono-london]

Hi,
I am working exactly on the same type of project on SEC, and have same issue.
I was going through pandas doc and wondered is could be done using converter and a bs4 function to get href, or if converters will the the "text that is hyperlinked", which I suppose it will...

[TomAugspurger]

Probably in scope. I don't think anyone is working on this.

It seems like #14608 has the start of a proposed API.

[hindamosh]

Pandas is awesome really , also I have the same suggestion, in a try to scrape a website directly with the href tags with no need to other pacakges and /functions

[adamrossnelson]

Another reference that could be useful in developing this additional feature, option, argument...

https://stackoverflow.com/questions/42285417/how-to-preserve-links-when-scraping-a-table-with-beautiful-soup-and-pandas

[KIC]

The linked SO answer does not work with merged cells like rowspan i.e. https://www.interactivebrokers.com/en/index.php?f=1563&p=fut