Suggestions for html table parsing

[klonuo]

Hi,

I have lxml, and when trying to parse html with tables through pandas I found some problems:

1. HTMLParser is set to recover=False contrary to lxml's default value. This is unfortunate because parser will fail on any problem in html page, like missing closing tag or similar, which IMHO happens too often to be justifiable. If such strict rules should be considered, I'd suggest parsing html document with default parser values, and then apply this restriction only on table fragments.
2. HTMLParser will almost always deliver wrong content on tables encoded in anything other then ascii. There is no encoding argument in pandas.io.html.read_html() and lxml doesn't do magic unless http-equiv attribute is correctly declared in html document, or encoding argument is passed to HTMLParser.

[cpcloud]

@klonuo Thanks for the comments.

For 2, I think that's a fairly simple fix, and if you want to submit a PR that would be most welcome. Otherwise I can put something up for v0.14.1.

For 1, we chose recover=False because with incorrect HTML lxml will actually drop data and we decided that slower parsing was more acceptable than data loss. The problem with selectively applying restrictions is that if you lose data in the initial parse, there's no way to recover it.

[klonuo]

@cpcloud Thanks for your prompt reply

I think it's best if you could provide fix, as I'm not familiar with pandas coding rules or internals. I just browsed html module, as I was getting errors, while parsing manually with lxml didn't have problems.

As for 1, I reported what I find unfortunate. Again I don't know libxml2 internals, but from what I read recoverer should be pretty advanced feature. IMHO disabling it shouldn't slow the parsing, nor enabling it should drop valuable data, but lets assume you did tests before applying it. Still missing closing paragraph tag wont let me see the table ;)

[cpcloud]

Yep we test everything and that is in fact how I discovered data were being dropped. I compared it with the results of the html5lib parser and the latter handles invalid HTML in a saner way than lxml. It was a conscious data-driven decision to have it behave this way.

[klonuo]

OK, your call. Maybe having it optional, defaulting to False, wont introduce too many arguments, while allowing users an option to parse non strict html instead failing.

[cpcloud]

@klonuo Fair enough, simple tables don't work. Thanks for the report.

[cpcloud]

In [48]: cat blah.html
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>0</th>
      <th>1</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>        Bãｃòｎ ìｐｓｕｍ Ꮷòｌｏｒ ｓíｔ ａｍëｔ</td>
      <td>        Bãｃòｎ ìｐｓｕｍ Ꮷòｌｏｒ ｓíｔ ａｍëｔ</td>
    </tr>
    <tr>
      <th>1</th>
      <td> Bɑɭｌ ｔíｐ ｃɑｐìｃｏｌã ｔｅｍρòｒ ρàｒìàｔｕｒ</td>
      <td> Bɑɭｌ ｔíｐ ｃɑｐìｃｏｌã ｔｅｍρòｒ ρàｒìàｔｕｒ</td>
    </tr>
  </tbody>
</table>
In [49]: pd.read_html('blah.html', index_col=0)
Out[49]:
[                                                   0  \
 0  Bãï½�òï½� ìï½�ï½�ï½�ï½� á�§Ã²ï½�ï½�ï½� ï½�Ã...
 1  B�ɭ� �í� ���ì���ã ��...

                                                    1
 0  Bãï½�òï½� ìï½�ï½�ï½�ï½� á�§Ã²ï½�ï½�ï½� ï½�Ã...
 1  B�ɭ� �í� ���ì���ã ��...  ]