Extracting text from HTML file using Python

 

I'd like to extract the text from an HTML file using Python. I want essentially the same output I would get if I copied the text from a browser and pasted it into notepad.

I'd like something more robust than using regular expressions that may fail on poorly formed HTML. I've seen many people recommend Beautiful Soup, but I've had a few problems using it. For one, it picked up unwanted text, such as JavaScript source. Also, it did not interpret HTML entities. For example, I would expect ' in HTML source to be converted to an apostrophe in text, just as if I'd pasted the browser content into notepad.

Update html2text looks promising. It handles HTML entities correctly and ignores JavaScript. However, it does not exactly produce plain text; it produces markdown that would then have to be turned into plain text. It comes with no examples or documentation, but the code looks clean.

Related questions:

  • Filter out HTML tags and resolve entities in python
  • Convert XML/HTML Entities into Unicode String in Python

    • html2text是一个Python程序，在这方面做得很好。

    NOTE: NTLK no longer supports clean_html function

    Original answer below, and an alternative in the comments sections.

    Use NLTK

    I wasted my 4-5 hours fixing the issues with html2text. Luckily i could encounter NLTK.
    It works magically. 

    import nltk   
from urllib import urlopen

url = "http://news.bbc.co.uk/2/hi/health/2284783.stm"    
html = urlopen(url).read()    
raw = nltk.clean_html(html)  
print(raw)

    The best piece of code I found for extracting text without getting javascript or not wanted things : 

    import urllib
from bs4 import BeautifulSoup

url = "http://news.bbc.co.uk/2/hi/health/2284783.stm"
html = urllib.urlopen(url).read()
soup = BeautifulSoup(html)

# kill all script and style elements
for script in soup(["script", "style"]):
    script.extract()    # rip it out

# get text
text = soup.get_text()

# break into lines and remove leading and trailing space on each
lines = (line.strip() for line in text.splitlines())
# break multi-headlines into a line each
chunks = (phrase.strip() for line in lines for phrase in line.split("  "))
# drop blank lines
text = 'n'.join(chunk for chunk in chunks if chunk)

print(text)

    You just have to install BeautifulSoup before : 

    pip install beautifulsoup4
    链接地址: http://www.djcxy.com/p/76882.html

    上一篇: Python RegEx匹配换行符

    下一篇: 使用Python从HTML文件中提取文本