用NLTK创建一个新的语料库
我认为我的标题的答案往往是去阅读文件，但我跑过NLTK书，但它没有给出答案。 我对python很陌生。
我有一堆.txt文件，我希望能够使用NLTK为语料库nltk_data提供的语料库nltk_data 。
我试过PlaintextCorpusReader但我无法超越：
>>>import nltk
>>>from nltk.corpus import PlaintextCorpusReader
>>>corpus_root = './'
>>>newcorpus = PlaintextCorpusReader(corpus_root, '.*')
>>>newcorpus.words()
我如何细分newcorpus使用PUNKT句子？ 我尝试过使用punkt函数，但punkt函数无法读取PlaintextCorpusReader类？
你还可以引导我如何将分段数据写入文本文件？
编辑：这个问题有一次赏金，现在有第二次赏金。 查看赏金箱中的文字。
我认为PlaintextCorpusReader已经使用punkt分词器分割输入，至少如果您的输入语言是英语。
PlainTextCorpusReader的构造函数
def __init__(self, root, fileids,
word_tokenizer=WordPunctTokenizer(),
sent_tokenizer=nltk.data.LazyLoader(
'tokenizers/punkt/english.pickle'),
para_block_reader=read_blankline_block,
encoding='utf8'):
你可以给读者一个单词和句子标记nltk.data.LazyLoader('tokenizers/punkt/english.pickle') ，但是对于后者，默认情况下已经是nltk.data.LazyLoader('tokenizers/punkt/english.pickle') 。
对于单个字符串，将使用一个标记器，如下所述（这里解释，请参阅第5节中的punkt标记器）。
>>> import nltk.data
>>> text = """
... Punkt knows that the periods in Mr. Smith and Johann S. Bach
... do not mark sentence boundaries. And sometimes sentences
... can start with non-capitalized words. i is a good variable
... name.
... """
>>> tokenizer = nltk.data.load('tokenizers/punkt/english.pickle')
>>> tokenizer.tokenize(text.strip())
经过多年的研究，它是如何工作的，下面是最新的教程
如何创建一个文本文件目录的NLTK语料库？
主要想法是使用nltk.corpus.reader软件包。 如果您有英文文本文件的目录，最好使用PlaintextCorpusReader 。
如果你有一个如下所示的目录：
newcorpus/
file1.txt
file2.txt
...
只需使用这些代码行，就可以得到一个语料库：
import os
from nltk.corpus.reader.plaintext import PlaintextCorpusReader
corpusdir = 'newcorpus/' # Directory of corpus.
newcorpus = PlaintextCorpusReader(corpusdir, '.*')
注：该PlaintextCorpusReader将使用默认nltk.tokenize.sent_tokenize()和nltk.tokenize.word_tokenize()到你的文本拆分成句子和单词和这些功能都建立英语中，它可能不适用于所有语言。
以下是创建测试文本文件以及如何使用NLTK创建语料库以及如何在不同级别访问语料库的完整代码：
import os
from nltk.corpus.reader.plaintext import PlaintextCorpusReader
# Let's create a corpus with 2 texts in different textfile.
txt1 = """This is a foo bar sentence.nAnd this is the first txtfile in the corpus."""
txt2 = """Are you a foo bar? Yes I am. Possibly, everyone is.n"""
corpus = [txt1,txt2]
# Make new dir for the corpus.
corpusdir = 'newcorpus/'
if not os.path.isdir(corpusdir):
os.mkdir(corpusdir)
# Output the files into the directory.
filename = 0
for text in corpus:
filename+=1
with open(corpusdir+str(filename)+'.txt','w') as fout:
print>>fout, text
# Check that our corpus do exist and the files are correct.
assert os.path.isdir(corpusdir)
for infile, text in zip(sorted(os.listdir(corpusdir)),corpus):
assert open(corpusdir+infile,'r').read().strip() == text.strip()
# Create a new corpus by specifying the parameters
# (1) directory of the new corpus
# (2) the fileids of the corpus
# NOTE: in this case the fileids are simply the filenames.
newcorpus = PlaintextCorpusReader('newcorpus/', '.*')
# Access each file in the corpus.
for infile in sorted(newcorpus.fileids()):
print infile # The fileids of each file.
with newcorpus.open(infile) as fin: # Opens the file.
print fin.read().strip() # Prints the content of the file
print
# Access the plaintext; outputs pure string/basestring.
print newcorpus.raw().strip()
print
# Access paragraphs in the corpus. (list of list of list of strings)
# NOTE: NLTK automatically calls nltk.tokenize.sent_tokenize and
# nltk.tokenize.word_tokenize.
#
# Each element in the outermost list is a paragraph, and
# Each paragraph contains sentence(s), and
# Each sentence contains token(s)
print newcorpus.paras()
print
# To access pargraphs of a specific fileid.
print newcorpus.paras(newcorpus.fileids()[0])
# Access sentences in the corpus. (list of list of strings)
# NOTE: That the texts are flattened into sentences that contains tokens.
print newcorpus.sents()
print
# To access sentences of a specific fileid.
print newcorpus.sents(newcorpus.fileids()[0])
# Access just tokens/words in the corpus. (list of strings)
print newcorpus.words()
# To access tokens of a specific fileid.
print newcorpus.words(newcorpus.fileids()[0])
最后，要阅读文本目录并用另一种语言创建一个NLTK语料库，首先必须确保您有一个Python可调用单词标记化和句子标记化模块，它们接受字符串/碱基字符串输入并生成如下输出：
>>> from nltk.tokenize import sent_tokenize, word_tokenize
>>> txt1 = """This is a foo bar sentence.nAnd this is the first txtfile in the corpus."""
>>> sent_tokenize(txt1)
['This is a foo bar sentence.', 'And this is the first txtfile in the corpus.']
>>> word_tokenize(sent_tokenize(txt1)[0])
['This', 'is', 'a', 'foo', 'bar', 'sentence', '.']
>>> import nltk
>>> from nltk.corpus import PlaintextCorpusReader
>>> corpus_root = './'
>>> newcorpus = PlaintextCorpusReader(corpus_root, '.*')
"""
if the ./ dir contains the file my_corpus.txt, then you
can view say all the words it by doing this
"""
>>> newcorpus.words('my_corpus.txt')