Discovering "templates" in a given text?

If I have significant amounts of text and am trying to discover templates that occur most frequently, I was thinking of solving it using the N-Gram approach and in fact it was suggested as a solution in this question as well but my requirement is slightly different. Just to clarify, I have some text like this:

I wake up every day morning and read the newspaper and then go to work
I wake up every day morning and eat my breakfast and then go to work
I am not sure that this is the solution but I will try
I am not sure that this is the answer but I will try
I am not feeling well today but I will get the work done and deliver it tomorrow
I was not feeling well yesterday but I will get the work done and let you know by tomorrow

and am trying to extract "templates" like this:

I wake up every day morning and ... and then go to work
I am not sure that this is the ... but I will try
I ... not feeling well ... but I will get the work done and ... tomorrow

I am looking for an approach that can scale to million of lines of text so I was just wondering if I can adapt the same N-gram approach to solve this problem or are there any alternatives?

---

Millions of lines of text isn't a really big number :)

What you're looking for is at least similar to collocation finding. You could try to compute pointwise mutual information on n-grams. See Manning & Schütze (1999) for this and other approaches to the problem.

链接地址: http://www.djcxy.com/p/18134.html

上一篇: 如何将给定的文本分解成字典中的单词？

下一篇: 发现给定文本中的“模板”？