How it works
The same input always returns the same phrases. Each phrase is a slice of the string you passed. StartByte and EndByte are indexes into that string.
- Segment. Unicode sentence and word boundaries (UAX #29) keep the original byte offsets. Known abbreviations, initials, and dotted forms such as "U.S." stay in one sentence.
- Fix tokens. Hyphenated compounds such as "e-mail" stay one token. Elided articles ("l'", "d'", "dell'") and English "'s" split off, with straight or curly apostrophes. German compounds stay single tokens.
- Tag. A small model per language labels each token. Verbs and helper verbs never start, end, or appear inside a phrase. Confident adverbs are treated the same way. A word the tagger rarely calls a noun or a name cannot stand as the head of a phrase.
- Candidates. Contiguous runs of at most three words inside one sentence, counting connectors such as "of", "de", and "für". Punctuation, line breaks, verbs, and adverbs end a run. A connector can sit inside a phrase, never at either end, and never more than two in a row. A single word is kept only when it is a name, capitalized like a name, or a common noun that is rare in general text.
- Score. The score is the equal-weight mean of how often the phrase occurs, how early it appears, how many sentences it spans, how often it looks like a name, how tightly its words stay together, and how uncommon its content words are in a general corpus.
- Select. In rank order, skip a phrase contained in a longer one that ranks at least as high, skip near duplicates, skip anything that overlaps a phrase already chosen, and keep the earliest non-overlapping occurrence. Scores are then divided by the best selected score.
Tagger
The tagger is a left-to-right averaged perceptron. For each word it scores labels from the word, its shape, its neighbours, and the labels it has just given. Frequent words that almost always have one label are looked up instead. The weights were learned from Universal Dependencies treebanks, split with this extractor's own tokenizer.
On each treebank's held-out sentences, verb precision is the share of words labelled as verbs that really are verbs. Verb recall is the share of real verbs that get the verb label.
| Language | Training data | Words correct | Verb precision | Verb recall |
|---|---|---|---|---|
| English | EWT, 209k tokens | 94.0% | 96.6% | 97.4% |
| French | GSD, 346k tokens | 96.8% | 96.7% | 98.1% |
| German | GSD, 260k tokens | 93.9% | 96.7% | 97.4% |
| Spanish | GSD, 376k tokens | 95.6% | 97.9% | 96.9% |
| Italian | ParlaMint, PUD, TWITTIRO, MarkIT, 90k tokens | 94.4% | 95.5% | 96.4% |
Italian uses only openly licensed treebanks, about a quarter as much data as the others, so it is the weakest of the five. A model is decoded the first time its language is used. A language with no model still runs, and it keeps verbs.
Limitations
- You pass the language code on every call.
- The input should already be article text. Menus, footers, and other page chrome can come back as phrases.
- The tagger misses some verbs, and sometimes drops a word that is not a verb.
- Names longer than three words are split.
- A score is a ranking signal for one document. The best phrase in that document scores 1. Scores from two documents are not comparable.
Accuracy is measured on texts that are not in the repository. The tools that score those texts live with the source, and they run on any text you have locally.