How it works
Words are extracted with the standard word rules and normalized before comparison: Unicode is converted to its composed form, curly apostrophes become straight, and, unless you choose case-sensitive mode, everything is lowercased. Each normalized word appears once in the list.
“Unique share” divides unique words by total words. Linguists call this the type-token ratio; higher values suggest more varied vocabulary, but the ratio naturally falls as texts get longer.
Rules this tool follows
- Punctuation at the edges of words is removed; punctuation inside words is kept (don't, e-mail).
- In case-insensitive mode the list is shown in lowercase.
- List order is either the order words first appear or alphabetical.
Example
Input
It’s fine. It's FINE, really.
Output
it's fine really
Common uses
- Building vocabulary or glossary lists from a document.
- Comparing vocabulary variety between texts of similar length.
- Creating a deduplicated tag or keyword list from free text.
Limitations and notes
- Inflected forms (child, children) count as different words.
- Do not compare unique-share percentages between texts of very different lengths.