Remove Duplicate Lines

Remove repeated lines from a list or text. The first occurrence of each line is kept in its original position, and the tool reports how many duplicates it removed.

Options

Processed in your browser. Nothing you type is uploaded.

How it works

The tool reads your text line by line and remembers every line it has seen. When a line matches one seen earlier, it is dropped. Order is preserved, so the result is your original list with the repeats removed. It is not sorted unless you sort it afterwards.

By default, spaces at the start and end of a line are ignored when comparing, because stray whitespace is the most common reason “identical” lines fail to match. The line that is kept is left exactly as it was.

Rules this tool follows

  • Case sensitive (on by default): “Apple” and “apple” are different. Turn it off to treat them as duplicates.
  • Ignore surrounding spaces (on by default): “ apple” and “apple ” match.
  • Keep all empty lines (on by default) leaves blank lines in place so paragraph spacing survives. Turn it off to keep only the first blank line.

Example

Input

banana
apple
  cherry
apple
Banana
cherry

Output

banana
apple
  cherry
Banana
“apple” and “cherry” (with the spaces ignored) were removed. “Banana” stays because the comparison is case-sensitive.

Common uses

  • Cleaning email lists, keyword lists and URL lists.
  • Removing repeated rows before pasting into a spreadsheet.
  • Deduplicating log lines or error messages.

Limitations and notes

  • Only whole lines are compared. To find repeated words inside a text, use the Word Frequency Counter.
  • Lines that differ only in invisible characters, such as a zero-width space, are treated as different.

Questions

Does it sort the lines?

No. The original order is kept. Use Sort Lines afterwards if you want an alphabetical list.

Which copy is kept?

The first one. Later copies are removed.

All text cleaners