Duplicate lines creep into email lists, keyword lists, exported data and logs. Removing them is simple, until two lines that look identical refuse to match.

The quick way

  1. Paste your list into Remove Duplicate Lines.
  2. Choose whether “Apple” and “apple” count as the same line.
  3. Copy the result. The tool keeps the first occurrence of each line, in its original position, and reports how many repeats it removed.

Why “identical” lines don’t match

If duplicates survive, the lines are different in a way you can’t see:

  • Spaces at the start or end. Very common in data copied from spreadsheets. The tool ignores these by default. For a clean list, also run Trim Lines.
  • Capitalization. “Paris” and “paris” are different unless you turn off case sensitivity. Alternatively, convert everything with the Lowercase Converter first.
  • Different space characters. A non-breaking space looks like an ordinary space but is a different character. Remove Extra Spaces converts them.
  • Invisible characters. Zero-width spaces and byte-order marks sometimes appear in copied text. If two lines still won’t match, retype one of them.

Keep order or sort?

Removing duplicates while keeping the original order is usually what you want: the first time an item appears often matters, for example in a chronological log. If you want an alphabetical list, remove duplicates first and then use Sort Lines. Sorting first also works and makes remaining near-duplicates easy to spot, because they end up next to each other.

In a spreadsheet

  • Excel: select the column, then Data → Remove Duplicates. Newer versions also offer the UNIQUE() function, which returns distinct values without changing the original.
  • Google Sheets: Data → Data cleanup → Remove duplicates, or =UNIQUE(A:A).

Excel’s Remove Duplicates treats “Apple” and “apple” as the same value, which is different from the default here; check how your own spreadsheet behaves before relying on it.

On the command line

On macOS and Linux, sort -u file.txt removes duplicates but also sorts the lines. To keep the original order, use:

awk '!seen[$0]++' file.txt

This prints each line only the first time it appears, the same approach this site’s tool uses.