Text cleaning, also called text normalization, turns messy input into consistent text that can be compared, counted, sorted or imported. The same few operations appear in spreadsheets, databases, data pipelines and natural-language processing.

The core operations

Trimming removes whitespace at the start and end of each value or line. It is the single most useful step, because trailing spaces are invisible and very common. (Trim Lines)

Whitespace normalization collapses runs of spaces and tabs, and converts special spaces (non-breaking, thin, ideographic) into ordinary ones. (Remove Extra Spaces)

Line-ending normalization converts CRLF and CR to LF. See line endings explained.

Case folding converts text to a single case, usually lowercase, so that “Apple” and “apple” compare equal. (Lowercase)

Unicode normalization makes characters that look identical also compare identical, for example “é” stored as one code point versus “e” plus a combining accent. NFC is the usual form for storage. See Unicode characters explained.

Deduplication removes repeated values. (Remove Duplicate Lines)

Filtering removes unwanted content: empty lines, punctuation, numbers or markup. (Remove Empty Lines, Remove Punctuation, HTML to Text)

Order matters

Apply transformations that make values equal before steps that compare values:

  1. Convert markup to text and decode entities.
  2. Normalize line endings and whitespace.
  3. Trim.
  4. Fold case (if case is not meaningful).
  5. Deduplicate.
  6. Sort.

Deduplicating before trimming, for example, leaves “apple” and “apple ” as two separate entries.

Destructive versus reversible

Some steps lose information: lowercasing loses proper-noun capitals, removing punctuation turns “3.5” into “35”, and deduplication discards counts. Keep the original, clean a copy, and apply only the steps your purpose needs. Every tool on this site writes to a separate output box for this reason.

Cleaning for humans versus machines

Cleaning prose for people, such as fixing a pasted PDF, usually means restoring structure: joining broken lines and removing double spaces, while keeping case and punctuation. Cleaning for matching and analysis goes further: case folding, punctuation removal and deduplication. Decide which you are doing before you start.