Computers store numbers, not letters. A character encoding is the rule that maps characters to bytes and back. If text is written with one encoding and read with another, letters turn into gibberish.

A short history

ASCII (1960s) defined 128 characters in 7 bits: English letters, digits, punctuation and control codes. It had no room for “é”, “ß” or “ж”.

8-bit code pages used the remaining 128 values for regional characters: ISO 8859-1 (Latin-1) and Windows-1252 for Western Europe, others for Cyrillic, Greek and more. A byte meant different things depending on the code page, so exchanging text between regions was unreliable.

Unicode gave every character in every writing system a single number (a code point). UTF-8, one way of storing those numbers, uses 1 byte for ASCII characters and 2 to 4 bytes for everything else. Because it is backward-compatible with ASCII and handles every language, it became the dominant encoding on the web and in modern software.

Mojibake: how text gets garbled

The classic symptom is accented letters turning into pairs of odd characters. The word “café” in UTF-8 stores “é” as two bytes, C3 A9. If software reads those bytes as Windows-1252, it shows two characters: “é”, so the word appears as “café”.

Other symptoms:

  • � (replacement character): bytes that aren’t valid in the expected encoding.
  • Question marks replacing letters: the text was converted to an encoding that lacks those characters.
  • “’” where an apostrophe should be: a UTF-8 curly apostrophe (’) read as Windows-1252.

Avoiding encoding problems

  • Save text files as UTF-8. Modern editors default to it; older Windows programs sometimes don’t.
  • Declare the encoding in web pages with <meta charset="utf-8">, and in HTTP headers.
  • Check CSV imports. Spreadsheet programs may assume a regional encoding when opening CSV files. Use the import dialog and choose UTF-8.
  • Don’t convert twice. Re-saving already-garbled text “fixes” nothing and can make it permanent.

Byte order marks

Some programs add a byte order mark (BOM), the bytes EF BB BF, at the start of UTF-8 files. It is invisible in most editors but can appear as stray characters in scripts, cause the first CSV column name to fail to match, or break some parsers. UTF-8 without a BOM is generally the safer choice.

Encoding versus escaping

Encodings map characters to bytes. Escaping schemes, such as URL percent-encoding and HTML entities, represent special characters using safe characters inside a particular format. Both are called “encoding” in everyday speech, but they solve different problems.