What is a Byte Order Mark (BOM)?

A BOM is a special sequence of bytes at the very beginning of a text file (such as EF BB BF for UTF-8 or FF FE for UTF-16LE) that signals the character encoding and endianness to the software reading the file.

What is Mojibake and how do I fix it?

Mojibake refers to corrupted or scrambled text (like Ã© instead of é) caused when a text file written in one encoding is decoded using a different encoding. Converting the file back to UTF-8 resolves it.

Character Encodings Explained: UTF-8, UTF-16, ASCII, and Windows-1252

What is Character Encoding?

Computers do not store letters, emojis, or hieroglyphs; they only store binary zeros and ones (0 and 1). A character encoding is a standardized key table that translates integer numbers (known as Code Points) into sequences of bytes.

The Evolution of Character Sets

1. ASCII (1963)

- 7 bits per character (Range: 0x00 to 0x7F). - Total 128 characters. - Sufficient only for American English.

2. Windows-1252 / ANSI

- 8 bits per character (Range: 0x00 to 0xFF). - Included accented vowels (é, à, ü) and typographical symbols (£, €, ©). - Flaw: Incompatible with Cyrillic, Greek, Arabic, and Asian scripts.

3. Unicode & UTF-8 (1993 - Present)

- Universal character repertoire exceeding 1,114,112 code point slots. - UTF-8: Variable length from 1 to 4 bytes. - ASCII characters (A-Z) = 1 byte. - Greek, Arabic, Latin Extended = 2 bytes. - CJK (Chinese, Japanese, Korean) = 3 bytes. - Emojis and historical scripts = 4 bytes. - Over 98% of the modern internet is powered by UTF-8.