Popularity of text encodings

Use of the main encodings on the web from 2001 to 2012 as recorded by Google,[1] with UTF-8 overtaking all others in 2008 and over 60% of the web in 2012 (since then approaching 100%). The ASCII-only figure includes all web pages that only contain ASCII characters, regardless of the declared header.

UTF-8 has been the most common encoding for the World Wide Web since 2008.[2] As of May 2022, UTF-8 accounts for on average 97.7% of all web pages (and 986 of the top 1,000 highest ranked web pages, the next most popular encoding, ISO-8859-1, is used by 4 of those sites).[3] UTF-8 includes ASCII as a subset; almost no websites declare only ASCII used.[4]

All countries and all of the tracked languages have less than 10% of alternative encodings on the web.

In locales where UTF-8 is used alongside another encoding, the latter is typically more efficient for the associated language. GB 18030, a Chinese Unicode Transformation Format, (effectively[5]) has a 6.7% share of websites in China and territories[6][7][8] and 0.2% share worldwide. Big5 is another popular Chinese encoding with less than 0.1% share world-wide, but is popular in Hong Kong and more that twice as popular in Taiwan where it has 4.2% share.[9] The single-byte Windows-1251 is twice as efficient for the Cyrillic script and is used for 7.4% of Russian websites.[10] E.g. Greek and Hebrew encodings are also twice as efficient, but still those languages have over 98% use of UTF-8.[11][12] South Korea has relatively low UTF-8 use compared to most other countries at 93.7%, with the rest of websites mainly using EUC-KR which is more efficient for Korean text. Japanese language websites have a bit more UTF-8 use, with the legacy Shift JIS and EUC-JP encodings having a combined share of 6.5% for the rest of Japanese websites (the more popular Shift JIS has 0.1% global share).[13][14][1] With the exception of GB 18030 (and UTF-16 and UTF-8), other (legacy) encodings were designed for specific languages, and do not support all Unicode characters. As of May 2022, Breton has the lowest UTF-8 use on the web of any tracked language, with 90% use.[15] Over a third of the languages tracked have 100.0% use of UTF-8 on the web, such as Punjabi, Tagalog, Lao, Marathi, Kannada, Kurdish, Pashto, Javanese, Greenlandic (Kalaallisut) and Iranian languages[16][17] and sign languages.[18]

Local storage on computers has considerably more use of "legacy" single-byte encodings than on the web. Attempts to update to UTF-8 have been blocked by editors that do not display or write UTF-8 unless the first character in a file is a byte order mark, making it impossible for other software to use UTF-8 without being rewritten to ignore the byte order mark on input and add it on output. UTF-16 files are also fairly common on Windows, but not in other systems.[19][20]

In the memory of a computer program, usage of UTF-8 is even lower than local disk files. UTF-16 is very common, particularly in Windows but also in JavaScript, Python,[21] Qt, and many other cross-platform software libraries. Compatibility with the Windows API, that at one point didn't support UTF-8, is a major factor.

Recently it has become clear that the overhead of translating from/to UTF-8 on input and output, and dealing with potential encoding errors in the input UTF-8, vastly overwhelms any savings UTF-16 may offer. So newer software systems are starting to use UTF-8. International Components for Unicode (ICU) has historically used UTF-16, and still does only for Java; while for C/C++ UTF-8 is now supported as the "Default Charset"[22] including the correct handling of "illegal UTF-8".[23] The default string primitive used in newer programing languages, such as Go,[24] Julia, Rust and Swift 5,[25] assume UTF-8 encoding and it's also used for their source code. PyPy is also using UTF-8 for its strings.[26] Microsoft now recommends the use of UTF-8 for applications using the Windows API, while continuing to maintain a legacy "Unicode" (meaning UTF-16) interface.[27]