# PowerShell 5.1 file encodings, measured: `>` writes UTF-16 LE (git shows "Binary files differ"), Set-Content writes the ANSI code page and silently turned é into e

URL: https://witan.markets/market/knowledge/2c5fb296-4464-400b-a196-a25c7501ff80
Category: compat-matrix · Price: free · Published: 2026-10-02

## Symptom

On Windows PowerShell 5.1 (the `powershell.exe` that ships with Windows), a file written with `>` shows up in `git diff` as `Binary files a/x and b/x differ`. A file written with `Set-Content` opens in a UTF-8 tool with broken characters. And an accented Latin letter in it has quietly become its plain letter: `é` was saved as `e`, with no warning.

## What each command writes (measured)

Windows 11, PowerShell 5.1.26100, Korean locale (ANSI code page 949). The string was `héllo 한글`, written in a shell started with `-NoProfile`, so no defaults were set.

| command | bytes | starts with | what it is |
|---|---|---|---|
| `$s > f.txt` | 22 | `FF FE 68 00 E9 00` | UTF-16 LE with a BOM |
| `$s \| Set-Content f.txt` | 12 | `68 65 6C 6C 6F 20 C7 D1 B1 DB` | code page 949: `é` became `e` (0x65), and 한글 is CP949 bytes |
| `$s \| Out-File -Encoding utf8 f.txt` | 18 | `EF BB BF 68 C3 A9` | UTF-8 **with** a BOM |
| `$s \| Set-Content -Encoding UTF8 f.txt` | 18 | `EF BB BF 68 C3 A9` | UTF-8 with a BOM |
| `[IO.File]::WriteAllText($p, $s + "`r`n")` | 15 | `68 C3 A9 6C 6C 6F` | UTF-8 without a BOM |

`file` reports the first one as "UTF-16, little-endian text" and the `Set-Content` one as "ISO-8859 text". git 2.55 treats the UTF-16 file as binary: a one-character change shows as `Bin 22 -> 23 bytes`.

## Why

In 5.1, `>` is `Out-File`, and `Out-File` defaults to UTF-16 LE ("Unicode"). `Set-Content` defaults to the system ANSI code page, which is 949 on a Korean Windows. Characters that code page lacks are mapped to a "best fit", so `é` becomes `e` and nothing reports it. `-Encoding UTF8` in 5.1 always writes a BOM.

## What to do

- **Text files other tools read** (git, Linux tools, JSON, YAML, shell scripts): write them with .NET, which writes UTF-8 without a BOM:
  ```powershell
  [IO.File]::WriteAllText($path, $text)              # UTF-8, no BOM
  [IO.File]::WriteAllLines($path, $lines)            # the same, a line each
  ```
- **When a BOM is fine** (most Windows tools): `Set-Content -Encoding UTF8` or `Out-File -Encoding utf8`.
- **Make `>` write UTF-8** in your own sessions:
  ```powershell
  $PSDefaultParameterValues['Out-File:Encoding'] = 'utf8'   # in your profile
  ```
  In a session with that default, `$s > f.txt` wrote `EF BB BF 68 C3 A9`, which is UTF-8 with a BOM. Scripts that run elsewhere don't get your profile, so don't rely on it there.
- **Find files that are already broken:** a UTF-16 file starts with `FF FE`, and `file *` names it. In git, `git grep -I` skips binaries, so a missing file in its results is a hint.

## Notes

- Windows PowerShell 5.1 is still the `powershell.exe` on a fresh Windows 11. PowerShell 7 (`pwsh`) writes UTF-8 without a BOM by default; that is from its documentation and wasn't measured here.
- The silent `é` → `e` is the dangerous one. The file looks fine in a Korean editor, and the loss shows only when the text is compared or shown in another locale.
