Duplicate lines creep into every list eventually: email exports from two sources, keyword lists merged from several tools, log files, product SKUs, guest lists. The fastest fix is to paste the list into Remove Duplicate Lines, which keeps the first occurrence of every line, preserves the original order, and lets you ignore case and extra spaces.
If your data already lives in a spreadsheet, editor or terminal, here is how to do the same thing there — and the details that decide whether two lines really count as "the same".
Decide what counts as a duplicate
Before removing anything, decide on three things:
- Case. Is
Applethe same asapple? For email addresses and most keywords, yes. For code identifiers and passwords, no. - Whitespace. Is
apple(with a trailing space) the same asapple? Almost always yes — trailing spaces are usually accidents. - Order. Do you need to keep the original order (first occurrence wins), or is a sorted result fine?
Most "my duplicates weren't removed" problems come from invisible differences: trailing spaces, non-breaking spaces from web pages, or different line endings. The Whitespace Remover and Invisible Character tools help find those.
Method 1: Online, in one step
- Open Remove Duplicate Lines.
- Paste your list, one item per line.
- Choose Case sensitive or not, keep Trim whitespace on, and optionally remove blank lines.
- Copy the result.
The original order is kept, and the counter shows how many duplicates were removed. Nothing is uploaded — the processing happens in your browser, which matters for customer lists and internal data.
Method 2: Microsoft Excel
Remove Duplicates (changes the data in place):
- Select the column (or the whole table).
- Go to Data → Remove Duplicates.
- Choose which columns to compare and click OK.
Excel's comparison is not case-sensitive and keeps the first occurrence. It does not ignore trailing spaces, so clean the data with =TRIM(A2) first if needed.
UNIQUE function (Excel 365 and Excel 2021+, leaves the original untouched):
=UNIQUE(A2:A500)
To also sort the result: =SORT(UNIQUE(A2:A500)).
Highlight instead of delete: Home → Conditional Formatting → Highlight Cells Rules → Duplicate Values shows duplicates in colour so you can review them first.
Method 3: Google Sheets
- Data → Data cleanup → Remove duplicates works like Excel's command, with an option to say whether the data has a header row.
- Or use a formula:
=UNIQUE(A2:A500), and=SORT(UNIQUE(A2:A500))for a sorted list. - Data → Data cleanup → Trim whitespace removes the leading and trailing spaces that stop duplicates from matching.
Method 4: Text editors
Notepad++ (Windows): Edit → Line Operations → Remove Duplicate Lines removes every repeated line, keeping the first. Remove Consecutive Duplicate Lines only removes repeats that are next to each other, so use it after sorting.
VS Code: open the Command Palette (Ctrl/Cmd + Shift + P) and run Delete Duplicate Lines on a selection, or sort first with Sort Lines Ascending and then remove repeats.
Sublime Text: Edit → Permute Lines → Unique keeps the first occurrence of each line.
Method 5: Command line
Sort and de-duplicate (macOS, Linux, WSL):
sort -u list.txt > unique.txt
Fast and simple, but the output is sorted, not in the original order. Add -f to ignore case: sort -fu list.txt.
Keep the original order with awk:
awk '!seen[$0]++' list.txt > unique.txt
This prints each line the first time it appears and skips it afterwards. For a case-insensitive version: awk '!seen[tolower($0)]++' list.txt.
Count how often each line appears:
sort list.txt | uniq -c | sort -rn
Note that uniq on its own only removes adjacent duplicates, which is why it is almost always used after sort.
PowerShell (Windows):
Get-Content list.txt | Select-Object -Unique | Set-Content unique.txt
Select-Object -Unique is case-sensitive; for a case-insensitive version, use Sort-Object -Unique, which also sorts the output.
Which method should you use?
| Situation | Best option |
|---|---|
| Quick one-off list, keep order | Remove Duplicate Lines |
| Data in a table with several columns | Excel or Google Sheets Remove duplicates |
| Need a live, updating result | =UNIQUE() in Excel 365 or Google Sheets |
| Huge files (millions of lines) | sort -u or awk on the command line |
| Code or config files | Your editor's line operations |
Frequently asked questions
How do I remove duplicates but keep the first one?
The online tool, Excel's Remove Duplicates, Google Sheets, Notepad++ and awk '!seen[$0]++' all keep the first occurrence and drop later copies.
Why are some duplicates not removed?
The lines probably differ invisibly — trailing spaces, non-breaking spaces or different capitalisation. Turn on whitespace trimming and case-insensitive matching.
Can I remove duplicates without sorting?
Yes. Use the online tool, =UNIQUE(), Notepad++'s Remove Duplicate Lines, or awk. Only sort -u and uniq require sorted input.
How do I find duplicates without deleting them?
Use conditional formatting in Excel or Google Sheets, or sort list.txt | uniq -d on the command line to print only the repeated lines.
Final takeaway
Decide whether case and spaces matter, then pick the tool that fits where your data lives. For most everyday lists, pasting into Remove Duplicate Lines is the quickest route — and pair it with Sort Lines or the Alphabetizer if you also want the result in order.