About the Remove Duplicate Lines
Paste a list with one item per line and Remove Duplicate Lines keeps each line only once. The first time a line appears is the one that stays, so the rest of your list keeps its original order.
It is useful for cleaning email lists, keywords, tags, product codes, log entries, or any column copied from a spreadsheet. You decide whether "Apple" and "apple" count as the same line, whether stray spaces matter and whether blank lines should go.
The tool can also do the opposite and list only the lines that repeat, or only those that appear exactly once. You can sort the result alphabetically or put the most repeated lines first. Everything happens in your browser, so your list never leaves your device.
How to use it
- Add your listPaste lines into the left box, drop a file onto it, or use Open file for .txt and .csv files. Sample loads a short list with a few duplicates.
- Choose which lines to keepRemove duplicates keeps one copy of every line. Show duplicates lists only lines that appear more than once. Lines used once lists only lines that never repeat.
- Pick an orderKeep original order, sort A to Z or Z to A, or show the most repeated lines first.
- Set the matching optionsMatch case, ignore spaces at the start and end, and remove empty lines. All three are on by default.
- Copy or downloadThe status line shows how many duplicates were removed. Copy the result or download it as a .txt file.
Key features
First occurrence kept
Duplicates are removed after the first copy, so the order of your list stays the same unless you choose to sort it.
Three views
Unique lines, only the repeated lines, or only the lines that appear once.
Natural sorting
Numbers inside lines sort by value, so "item 2" comes before "item 10". Sorting ignores case and accents.
Most repeated first
Order the result by how often each line appears, to see the most common entries at the top.
Flexible matching
Treat uppercase and lowercase as the same, ignore leading and trailing spaces, and drop blank lines.
Private by design
There is no upload and no account. Your list is processed on your own device.
Examples
Cleaning a short list
The Sample button loads this list, which has repeats, a different capital letter, a line with leading spaces and a blank line:
apple
banana
Apple
cherry
banana
cherry
date
apple| Settings | Result |
|---|---|
| Default (match case, ignore spaces, remove empty lines) | apple, banana, Apple, cherry, date 5 unique lines, 3 duplicates removed |
| Match case off | apple, banana, cherry, date 4 unique lines, 4 duplicates removed |
| Ignore spaces off | apple, banana, Apple, cherry, " cherry", date 6 unique lines, 2 duplicates removed |
| Show duplicates | apple, banana, cherry 3 lines repeat |
| Lines used once | Apple, date 2 lines appear only once |
The results are shown with commas here to save space. In the tool, each line stays on its own line.
Deduplicating email addresses
ana@example.com
Ben@Example.com
ana@example.com
ben@example.com
cara@example.comWith Match case turned off, you get 3 unique lines, 2 duplicates removed: ana@example.com, Ben@Example.com and cara@example.com. The third line had a trailing space and still counts as a duplicate, because spaces at the start and end are ignored. Ben's address keeps the spelling of its first appearance.
Finding the most common answers
Paste the answers to a survey question, one per line:
Email
Chat
Phone
Phone
Email
Phone
chat
Social mediaWith Most repeated first and Match case off, the result is Phone, Email, Chat, Social media. Phone appears three times. Email and Chat appear twice each, so they keep the order in which they first appeared.
Sorting with numbers
Sort A to Z puts Item 1, item 2 and item 10 in that order, because numbers are compared by value instead of digit by digit. Accented letters sort with their base letter, so émile comes right after Edgar.
Tips
- Turn Match case off for email addresses, domains and tags. Addresses that differ only in capitals reach the same inbox.
- Use Show duplicates before you delete anything. It tells you exactly which entries were repeated, which helps you find the source of the problem.
- Lines used once is a quick way to find the odd ones out, for example a product code that appears in one export but not in the other. Paste both lists one after the other.
- For a column from a spreadsheet, copy the column, paste it here, then copy the result back. Each cell arrives as one line.
- If your lines are separated by commas instead of line breaks, split them first with the Text Separator tool.
Common mistakes and limits
- Expecting near-duplicates to matchOnly identical lines are merged. "Acme Inc." and "Acme Inc" differ by a period and both stay. Clean up punctuation first with Text Replacer if needed.
- Forgetting that whole lines are comparedA line with several items, such as "red, blue, red", is one line. Duplicates inside a line are not removed.
- Turning off Match case for codesSome IDs, passwords and file names on Linux servers are case-sensitive. Keep Match case on when "ABC1" and "abc1" are different things.
- Losing intentional blank linesRemove empty lines is on by default. Turn it off if blank lines separate groups in your list; one blank line is then kept like any other line.
- Expecting counts in the outputMost repeated first orders the lines by count, but the counts themselves are not shown. Show duplicates tells you which lines repeat.