Duplicate Line Remover & Alphabetizer
Deduplicate and optionally alphabetise lines in a text list. Handles exact and fuzzy matches.
Duplicate Line Remover & Alphabetizer
Sort, deduplicate, and clean lists β with a diff of what was removed
How to Use the Duplicate Line Remover
Paste any list with duplicates into the tool and get a clean, deduplicated version instantly. Sort, trim, and control exactly how duplicates are handled.
Enter any text containing duplicate lines into the input panel. The tool processes it in real time as you type or paste.
Toggle between exact match, case-insensitive comparison, or whitespace-trimmed comparison to control how the tool identifies duplicates.
Choose whether to preserve original order, keep only the first or last occurrence, and remove blank lines. Enable line numbering for ranked lists.
Copy the deduplicated output to your clipboard or use the diff panel to review exactly which lines were removed before finalizing.
Related tools
How the Duplicate Line Remover Works
The tool processes text through a multi-stage pipeline that splits, normalizes, filters, and deduplicates lines before optionally sorting the result.
Line Comparison Modes
Exact matching compares each line character-for-character, so "Apple" and "apple" are treated as different lines. Case-insensitive mode lowercases both lines before comparison, making "Apple" and "apple" duplicates while preserving the original casing of the kept line. Trimmed mode strips leading and trailing whitespace before comparing, so indented and non-indented versions of the same text are recognized as duplicates.
Order Preservation
The tool uses a Set data structure for O(1) average-time duplicate detection. When the first-occurrence rule is active, the earliest line is always kept, producing stable, predictable output. When a sort mode is selected, order preservation is relaxed and the sorted result determines which occurrence survives deduplication.
Blank Line Handling
Blank lines are treated as a special category separate from duplicates. When the blank-line filter is enabled, empty lines and whitespace-only lines are stripped before deduplication runs. This prevents multiple blank lines from appearing in the output and ensures the duplicate filter only operates on meaningful content.
Frequently Asked Questions
How are whitespace-only differences handled?
With trim mode enabled, leading and trailing whitespace is stripped before comparison, so " hello " and "hello" are treated as duplicates. The first occurrence is kept in its trimmed form. With trim mode off, whitespace differences make lines unique and both are preserved.
Can I keep a count of how many duplicates were removed?
Yes. The diff panel on the right shows every removed line with a red cross marker and a count multiplier (for example, x3 means three copies were removed). The stats bar above the panels also displays the total number of duplicates and blank lines removed.
Is there a maximum input size?
There is no hard limit. Deduplication runs in O(n) time using a Set, and sorting adds O(n log n). Lists of tens of thousands of lines process in milliseconds. Very large pastes with millions of lines may briefly pause the browser tab but will not crash or lose data.
What happens to the first occurrence when duplicates are removed?
The first occurrence is always preserved in the output, with its original content and casing intact. Subsequent identical lines are removed. This applies regardless of sort mode, ensuring the output is deterministic and auditable.
Can I use the tool for CSV or JSON data?
The tool operates line-by-line and does not parse structured formats. For CSV data, split values into one-per-line format first using a delimiter tool, process here, then rejoin. For JSON arrays, paste each value on its own line without brackets or commas.
Does case-insensitive deduplication change the output text?
No. Case-insensitive mode only affects duplicate detection. If "Apple" appears first and "apple" appears later, the output keeps "Apple" exactly as it was typed. The output text is never modified beyond optional whitespace trimming.