How to Remove Duplicates from List in Java

1. Introduction

Removing duplicates from a list in Java is a common and essential task, especially when handling data where duplicate values can affect results or efficiency. In this guide, we’ll explore how to remove duplicate elements from lists in Java with step-by-step examples, covering both basic lists and lists of objects. You’ll learn several ways to remove duplicates, from simple approaches in plain Java to more powerful solutions using Java 8 Streams.

Let’s dive into different ways to remove duplicates from a list in Java so you can choose the most efficient method for your needs.

2. Basic Approach to Remove Duplicates Using Plain Java

A straightforward way to remove duplicates from a list in Java is to use a Set. Since sets do not allow duplicate elements, converting a list to a set and then back to a list removes duplicates automatically. This method works well if you need a simple solution to remove duplicate values from a basic list of elements.

Example: Remove Duplicate Elements from a List Using HashSet

import java.util.ArrayList;
import java.util.HashSet;
import java.util.List;
import java.util.Set;

public class RemoveDuplicates {
    public static void main(String[] args) {
        List<String> names = List.of("Alice", "Bob", "Alice", "Charlie");
        System.out.println("List containing duplicates: " + names);

        Set<String> uniqueNames = new HashSet<>(names); // Duplicates are removed here
        names = new ArrayList<>(uniqueNames); // Convert set back to list if needed

        System.out.println("List after removing duplicates: " + names);
    }
}

Explanation:

• Convert to Set: We convert the list to a HashSet to remove duplicate values.
• Convert Back to List: If you need a list format, convert it back to a list.

Output:

List containing duplicates: [Alice, Bob, Alice, Charlie]
List after removing duplicates: [Alice, Bob, Charlie]

This method works well for simple cases but may not be ideal if you’re dealing with lists of objects.
To preserve the order of elements, consider using a LinkedHashSet.

3. Removing Duplicates Using Java 8 Streams

Java 8 introduced the Stream API, providing a concise and efficient way to remove duplicates from lists. Using the .distinct() method, you can easily filter out duplicate elements in a list while keeping the order intact.

Example: Remove Duplicate Elements from List Using Stream in Java 8

import java.util.List;
import java.util.stream.Collectors;

public class RemoveDuplicates {
    public static void main(String[] args) {
        List<String> names = List.of("Alice", "Bob", "Alice", "Charlie");
        System.out.println("List containing duplicates: " + names);

        List<String> uniqueNames = names.stream()
                .distinct()
                .collect(Collectors.toList());

        System.out.println("List after removing duplicates: " + uniqueNames);
    }
}

Explanation:

• Stream.distinct(): The distinct() method in Java 8 Stream API filters out duplicate elements, making it ideal for removing duplicates from a list.
• Collect to List: We use collect(Collectors.toList()) to gather the results back into a list.

Output:

List containing duplicates: [Alice, Bob, Alice, Charlie]
List after removing duplicates: [Alice, Bob, Charlie]

4. Removing Duplicates Using Guava Library

If you’re using the Guava library or planning to include it in your project, you can choose between Sets.newHashSet() for quick deduplication or Sets.newLinkedHashSet() if you want to preserve the order of elements.

Guava Dependency: To use Guava in your project, add the following dependency to your pom.xml:

<dependency>
  <groupId>com.google.guava</groupId>
  <artifactId>guava</artifactId>
  <version>33.3.1-jre</version>
</dependency>

You can find the latest version of Guava on Maven Central.

Example: Using Guava’s Sets.newHashSet() and Sets.newLinkedHashSet()

import com.google.common.collect.Sets;

// To remove duplicates without preserving order
List<String> uniqueNamesHashSet = new ArrayList<>(Sets.newHashSet(names));

// To remove duplicates while preserving the order
List<String> uniqueNamesLinkedHashSet = new ArrayList<>(Sets.newLinkedHashSet(names));

Explanation:

• Sets.newHashSet(): Removes duplicates by converting the list into a HashSet. This method is fast, but it does not preserve the order of elements, as HashSet does not maintain insertion order.
• Sets.newLinkedHashSet(): Removes duplicates while maintaining the insertion order of elements, as LinkedHashSet preserves the order in which elements were added.

Both methods are useful depending on whether you need to maintain the order of elements. 

5. Removing Duplicate Objects from List

When dealing with lists of custom objects, you might need to remove duplicates based on specific properties of the objects, such as an ID or name. Below are methods to handle these cases.

Defining the Person Class:

For these examples, let’s use a Person class with name and age properties.

class Person {
    private String name;
    private int age;

    // All args constructors
    // Getters and Setters
}

5.1. Remove Duplicate Objects Based on a Single Property

Let’s say we want to remove duplicate Person objects based on the name property.

Example: Remove Duplicate Objects from List by Property in Java 8

import java.util.List;
import java.util.stream.Collectors;

public class RemoveDuplicates {
    public static void main(String[] args) {

        List<Person> people = List.of(
                new Person("Alice", 30),
                new Person("Bob", 25),
                new Person("Alice", 29), // Duplicate based on name
                new Person("Charlie", 25)
        );
        System.out.println("List containing duplicates: " + people);

        // Remove duplicates based on 'name' property
        List<Person> uniquePeople = people.stream()
                .collect(Collectors.toMap(
                        Person::getName,    // Unique key by 'name'
                        person -> person,   // Map value is the person object
                        (existing, replacement) -> existing // Keep first occurrence
                ))
                .values()
                .stream()
                .collect(Collectors.toList());

        System.out.println("List after removing duplicates: " + uniquePeople);
    }
}

Explanation:

• toMap Method: Using Collectors.toMap(), we map each Person by a unique property—in this case, name.
• Conflict Resolution: The third parameter in toMap() (existing, replacement) -> existing keeps the first occurrence and ignores the rest, effectively removing duplicates.
• Convert to List: Finally, we convert the map’s values back to a list.

Output:

List containing duplicates: [Alice - 30, Bob - 25, Alice - 29, Charlie - 25]
List after removing duplicates: [Bob - 25, Alice - 30, Charlie - 25]

5.2. Remove Duplicate Objects Based on Multiple Properties

Sometimes you might need to remove duplicates based on more than one property. For instance, let’s say we want to remove duplicate Person objects based on both name and age.

Example: Remove Duplicate Objects from List by Multiple Properties

import java.util.List;
import java.util.stream.Collectors;

public class RemoveDuplicates {
    public static void main(String[] args) {

        List<Person> people = List.of(
                new Person("Alice", 30),
                new Person("Bob", 25),
                new Person("Alice", 30), // Duplicate based on name and age
                new Person("Charlie", 25)
        );
        System.out.println("List containing duplicates: " + people);

        // Remove duplicates based on both 'name' and 'age' properties
        List<Person> uniquePeople = people.stream()
                .collect(Collectors.toMap(
                        person -> person.getName() + "-" + person.getAge(), // Composite key
                        person -> person,
                        (existing, replacement) -> existing
                ))
                .values()
                .stream()
                .collect(Collectors.toList());

        System.out.println("List after removing duplicates: " + uniquePeople);
    }
}

Explanation:

• Composite Key: By combining name and age into a single string (like "Alice-30"), we create a unique key based on multiple properties.
• toMap for Deduplication: Using toMap() and the composite key, duplicates are removed if both name and age match.
• Conversion to List: We convert the map values back to a list, which now contains unique elements based on the specified properties.

Output:

List containing duplicates: [Alice - 30, Bob - 25, Alice - 30, Charlie - 25]
List after removing duplicates: [Charlie - 25, Alice - 30, Bob - 25]

6. Things to Consider

Here are some important considerations to keep in mind while removing duplicates:

• Handling Null Values: When removing duplicates, ensure that your list doesn’t contain null values or handle them appropriately. Null values can sometimes cause issues depending on the method you use for deduplication.
• Preserving Original Order: If maintaining the original order of elements is important, be mindful of the approach you choose. For example, using a LinkedHashSet or streams with distinct() ensures the order is preserved.
• Performance Implications: The efficiency of your deduplication method can vary. For large lists, using a Set or streams can be faster, but if you’re dealing with complex objects or multiple properties, performance may vary depending on how you handle equality and comparisons.
• Handling Immutable Lists: If you’re working with immutable lists (e.g., List.of()), be careful when removing duplicates, as these lists can’t be modified. You might need to convert them to a mutable list before applying deduplication techniques.

7. FAQs

8. Conclusion

In conclusion, removing duplicates from a list in Java helps keep your data clean and organized. Whether you’re working with basic lists or more complex objects, knowing how to handle duplicates ensures your application runs smoothly and efficiently.

9. Learn More