The most efficient implementation really depends on your requirements.
From what you've written: So, I have a large file containing 3 million lines of words. And I need to see if there is any duplicates., I assume you're only looking to check whether there is a duplicate line.
In such case you don't need to count how many duplicates there are and using the HashSet and the old, good string hashing function might be good enough (or even better).
Here's the example:
boolean hasDuplicate = false;
Set<String> lines = new HashSet<String>();
while ( (line = list.readLine()) != null && !hasDuplicate )
{
if (lines.contains(line)) {
hasDuplicate = true;
}
lines.add(line);
}
if (hasDuplicate){
System.out.print("NOT UNIQUE");
} else {
System.out.print("UNIQUE");
}
list.close();
}