2 Ways To Find Duplicate Files On a Mac

If you suspect that you have some large duplicate files on your Mac, you can find them without any special software. You can use the Finder to search for files and sort them so duplicates are together. You can also use the Terminal to find duplicates with a multi-part command.



Here’s the code for the Terminal command. This should all be on one single long line.

find . -type f -size +1M -exec cksum {} \; | tee /tmp/filelist.tmp | cut -f 1,2 -d ' ' | sort | uniq -d | grep -hif - /tmp/filelist.tmp | sort -nrk2; rm /tmp/filelist.tmp

Note: I have added a part at the end that is not in the video to delete the temporary file to keep things cleaner.

Comments: 27 Responses to “2 Ways To Find Duplicate Files On a Mac”

    Jacques Huot
    2 months ago

    Hello. Did a copy, paste of the code into "terminal" nothing happened. Apologies for what will, no doubt, turn out to be a dumb question.
    Love your content.
    Jacques

    Scott Palluth
    2 months ago

    This is a great tool. Thank you. Is there a way that a shortcut can be madero run this as a shortcut command "duplicates'?

    2 months ago

    Scott: You should be able to do it as a Shortcut, yes. Try it.

    2 months ago

    Jacques: did it all appear as one long line? Did you press return after pasting it?

    Wes W
    2 months ago

    The command as-written works, but was reading several hours on my flash at about 380-MB/s before I killed it (many TB). Swapping out cksum for md5, check sums are now reading at about 555-MB/s (46% faster). Installed md5 as part of "brew install md5sha1sum" - otherwise, work great, thanks Gary.

    chris pearce
    2 months ago

    Is there a limitation of not working on the icloud drive data, or is there a trick to getting the path right. I can get the local documents working but the cloud stoared files seem to be overlooked. possible path issues. Any hint?

    2 months ago

    Chris: just start in the top level of iCloud Drive and it should work. But I wouldn't try it if you have "Optimize" turned on, as there is no way for the checksum to work right if the file isn't really local.

    Al
    2 months ago

    This is great. I can get a lot of use out if it. But I am getting an error: "grep: -: No such file or directory". I think this refers to the "grep -hif -". What is the purpose of the '-' at the end before the filename? Could that be my issue? I am on macOS 11.6, could that be the case?

    Thanks.

    2 months ago

    Al: Make sure the path after the dash in the grep is correct. That's what the error is telling you.

    Murray Walker
    2 months ago

    I too have the same issue as AI above, in that it shows the same error "grep: -: No such file or directory"...I have cut and pasted from above so its not a syntax error...any ideas Gary?? Thanks for all your great videos btw, theyve been so helpful in getting my new MacBook Pro under control!!!

    1 month ago

    Murray: If the error is the same, then it sounds like the /tmp/filelist.tmp is not being created. Maybe you just don't have any duplicates at all?

    Dan
    1 month ago

    Gary,
    I get zsh: permission denied: whenever I try the terminal command. Any ideas?

    1 month ago

    Dan: Maybe try it piece by piece to see where the problem is.

    Kay Fisher
    1 month ago

    I ran it. Quite an eye opener. Most of my dupilicates were from FCP
    3765589312 357053440 ./Movies/FCP Cache/FCP 2021 Cache.fcpcache/Sailing 28-Oct-2021/Render Files...
    3765589312 357053440 ./Movies/FCP Cache/FCP 2021 Cache.fcpcache/Sailing 28-Oct-2021/Render Files...
    But I got a lot of stuff from: ./Library/Trial/Treatments
    I googled that but it looks like Apple is dropping the ball here!

    1 month ago

    Kay: I would really just run this on your Documents folder, not your Home folder. You don't have to worry about things in the Library or inside of project/library files. I should have stressed that more in the video.

    Taylor Francis
    1 month ago

    I'm getting the "grep: -: No such file or directory" error. I've double checked the file name spelling AND I've guaranteed there are duplicates.... any ideas??? Thanks!

    1 month ago

    Taylor : perhaps you aren’t getting any duplicates?

    Michael Cliff
    1 month ago

    Gary - I'm getting the same problem as Taylor, Murray and Al - "No such file or directory" error. I've also made sure there are duplicates and done it step by step as in your video. There are no errors until the "grep" command. I'm running Catalina 10.15.7 on a MacBook Pro. I am not used to using Unix so would appreciate any more help. In general thanks for all your ideas and suggestions.

    1 month ago

    Michael: Wait, you are in Catalina? That could explain it. Also, are you using zsh, or bash in terminal? What I'm showing here is using zsh, the default shell in Terminal in Monterey.

    Taylor Francis
    1 month ago

    I'm running zsh, but my OS is Big Sur...

    1 month ago

    Taylor: Could be it, no way for me to test to be sure.

    Michael Cliff
    1 month ago

    Gary - I'm using zsh in terminal and still can't get it to work. It works as far as the end of "uniq-d". I made a file called 888.txt with the output of "uniq-d" and then ran "% cat 888.txt | grep -hif - /tmp/filelist.tmp" and got the missing file error and realised it was looking for a file called"-" so I removed this "-" after the -hif and it ran with no error but also with no output.

    Jim Owens
    1 month ago

    Hi Gary. Is there any way to eliminate duplicate photographs? Thanks!

    Charles Holder
    1 month ago

    Attempting to run this tool, I found difficulties because of “operation not permitted”. Changing privacy settings to allow terminal full disk access improved the situation but not completely. Running in documents, “operation not permitted” still appeared for an older machine ‘documents’ residing inside the current machine/user documents. Why would this older machine ‘documents’ not be accessible with full disk access enabled? It does contain a PW restricted folder containing health data.

    Charles Holder
    1 month ago

    I was able to run this tool after ‘cleaning up’ the drive to remove seeming remnants of the old drive, PW protected stuff, and downloading all iCloud files. I will next change the size specification to get to duplicates smaller than 1M.

    Your UTube guides are greatly appreciated; always clear and succinct.

    Grant
    1 month ago

    Very useful thanks. Would be even better if the duplicate file names were saved to a file with 'clickable' locations to access them easily.

Leave a New Comment Related to "2 Ways To Find Duplicate Files On a Mac"

:
:
:
0/500 (500 character limit -- please state your comment succinctly and do not try to get around this limit by posting two comments)