2 Ways To Find Duplicate Files On a Mac

If you suspect that you have some large duplicate files on your Mac, you can find them without any special software. You can use the Finder to search for files and sort them so duplicates are together. You can also use the Terminal to find duplicates with a multi-part command.



Here’s the code for the Terminal command. This should all be on one single long line.

find . -type f -size +1M -exec cksum {} \; | tee /tmp/filelist.tmp | cut -f 1,2 -d ' ' | sort | uniq -d | grep -hif - /tmp/filelist.tmp | sort -nrk2; rm /tmp/filelist.tmp

Note: I have added a part at the end that is not in the video to delete the temporary file to keep things cleaner.

Comments: 34 Responses to “2 Ways To Find Duplicate Files On a Mac”

    Jacques Huot
    9 months ago

    Hello. Did a copy, paste of the code into "terminal" nothing happened. Apologies for what will, no doubt, turn out to be a dumb question.
    Love your content.
    Jacques

    Scott Palluth
    9 months ago

    This is a great tool. Thank you. Is there a way that a shortcut can be madero run this as a shortcut command "duplicates'?

    9 months ago

    Scott: You should be able to do it as a Shortcut, yes. Try it.

    9 months ago

    Jacques: did it all appear as one long line? Did you press return after pasting it?

    Wes W
    9 months ago

    The command as-written works, but was reading several hours on my flash at about 380-MB/s before I killed it (many TB). Swapping out cksum for md5, check sums are now reading at about 555-MB/s (46% faster). Installed md5 as part of "brew install md5sha1sum" - otherwise, work great, thanks Gary.

    chris pearce
    9 months ago

    Is there a limitation of not working on the icloud drive data, or is there a trick to getting the path right. I can get the local documents working but the cloud stoared files seem to be overlooked. possible path issues. Any hint?

    9 months ago

    Chris: just start in the top level of iCloud Drive and it should work. But I wouldn't try it if you have "Optimize" turned on, as there is no way for the checksum to work right if the file isn't really local.

    Al
    9 months ago

    This is great. I can get a lot of use out if it. But I am getting an error: "grep: -: No such file or directory". I think this refers to the "grep -hif -". What is the purpose of the '-' at the end before the filename? Could that be my issue? I am on macOS 11.6, could that be the case?

    Thanks.

    9 months ago

    Al: Make sure the path after the dash in the grep is correct. That's what the error is telling you.

    Murray Walker
    9 months ago

    I too have the same issue as AI above, in that it shows the same error "grep: -: No such file or directory"...I have cut and pasted from above so its not a syntax error...any ideas Gary?? Thanks for all your great videos btw, theyve been so helpful in getting my new MacBook Pro under control!!!

    9 months ago

    Murray: If the error is the same, then it sounds like the /tmp/filelist.tmp is not being created. Maybe you just don't have any duplicates at all?

    Dan
    9 months ago

    Gary,
    I get zsh: permission denied: whenever I try the terminal command. Any ideas?

    9 months ago

    Dan: Maybe try it piece by piece to see where the problem is.

    Kay Fisher
    9 months ago

    I ran it. Quite an eye opener. Most of my dupilicates were from FCP
    3765589312 357053440 ./Movies/FCP Cache/FCP 2021 Cache.fcpcache/Sailing 28-Oct-2021/Render Files...
    3765589312 357053440 ./Movies/FCP Cache/FCP 2021 Cache.fcpcache/Sailing 28-Oct-2021/Render Files...
    But I got a lot of stuff from: ./Library/Trial/Treatments
    I googled that but it looks like Apple is dropping the ball here!

    9 months ago

    Kay: I would really just run this on your Documents folder, not your Home folder. You don't have to worry about things in the Library or inside of project/library files. I should have stressed that more in the video.

    Taylor Francis
    9 months ago

    I'm getting the "grep: -: No such file or directory" error. I've double checked the file name spelling AND I've guaranteed there are duplicates.... any ideas??? Thanks!

    9 months ago

    Taylor : perhaps you aren’t getting any duplicates?

    Michael Cliff
    9 months ago

    Gary - I'm getting the same problem as Taylor, Murray and Al - "No such file or directory" error. I've also made sure there are duplicates and done it step by step as in your video. There are no errors until the "grep" command. I'm running Catalina 10.15.7 on a MacBook Pro. I am not used to using Unix so would appreciate any more help. In general thanks for all your ideas and suggestions.

    9 months ago

    Michael: Wait, you are in Catalina? That could explain it. Also, are you using zsh, or bash in terminal? What I'm showing here is using zsh, the default shell in Terminal in Monterey.

    Taylor Francis
    9 months ago

    I'm running zsh, but my OS is Big Sur...

    9 months ago

    Taylor: Could be it, no way for me to test to be sure.

    Michael Cliff
    9 months ago

    Gary - I'm using zsh in terminal and still can't get it to work. It works as far as the end of "uniq-d". I made a file called 888.txt with the output of "uniq-d" and then ran "% cat 888.txt | grep -hif - /tmp/filelist.tmp" and got the missing file error and realised it was looking for a file called"-" so I removed this "-" after the -hif and it ran with no error but also with no output.

    Jim Owens
    9 months ago

    Hi Gary. Is there any way to eliminate duplicate photographs? Thanks!

    Charles Holder
    9 months ago

    Attempting to run this tool, I found difficulties because of “operation not permitted”. Changing privacy settings to allow terminal full disk access improved the situation but not completely. Running in documents, “operation not permitted” still appeared for an older machine ‘documents’ residing inside the current machine/user documents. Why would this older machine ‘documents’ not be accessible with full disk access enabled? It does contain a PW restricted folder containing health data.

    Charles Holder
    9 months ago

    I was able to run this tool after ‘cleaning up’ the drive to remove seeming remnants of the old drive, PW protected stuff, and downloading all iCloud files. I will next change the size specification to get to duplicates smaller than 1M.

    Your UTube guides are greatly appreciated; always clear and succinct.

    Grant
    8 months ago

    Very useful thanks. Would be even better if the duplicate file names were saved to a file with 'clickable' locations to access them easily.

    Steve Urich
    4 months ago

    I am also getting the error message about no file found. Believe me I have duplicates. I run Gemini duplicate finder and it shows plenty. But it does not make it easy to get rid of the files that are not where they belong.

    Any recommended troubleshooting steps?

    I am going to try and run it one step at a time like you did in the video, but if there are many users that are getting the same error then their might be a common problem.

    4 months ago

    Steve: Are you sure you are running the Terminal command in the right place? If that other app finds something in folder A and you are running the shell script in folder B, then you are looking in two different places.

    Rune Schjoenning
    1 month ago

    Hi Gary thanks for excellent instruction. Every line in the script works perfectly until the "grep" instruction.

    find . -type f -size +1M -exec cksum {} \; | tee /tmp/filelist.tmp | cut -f 1,2 -d ' ' | sort | uniq -d | grep -hif - /tmp/filelist.tmp | sort -nrk2
    grep: -: No such file or directory

    The filelist.tmp is generated with duplicate files in it. I run OS High Sierra. Terminal Version 2.8.3 (404.1)

    Thanks again kind regards Rune

    1 month ago

    Rune: Probably because you are running such an old version of macOS.

    Rune Schjoenning
    1 month ago

    Yes probably. I tried a simpler 'grep' (picking out a text from a txt file), first from a dokument folder, then I moved the txt file to the tmp folder. Dragged the folder to terminal, and got a path similar to what I had tried earlier. No trouble picking the text from there either. So the path seems to be working, but maybe a more complex grep command is the problem on my old OS? kind regards Rune

    Sharyn
    1 month ago

    Hi Gary, will this work on Ventura? Great content by the way, thank you.

    1 month ago

    Sharyn: The Terminal command? It should. Try it and see.

Leave a New Comment Related to "2 Ways To Find Duplicate Files On a Mac"

:
:
:
0/500 (500 character limit -- please state your comment succinctly and do not try to get around this limit by posting two comments)