If you suspect that you have some large duplicate files on your Mac, you can find them without any special software. You can use the Finder to search for files and sort them so duplicates are together. You can also use the Terminal to find duplicates with a multi-part command.
Check out 2 Ways To Find Duplicate Files On a Mac at YouTube for closed captioning and more options.
Here’s the code for the Terminal command. This should all be on one single long line.
find . -type f -size +1M -exec cksum {} \; | tee /tmp/filelist.tmp | cut -f 1,2 -d ' ' | sort | uniq -d | grep -hif - /tmp/filelist.tmp | sort -nrk2; rm /tmp/filelist.tmp
Note: I have added a part at the end that is not in the video to delete the temporary file to keep things cleaner.
Hello. Did a copy, paste of the code into "terminal" nothing happened. Apologies for what will, no doubt, turn out to be a dumb question.
Love your content.
Jacques
This is a great tool. Thank you. Is there a way that a shortcut can be madero run this as a shortcut command "duplicates'?
Scott: You should be able to do it as a Shortcut, yes. Try it.
Jacques: did it all appear as one long line? Did you press return after pasting it?
The command as-written works, but was reading several hours on my flash at about 380-MB/s before I killed it (many TB). Swapping out cksum for md5, check sums are now reading at about 555-MB/s (46% faster). Installed md5 as part of "brew install md5sha1sum" - otherwise, work great, thanks Gary.
Is there a limitation of not working on the icloud drive data, or is there a trick to getting the path right. I can get the local documents working but the cloud stoared files seem to be overlooked. possible path issues. Any hint?
Chris: just start in the top level of iCloud Drive and it should work. But I wouldn't try it if you have "Optimize" turned on, as there is no way for the checksum to work right if the file isn't really local.
This is great. I can get a lot of use out if it. But I am getting an error: "grep: -: No such file or directory". I think this refers to the "grep -hif -". What is the purpose of the '-' at the end before the filename? Could that be my issue? I am on macOS 11.6, could that be the case?
Thanks.
Al: Make sure the path after the dash in the grep is correct. That's what the error is telling you.
I too have the same issue as AI above, in that it shows the same error "grep: -: No such file or directory"...I have cut and pasted from above so its not a syntax error...any ideas Gary?? Thanks for all your great videos btw, theyve been so helpful in getting my new MacBook Pro under control!!!
Murray: If the error is the same, then it sounds like the /tmp/filelist.tmp is not being created. Maybe you just don't have any duplicates at all?
Gary,
I get zsh: permission denied: whenever I try the terminal command. Any ideas?
Dan: Maybe try it piece by piece to see where the problem is.
I ran it. Quite an eye opener. Most of my dupilicates were from FCP
3765589312 357053440 ./Movies/FCP Cache/FCP 2021 Cache.fcpcache/Sailing 28-Oct-2021/Render Files...
3765589312 357053440 ./Movies/FCP Cache/FCP 2021 Cache.fcpcache/Sailing 28-Oct-2021/Render Files...
But I got a lot of stuff from: ./Library/Trial/Treatments
I googled that but it looks like Apple is dropping the ball here!
Kay: I would really just run this on your Documents folder, not your Home folder. You don't have to worry about things in the Library or inside of project/library files. I should have stressed that more in the video.
I'm getting the "grep: -: No such file or directory" error. I've double checked the file name spelling AND I've guaranteed there are duplicates.... any ideas??? Thanks!
Taylor : perhaps you aren’t getting any duplicates?
Gary - I'm getting the same problem as Taylor, Murray and Al - "No such file or directory" error. I've also made sure there are duplicates and done it step by step as in your video. There are no errors until the "grep" command. I'm running Catalina 10.15.7 on a MacBook Pro. I am not used to using Unix so would appreciate any more help. In general thanks for all your ideas and suggestions.
Michael: Wait, you are in Catalina? That could explain it. Also, are you using zsh, or bash in terminal? What I'm showing here is using zsh, the default shell in Terminal in Monterey.
I'm running zsh, but my OS is Big Sur...
Taylor: Could be it, no way for me to test to be sure.
Gary - I'm using zsh in terminal and still can't get it to work. It works as far as the end of "uniq-d". I made a file called 888.txt with the output of "uniq-d" and then ran "% cat 888.txt | grep -hif - /tmp/filelist.tmp" and got the missing file error and realised it was looking for a file called"-" so I removed this "-" after the -hif and it ran with no error but also with no output.
Hi Gary. Is there any way to eliminate duplicate photographs? Thanks!
Jim: See https://macmost.com/forum/how-do-i-remove-duplicate-photos-from-imac.html
Attempting to run this tool, I found difficulties because of “operation not permitted”. Changing privacy settings to allow terminal full disk access improved the situation but not completely. Running in documents, “operation not permitted” still appeared for an older machine ‘documents’ residing inside the current machine/user documents. Why would this older machine ‘documents’ not be accessible with full disk access enabled? It does contain a PW restricted folder containing health data.
I was able to run this tool after ‘cleaning up’ the drive to remove seeming remnants of the old drive, PW protected stuff, and downloading all iCloud files. I will next change the size specification to get to duplicates smaller than 1M.
Your UTube guides are greatly appreciated; always clear and succinct.
Very useful thanks. Would be even better if the duplicate file names were saved to a file with 'clickable' locations to access them easily.