How To: Compress a Directory of Files into Individual Archives
r3dux | April 16, 2009I’ve got a stack of files all thrown together in the same directory, and I wanted them compressed – simple enough, eh? Only thing is I wanted each file compressed to its own archive, so I can see at a glance what’s there, and if for some reason an archive gets corrupted, it’s just one file lost and I can replace it instead of having to dick around repairing corrupted “blob” archives that contain the entire bunch of files. And I want to be able to specify all files with a given file extension to compress.
Although I wouldn’t be surprised if you could do this in 4 lines of Perl, I don’t know flippin’ Perl (yet), so I wrote a bash script to do the job.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 | # # Purpose: Script to compress all files of given extension to individual archives using 7z # Usage : zipeach.sh <extension-without-prefix-dot> i.e zipeach.sh n64 # Author : r3dux # Date : 16/04/2009 # #!/bin/bash count=0 # File counter got7z=$(which 7z) # Use "which" to check if there's a copy of 7z on the system # If there's no copy of 7z - we're not going to be doing much compressing... Exit stage left. if [ "$got7z" = "" ]; then echo "No copy of 7z found on system! Try running: sudo apt-get install p7zip-full" exit 0 fi # If we have 7z, and have been given a file extension parameter... if test "$1"; then # Stop the script from entering an infinite loop should user mistakenly enter 7z as filetype to compress... if [ $1 = "7z" ]; then echo "Recursion neatly sidestepped - no 7z filetypes ya scurvy seadog! =P" exit 0 fi echo "Starting Zipeach..." # For each file with the given extension in the current directory... for file in ./*.$1; do # If a file exists with given extension... if [ -e "$file" ]; then # Compress the file with maximum compression (-mx9) and use multiple threads for multiCPU machines (-mmt) # NOTE: Remove -mmt flag to run this on a single CPU box... 7z a -mx9 -mmt "$file".7z "$file" # Increment our file counter let count+=1 else # If no file of required extension has been found then notify user and quit echo "No files of extension .$1 found in current directory!" exit 0 fi # End of if file exists condition done # End of for each file loop # Exit when all files compressed echo # Cheap blank line =P echo "Zipeach completed. $count files of extension .$1 found and compressed." exit 0 else # No extension parameter given? No worky... echo echo 'Please run the script with an extension to compress i.e. "zipeach.sh n64" to compress each .n64 file into its own 7z archive.' exit 0 fi |
Bash, as it turns out, is a fiddly, finicky beast in that you really have to think about what the command-line will see under different circumstances and enclose variables in inverted commas or not in very precise ways (see this article to understand what I mean). All that if / fi stuff too… very odd.
To use the script, copy and paste it into a text file (in my case I’ve called it zipeach.sh), save it, make the file executable using chmod +x zipeach.sh and move it to /usr/bin or something so it’s in your path using sudo mv ./zipeach.sh /usr/bin/ – then run it inside any directory you want to zip files to individual archives by calling it with zipeach nds (for example) to compress all the .nds (Nintendo DS roms) in a folder into individual archives. Or use this link 
Anyway, job done – suggestions? improvements? props? death-threats? Let me know below!
Oh, and cheers to James McDonald for his WP-Syntax hack to stop all the embedded code appearing on a single (incredibly long) line!
Related posts:
Hey mate, good post.
I’ve been mulling this over for a day or so and curiosity got the better of me. I just wanted to know if it could be done in a one liner and I felt like a little bash coding.
7z is not commonly used in linux, so a lot of people won’t already have it installed, you should think to change it to a tar/gzip method. gzip and tar are pretty much installed by default, and 7zip can read and process these files too, so win-win on linux and windows.
Anyway, the one liner…
maxdepth -> only go 1 level (the current directory) . Remove if you want it to do the entire tree
type -> only process files. directories, symbolic links, yada, yada are ignored
-name ‘*.js’ -> I chose a directory with javascript files. Change to *.rom or whatever. Keep the quotes!
! -name ‘*.tar’ -> ignore all files ending in .tar
! -name ‘*.tar.gz’ -> ignore all files ending in .tar.gz, these can also be -iname to ignore case
execdir -> execute the code in the same directory that the file was found in
sh -c -> execute this shell, basically a piped command in this case
tar cvf – $1 -> tar the file, remember that tar is just a container!
gzip -c9 -> compress the tar file (maximum compression, taken from console which is where the tar file was saved)
Outputs the gzip result into the file $1.tar.gz – so basically the identified file with .tar.gz slapped on the end
: {} \; -> VERY necessary, i have only a vague idea as to why and cannot explain it.
However, I am not sure if this command can be alias’ed. Some very funky chicken dance happens when you try that.
Update: Fixed code tag, by using pre lang=”bash” tag
– r3dux
ZOMG! That is pretty awesome! =D
Very clever, very concise and well explained – most impressed…
60 lines compressed down to a single line? Expect no mercy in SF IV now!!
The reason I went with 7z is two-fold: 1.) 7z is a FOSS compression standard – think the .PNG of the compression world 2.) 7z compresses better than gzip, and has common Win32 clients – “… Usually, 7-Zip compresses to 7z format 30-70% better than to zip format. And 7-Zip compresses to zip format 2-10% better than most of other zip compatible programs.” – source: http://www.7-zip.org/
Since I’m after space-saving, 7z works well for me, and in my defense – I do check for 7z and offer a simple one-liner install if not currently available
But yah – you for the win, man!
Flawless Victory!
Do not submit to defeat so easily Ryu, we need to finish the fight! Putting on your boxing mitts and going all Tyler Durden on myself…
Looking at your quoted excerpt harping on about how great 7-Zip was and then seeing that it was taken from 7-zip.org had me thinking that it could be touched with a little bias. Nay, been doused with a double BAM! from the venerable spice weasel.
So I wanted to see if there was a more independent source available. A comparison of compression schemes has done quite a comprehensive test using a multitude of compression tactics on binary files and compared their compression rates. Indeed, your choice of 7-zip is the overall winner.
However, the sceptic in me views some of the figures as just a little too unbelievable. I would prefer to do my own test to check for similar figures as compression rates for 7-zip at 10% whilst every other utility only manages 50-60% just doesn’t seem right. Did this guy explode the compressed files elsewhere on his directory tree and check that those files still work? If he did then my commendations to both his investigative work and the 7z team.
If it is important to keep the linux meta-file info then the provided link gives some good advice, namely…
DO NOT USE the 7-zip format for backup purpose on Linux/Unix because :
* – 7-zip does not store the owner/group of the file.
On Linux/Unix, in order to backup directories you must use tar :
* – to backup a directory : tar cf – directory | 7zr a -si directory.tar.7z
* – to restore your backup : 7zr x -so directory.tar.7z | tar xf -
Important for files being backed up that are owned by many users, less important for files like roms though, as usually only one user owns them anyway.
If you would prefer a 7-zip version of my one-liner then you could use the following. The sh -c is no longer needed since a single command is being used. I’m not sure if the command needs to be 7zr or just 7z. Also a : and {} may no longer be needed, as shown, but you may need to tweak this.
I have not checked this as neither my home or work systems have it installed. You’d probably need to test this on a directory of copied files first.
Damn, that felt like a Hadoken -> Tatsumaki-Senpuukyaku -> Shin Shoryuken combo!
Damn, is that the best css you have for a block of cited text? Or are my submission skills below par once again.
Hehehe – Good job with the line modification for 7z, and a very valid point about 7z not storing user/group/other permissions… Didn’t really think much about that when making my script, as it’s not really for backup purposes – I’m just gonna store things in a compressed state to save a couple of gig… actually, now I have the script I can apply it to lots of roms… I’m not sure if my TOSEC Amiga set is compressed or not [ checks.... okay, actually it is - but I'm sure I've got a stack of stuff that isn't ].
I was a bit skeptical about the 7z claims I quoted, but just took it with a pinch of salt – I’d heard 7z was the b0mb, and had no intention of going all “Skullptura” on stuff – so, ya know, happy-happy joy-joy
And what’s with all the CSS tag bashing? Don’t like my code tag, don’t like my cite tag – which was exactly as it should be as standard – i.e. just italics, but tagged as a citation for screen-readers… Jeez! Modded cite to bold italic in a 14pt mono font – suggestions welcome!
Oh, and I’ve been doing some more Bash coding behind the scenes to strip invalid characters out of the filenames in my mp3 collection – recursively, no less. Expect a post soon! =D
QCF,QCF HP+MP+LP!!!!!11!!1 (LoLz!!)
It’s not a question of me not liking the tags, I just expected a different output. The site is good work and this is my way of helping you tweak it to perfection
I had in my mind something like what is quoted on SAS Tactics and methods which briefly mentions the beautiful art of tap-tap… job done.
Now I’d like to see that invalid characters piece of coding, when it’s ready of course. Sounds really useful.
haha, oooffffff. I retort your ultra combo with a Xianghua crybaby attack
Tried my best to find a vid of it but no dice.
Hi,
I must say after just thinking of doing the same myself yours is the only findable site with this issue covered, I’m trying to do it under windows and have next to no ability to sort it out myself (i tried and failed :-p)
would anyone know how to get a similar thing happening with a batch file, or maybe a vb script (i just about remember how that works)?
Thanks.
MedicalFlyer,
As a quick & dirty batch file method, you could use this:
Copy & paste the above two lines of code into a new batch file, maybe “compress.bat”, then run it and pass it an extension like “compress doc” to compress all files in the current directory with the extension .doc.
After the DO part, you’d want to put something like “pkzip a %%f %%f.zip” or the like instead of the simple echo I’ve put above.
Cheers!
Got a hit from a forum which saw the post, who also say that using “~n” will strip the file extension, for example:
Just saving this info here for future reference!
Hi,
Sorry for resurrecting an old topic, but I’ve been trying to modify the one liner to compress all files in all directories, including subdirectories, using 7zip but I can’t get it to work.
I’m a total novice and don’t really understand how any of this works. Here’s my failed stab at modifying the one liner:
find . -type f -name ‘*.*’ -execdir sh -c ’7z a -mx9 -mmt -tbzip2 %1.bz2 %1′ : {} \;
Would anyone be able to help?
Thanks.
Spunky,
Finally got some free time to have a look at your question – the following line will do the job, and will deal with spaces in filenames just fine:
The above will compress all files (except ones ending in .7z) in all directories to their own unique archives in whatever directory they’re currently in, so “./some documents/some doc.txt” will become “/some document/some doc.txt.7z” – which is what I think you were after…
All the best,
r3dux
A belated “You’re Welcome!”