How To: Compress a Directory of Files into Individual Archives

I’ve got a stack of files all thrown together in the same directory, and I wanted them compressed – simple enough, eh? Only thing is I wanted each file compressed to its own archive, so I can see at a glance what’s there, and if for some reason an archive gets corrupted, it’s just one file lost and I can replace it instead of having to dick around repairing corrupted “blob” archives that contain the entire bunch of files. And I want to be able to specify all files with a given file extension to compress.

Although I wouldn’t be surprised if you could do this in 4 lines of Perl, I don’t know flippin’ Perl (yet), so I wrote a bash script to do the job.

Bash, as it turns out, is a fiddly, finicky beast in that you really have to think about what the command-line will see under different circumstances and enclose variables in inverted commas or not in very precise ways (see this article to understand what I mean). All that if / fi stuff too… very odd.

To use the script, copy and paste it into a text file (in my case I’ve called it zipeach.sh), save it, make the file executable using chmod +x zipeach.sh and move it to /usr/bin or something so it’s in your path using sudo mv ./zipeach.sh /usr/bin/ – then run it inside any directory you want to zip files to individual archives by calling it with zipeach.sh nds (for example) to compress all the .nds (Nintendo DS roms) in a folder into individual archives. Or use this link ;)

Anyway, job done – suggestions? improvements? props? death-threats? Let me know below!

Oh, and cheers to James McDonald for his WP-Syntax hack to stop all the embedded code appearing on a single (incredibly long) line!

20 thoughts on “How To: Compress a Directory of Files into Individual Archives”

  1. Hey mate, good post.

    I’ve been mulling this over for a day or so and curiosity got the better of me. I just wanted to know if it could be done in a one liner and I felt like a little bash coding.

    7z is not commonly used in linux, so a lot of people won’t already have it installed, you should think to change it to a tar/gzip method. gzip and tar are pretty much installed by default, and 7zip can read and process these files too, so win-win on linux and windows.

    Anyway, the one liner…

    maxdepth -> only go 1 level (the current directory) . Remove if you want it to do the entire tree
    type -> only process files. directories, symbolic links, yada, yada are ignored
    -name ‘*.js’ -> I chose a directory with javascript files. Change to *.rom or whatever. Keep the quotes!
    ! -name ‘*.tar’ -> ignore all files ending in .tar
    ! -name ‘*.tar.gz’ -> ignore all files ending in .tar.gz, these can also be -iname to ignore case
    execdir -> execute the code in the same directory that the file was found in
    sh -c -> execute this shell, basically a piped command in this case
    tar cvf – $1 -> tar the file, remember that tar is just a container!
    gzip -c9 -> compress the tar file (maximum compression, taken from console which is where the tar file was saved)
    Outputs the gzip result into the file $1.tar.gz – so basically the identified file with .tar.gz slapped on the end
    : {} \; -> VERY necessary, i have only a vague idea as to why and cannot explain it.

    However, I am not sure if this command can be alias’ed. Some very funky chicken dance happens when you try that.

    Update: Fixed code tag, by using pre lang=”bash” tag ;) – r3dux

  2. ZOMG! That is pretty awesome! =D

    Very clever, very concise and well explained – most impressed…

    60 lines compressed down to a single line? Expect no mercy in SF IV now!!

    The reason I went with 7z is two-fold: 1.) 7z is a FOSS compression standard – think the .PNG of the compression world 2.) 7z compresses better than gzip, and has common Win32 clients – “… Usually, 7-Zip compresses to 7z format 30-70% better than to zip format. And 7-Zip compresses to zip format 2-10% better than most of other zip compatible programs.” – source: http://www.7-zip.org/

    Since I’m after space-saving, 7z works well for me, and in my defense – I do check for 7z and offer a simple one-liner install if not currently available :)

    But yah – you for the win, man!

    Flawless Victory!

  3. Do not submit to defeat so easily Ryu, we need to finish the fight! Putting on your boxing mitts and going all Tyler Durden on myself…

    Looking at your quoted excerpt harping on about how great 7-Zip was and then seeing that it was taken from 7-zip.org had me thinking that it could be touched with a little bias. Nay, been doused with a double BAM! from the venerable spice weasel.

    So I wanted to see if there was a more independent source available. A comparison of compression schemes has done quite a comprehensive test using a multitude of compression tactics on binary files and compared their compression rates. Indeed, your choice of 7-zip is the overall winner.

    However, the sceptic in me views some of the figures as just a little too unbelievable. I would prefer to do my own test to check for similar figures as compression rates for 7-zip at 10% whilst every other utility only manages 50-60% just doesn’t seem right. Did this guy explode the compressed files elsewhere on his directory tree and check that those files still work? If he did then my commendations to both his investigative work and the 7z team.

    If it is important to keep the linux meta-file info then the provided link gives some good advice, namely…

    DO NOT USE the 7-zip format for backup purpose on Linux/Unix because :
    * – 7-zip does not store the owner/group of the file.

    On Linux/Unix, in order to backup directories you must use tar :
    * – to backup a directory : tar cf – directory | 7zr a -si directory.tar.7z
    * – to restore your backup : 7zr x -so directory.tar.7z | tar xf –

    Important for files being backed up that are owned by many users, less important for files like roms though, as usually only one user owns them anyway.

    If you would prefer a 7-zip version of my one-liner then you could use the following. The sh -c is no longer needed since a single command is being used. I’m not sure if the command needs to be 7zr or just 7z. Also a : and {} may no longer be needed, as shown, but you may need to tweak this.

    I have not checked this as neither my home or work systems have it installed. You’d probably need to test this on a directory of copied files first.

    Damn, that felt like a Hadoken -> Tatsumaki-Senpuukyaku -> Shin Shoryuken combo!

  4. Damn, is that the best css you have for a block of cited text? Or are my submission skills below par once again.

  5. Hehehe – Good job with the line modification for 7z, and a very valid point about 7z not storing user/group/other permissions… Didn’t really think much about that when making my script, as it’s not really for backup purposes – I’m just gonna store things in a compressed state to save a couple of gig… actually, now I have the script I can apply it to lots of roms… I’m not sure if my TOSEC Amiga set is compressed or not [ checks…. okay, actually it is – but I’m sure I’ve got a stack of stuff that isn’t ].

    I was a bit skeptical about the 7z claims I quoted, but just took it with a pinch of salt – I’d heard 7z was the b0mb, and had no intention of going all “Skullptura” on stuff – so, ya know, happy-happy joy-joy ;)

    And what’s with all the CSS tag bashing? Don’t like my code tag, don’t like my cite tag – which was exactly as it should be as standard – i.e. just italics, but tagged as a citation for screen-readers… Jeez! Modded cite to bold italic in a 14pt mono font – suggestions welcome!

    Oh, and I’ve been doing some more Bash coding behind the scenes to strip invalid characters out of the filenames in my mp3 collection – recursively, no less. Expect a post soon! =D

    QCF,QCF HP+MP+LP!!!!!11!!1 (LoLz!!)

  6. It’s not a question of me not liking the tags, I just expected a different output. The site is good work and this is my way of helping you tweak it to perfection :)

    I had in my mind something like what is quoted on SAS Tactics and methods which briefly mentions the beautiful art of tap-tap… job done.

    Now I’d like to see that invalid characters piece of coding, when it’s ready of course. Sounds really useful.

    haha, oooffffff. I retort your ultra combo with a Xianghua crybaby attack :) Tried my best to find a vid of it but no dice.

  7. Hi,

    I must say after just thinking of doing the same myself yours is the only findable site with this issue covered, I’m trying to do it under windows and have next to no ability to sort it out myself (i tried and failed :-p)

    would anyone know how to get a similar thing happening with a batch file, or maybe a vb script (i just about remember how that works)?
    Thanks.

  8. MedicalFlyer,

    As a quick & dirty batch file method, you could use this:

    Copy & paste the above two lines of code into a new batch file, maybe “compress.bat”, then run it and pass it an extension like “compress doc” to compress all files in the current directory with the extension .doc.

    After the DO part, you’d want to put something like “pkzip a %%f %%f.zip” or the like instead of the simple echo I’ve put above.

    Cheers!

    1. Got a hit from a forum which saw the post, who also say that using “~n” will strip the file extension, for example:

      Just saving this info here for future reference! :)

  9. Hi,

    Sorry for resurrecting an old topic, but I’ve been trying to modify the one liner to compress all files in all directories, including subdirectories, using 7zip but I can’t get it to work.

    I’m a total novice and don’t really understand how any of this works. Here’s my failed stab at modifying the one liner:

    find . -type f -name ‘*.*’ -execdir sh -c ‘7z a -mx9 -mmt -tbzip2 %1.bz2 %1’ : {} \;

    Would anyone be able to help?

    Thanks.

  10. Spunky,

    Finally got some free time to have a look at your question – the following line will do the job, and will deal with spaces in filenames just fine:

    The above will compress all files (except ones ending in .7z) in all directories to their own unique archives in whatever directory they’re currently in, so “./some documents/some doc.txt” will become “/some document/some doc.txt.7z” – which is what I think you were after…

    All the best,
    r3dux

    1. Hi,
      To compress multiple directories into separate archive files I modified your one liner by changing -type f to -type d and everything seemed to be fine except I was gettin a strange, huge ./.7z file besides all of the required archive files. Not sure what needs to be modified to make it work ‘better’.
      Thanks,
      Pawel.

      1. I tried it with the -d instead of -f and got the ..7z file as well (which is a single archive of everything in all the directories). But I also noticed that the command went into every folder and compressed every subfolder into its own .7z file, too! Not what we want!

        The fix for this is to tell find to only traverse 1 directory deep (-maxdepth 1 – keep this as the first parameter specifed or find will complain), and then also tell it to exclude the current directory we’re running the command in (! -name ‘.’).

        The end result to compress all subdirectories in the directory where you run the command then becomes:

        But this will also exclude any directories that end with .7z, which you’re unlikely to have, so a better version is:

        Cheers!

        1. Thanks for sharing this with us. I am almost there. Only thing which is missing is relocating 7z files to another location. adding -o”/newpath/$1″.7z and other combination did not work for me. Any suggestion please? I have over 630 GBs collection in front of me to be compressed and archived, so you assistance is highly appreciated.
          Best,
          Pawel.

          1. Just place the destination path before the “$1”.7z part like this:

            For example, if I wanted to compress each directory into a 7z archive and have them placed in the folder ~/backups (where ~ gets substituted for the current users home directory, so for me would equate to /home/r3dux), I could use:

            Hope this helps.

            1. Thanks a lot! This works like a charm. You knew it. :)
              I am currently searching for more of you bash scripts. Since GUI is not very friendly for a screen reader under Linux I like to stay with the command line.
              Best,
              Pawel.

    2. I love your find . -type f -name ‘*’ ! -name ‘*.7z’ -execdir sh -c ‘7z a -mx9 -mmt “$1”.7z “$1″‘ : {} \; command, but I have one question.

      usually I add things like this to my bashrc file as an alias, but since it has single and double quotes, that is not an option.

      how can I make this a bashrc function, or a stand alone program that i can call at any time?

      thanks!

      1. Glad you like =D

        To make it easily executable, why not stick it in a bash script, make the bash script executable and then sling it in /usr/local/bin.

        Alternatively, Ctrl+R then typing the first part of the command (then hitting Ctrl+R again as many times as you like) allows you to search your bash command history, but I think a bash script is the better solution.

        Cheers,
        r3dux

Leave a Reply

Your email address will not be published.