How-To: Replace characters in filenames in Linux

Occassionally you’re likely to end up with a bunch of files with filenames that are underscore separated or space separated when you want them the other way around, and then you either have to manually rename them by hand, or live with it.

I had this issue the other day so I wrote a script to convert all filenames in the current directory to the specified format.

The script switches are:

  • -t – turns on [T]esting mode, which displays what would happen if you ran the script, but makes absolutely no changes to filenames.
  • -u – converts all [U]nderscores to spaces.
  • -s – converts all [S]paces to underscores.
  • -l – converts all filenames to [L]owercase.
  • -p – converts all filenames to u[P]percase.

It’d be nice if I could get all working recusively and offer an option to Capitalise Each Word in the filenames, but this will do for now.

  1. # Script to replace characters in filenames | September 2011 | r3dux
  2. # Note: This script will work on files in the present working directory only.
  4. # Initiate counters
  5. changeCount=0;
  6. ignoreCount=0;
  8. # Initiate flags
  9. displayUsage="FALSE"
  10. testRun="FALSE"
  11. u2sMode="FALSE"
  12. s2uMode="FALSE"
  13. toLowercase="FALSE"
  14. toUppercase="FALSE"
  15. proceed="FALSE"
  17. # Set flags from the user provided parameters
  18. while getopts "htuslp" optname
  19. do
  20. 	case "$optname" in
  21. 		"h")
  22. 			displayUsage="TRUE"
  23. 			;;
  24. 		"t")
  25. 			echo "Test mode only - no filenames will be modified..."
  26. 			testRun="TRUE"
  27. 			;;
  28. 		"u")
  29. 			echo "Mode set to convert underscores to spaces..."
  30. 			u2sMode="TRUE"
  31. 			proceed="TRUE"
  32. 		        ;;
  33. 	        "s")
  34. 			echo "Mode set to convert spaces to underscores..."
  35. 			s2uMode="TRUE"
  36. 			proceed="TRUE"
  37. 		        ;;
  38. 		"l")
  39. 			echo "Mode set to convert filenames to lowercase..."
  40. 			toLowercase="TRUE"
  41. 			;;
  42. 		"p")
  43. 			echo "Mode set to convert filenames to uppercase..."
  44. 			toUppercase="TRUE"
  45. 			;;
  46. 		# Catch-all for unknown switches
  47. 	        *)
  48. 			echo
  49. 			echo "Unknown error while processing parameters. Please use -h to display usage paramaters."
  50. 			echo
  51. 			exit 0
  52.         		;;
  53. 	esac
  54. done
  55. echo
  57. # If asked to, or if we got a parameter we don't recognise show the usage options
  58. if [ $displayUsage == "TRUE" ]; then
  59. 	echo "repchars is a small script to replace characters in filenames, or vice versa."
  60. 	echo
  61. 	echo "Usage: repchars [-t] (-u | -s) [-l | -p]"
  62. 	echo
  63. 	echo -e "-t\tturns on [T]esting mode, which displays what would happen if you ran the script, but makes absolutely no changes to filenames."
  64. 	echo -e "-u\tconverts all [U]nderscores to spaces."
  65. 	echo -e "-s\tconverts all [S]paces to underscores."
  66. 	echo -e "-l\tconverts all filenames to [L]owercase."
  67. 	echo -e "-p\tconverts all filenames to u[P]percase."
  68. 	echo
  69. 	echo "You can use use any combination of parameters except the mutually exclusive options -u and -s, and -l and -p."
  70. 	echo
  71. 	echo "It's recommended that you do a trial run with the -t switch to be sure the outcome is as you want it before commiting!"
  72. 	echo
  74. 	exit 0;
  75. fi
  77. # Moan if we have both 'toLowercase' and 'toUppercase' set (mutually exclusive)
  78. if [ $toUppercase == "TRUE" -a $toLowercase == "TRUE" ]; then
  79. 	echo "You cannot convert to both uppercase and lowercase at the same time."
  80. 	proceed="FALSE"
  81. fi
  83. # Moan if we have both 'spaces-to-underscores' and 'underscores-to-spaces' set (mutually exclusive)
  84. if [ $s2uMode == "TRUE" -a $u2sMode == "TRUE" ]; then
  85. 	echo "You cannot convert spaces to underscores and underscores to spaces at the same time."
  86. 	proceed="FALSE"
  87. fi
  89. # If the proceed flag isn't set, we don't proceed...
  90. if [ $proceed == "FALSE" ]; then
  91. 	echo "Please use repchars -h for usage instructions."
  92. 	echo
  93. 	exit 0;
  94. fi
  96. # Function to replace characters dependent on the flags set
  97. function replaceChars()
  98. {
  99. 	# Note: Once we're in this function $1 isn't the first parameter passed to the 
  100. 	# script, it's the first (and only) parameter passed to this function.
  102. 	# If appropriate, convert underscores to spaces
  103. 	if [ $u2sMode == "TRUE" ]; then
  104. 		newFilename=$(echo "$1" | tr _ ' ')
  105. 	fi
  107. 	# If appropriate, convert spaces to underscores
  108. 	if [ $s2uMode == "TRUE" ]; then
  109. 		newFilename=$(echo "$1" | tr ' ' _)
  110. 	fi
  112. 	# If apppropriate, convert filenames to lowercase
  113. 	if [ $toLowercase == "TRUE" ]; then
  114. 		newFilename=$(echo "$newFilename" | tr 'A-Z' 'a-z')
  115. 	fi
  117. 	# If apppropriate, convert filenames to uppercase
  118. 	if [ $toUppercase == "TRUE" ]; then
  119. 		newFilename=$(echo "$newFilename" | tr 'a-z' 'A-Z')
  120. 	fi
  122. 	# Only modify the filename if the original and new filenames are not the same thing
  123. 	# (i.e. there may be nothing to change, and if we mv the file to itself mv will complain)
  124. 	if [ "$1" != "$newFilename" ]; then
  126. 		let changeCount+=1
  128. 		if [ $testRun == "FALSE" ]; then
  129. 			echo "Original filename: $1"
  130. 			echo "New filename     : $newFilename"
  131. 			echo `mv "$1" "$newFilename"` # This actually renames the file
  132. 		else
  133. 			echo "*TESTRUN* Original filename: $1"
  134. 			echo "*TESTRUN* New filename     : $newFilename"
  135. 			echo
  136. 		fi
  137. 	else
  138. 		let ignoreCount+=1
  140. 		echo -e "No change required to filename: $1"
  141. 	fi
  142. }
  144. # Call the replacement function on each file
  145. for f in *; do
  147. 	replaceChars "$f" 
  149. done # End of file checking loop
  151. # Finally, display the counts of filename changes and exit
  152. echo
  153. if [ $testRun == "TRUE" ]; then
  154. 	echo "We would have changed the filenames of $changeCount file(s) and ignored $ignoreCount file(s) which required no changing."
  155. else
  156. 	echo "Changed the filenames of $changeCount file(s) and ignored $ignoreCount file(s) which required no changing."
  157. fi
  158. echo

Like most scripting, this entire thing could probably be done in a single line, but I found it useful to learn how to use getopts and such.

Hope someone else finds it useful! Oh, and if you need a do-everything file renaming script, I read that FixNames script does what it says on the tin.

How to: Compile Qt projects which use the Q_OBJECT macro in Code::Blocks

I need to write some C++ cross-platform GUI code soon, and after umming-and-ahhing about whether to use GTK or Qt, I’ve decided to try out the latter despite the recent Microsoft buy-out. However, as soon as I started to get into it and write some simple GUI code, I hit a snag; Qt uses what it calls slots to bind buttons to functions, where a function that can be executed as an action is defined as being a slot. To give an example, if you had a integer counter and you wanted to add five to the counter when you clicked the “Plus Five” button, then your onPlusFive function which actually increments the counter would be considered a slot.

This is all well and good, until you write the code and it won’t compile – instead giving you errors about vtables and such. To fix this, you need to do two things, the first of which is to add the Q_OBJECT macro definition to your QWidget-extending class.


I’m really new to Qt, so any experienced coders might roll their eyes at this – but I expected to be able to just write some code that creates a window, some buttons, layouts, menu options etc, and then just compile it and we’re good to go a-la Java’s AWT/Swing interface kits. But no, unfortunately not. Instead, you have to add the Q_OBJECT macro to your class like this, and then do some pre-processing:

#include <QtGui/QWidget>
#include <QtGui/QApplication>
#include <QtGui/QPushButton>
#include <QtGui/QLabel>
class Communicate : public QWidget
	Q_OBJECT // Define our class to use the Q_OBJECT macros (required for adding new slots)
		Communicate(QWidget *parent = 0); // Constructor
	private slots: // Functions for buttons
		void onPlusFive();
		void onMinusFive();
		QLabel *label;

That doesn’t seem too bad, right? You just plonk the word Q_OBJECT at the top of your class, and list all slots (functions that get executed as button actions) under the private slots: label. Well, yeah – it’s not so bad, but it’s not going to work that easily either. For this to work properly, we also need to do some pre-processing with a Qt utility called moc

Moc Pre-processing Steps

AFAIK, the pre-processing that needs to occur must happen on all header files (and header files only) that utilise Qt in your project. You can do it by hand like this:

moc <some-class>.h -o moc_<some-class>.cpp

For example, if you had a MainWindow class separated into .h and .cpp files, you might run:

moc MainWindow.h -o moc_MainWindow.cpp

Once that’s done, you need to add the newly generated moc_MainWindow.cpp file (for this example) to your Code::Blocks project, and it’ll all compile and work as expected.

So good so far, but what happens when you add a new slot? Yup, you have to go and manually re-run the command on all header files to generate the new moc_<some-class>.cpp files before it’ll play ball again – well sod that, let’s automate it!

Quick & Dirty Pre-Processing via Bash

I’m sure there are better ways of invoking the moc as a pre-build script, but I don’t know enough about pre-processing scripts to get it working (however, you might like to read this and have more luck than I did) – so I knocked together a quick bash script to generate all the moc files for us, like this:

genQtMoc Script

# Script to generate Qt moc files for all headers in the current directory | 16/06/2011 | r3dux
# For each header file in the current directory...
for file in ./*.h; do
	# If a file exists with a given extension...
	if [ -e "$file" ]; then
		fileWithoutPath=$(basename $file)
		moc "$file" -o "$mocName"
	fi # End of if file exists condition				
done # End of for each file loop

This is fine if all your .cpp and .h files are in the same folder like the project root, but if you’ve split them off into include and src directories you’ll have to twiddle the script a little to take that into account.

Putting it to work

Once you’ve put the script into a file called genQtMoc or such, and made it executable with chmod +x genQtMoc, sling it somewhere in the path like /usr/local/bin/ and we can then call for it to be executed as a pre-build script in Code::Blocks by going to Project | Build options… | Pre/post build steps and just putting the name of the script to call in the pre section, like this:

Code::Blocks Pre-Build Script

All sorted – the only other thing to re-iterate is that you need to have each moc_<some-file>.cpp file attached to your project, and that if you have the file open for editing in Code::Blocks, it’ll moan that “the file has been changed externally, would you like to reload it?” each time its contents change. You can get around this by simply not having the file open for editing (but still included in your project), which stops the annoying pop-up from occurring and is fine as it’s an auto-generated file that shouldn’t really be hand-edited anyway.

Anyways, I hope this helps anyone stuck in a similar situation – and please, if you do know a better of way of going about this, I’d love to hear about it. Cheers!

How to: Get absolute/relative file paths, filenames and extensions from a Bash script

I’ve been learning some Qt stuff today, and the resulting code requires pre-processing of header files to work correctly, which I thought I’d try to automate. As part of this, I needed to find out how to get at all the different elements of a file from a bash script, so I did some googling and got all the info I needed :)

The script itself isn’t particularly useful, but the component parts of how to get at paths, extensions, and plain filenames without extensions definitely is – check it out:

fileParts Shell Script

# Bash script to get filename details | 16-06-2011 | r3dux
# Usage: fileParts <extension-with-no-dot> i.e. fileParts cpp
# For each file with the given extension in the current directory...
for file in ./*.$1; do
	# If a file exists with a given extension...
	if [ -e "$file" ]; then
		fullPath=$(readlink -f "$file")
		echo "Full path and filename          is: " $fullPath
		echo "Relative path and filename      is: " $file
		fileWithoutPath=$(basename $file)
		echo "Filename only                   is: " $fileWithoutPath
		echo "File extension only             is: " $extension
		echo "Filename only without extension is: " $fileWithoutExtension
		echo "No files of type $1 found!"
		exit 0
	fi # End of if file exists condition				
done # End of for each file loop

I created two dummy “.h” files in my home folder and ran the script – this is what it outputs:

r3d2@r3dux-Aspire-8920:~$ ./fileParts h
Full path and filename          is:  /home/r3d2/a-simple-test-file.h
Relative path and filename      is:  ./a-simple-test-file.h
Filename only                   is:  a-simple-test-file.h
File extension only             is:  h
Filename only without extension is:  a-simple-test-file
Full path and filename          is:  /home/r3d2/file.with.lots.of.dots.h
Relative path and filename      is:  ./file.with.lots.of.dots.h
Filename only                   is:  file.with.lots.of.dots.h
File extension only             is:  h
Filename only without extension is:  file.with.lots.of.dots

That’s gotta come in useful, right? =D

Help scrape Google Video before it’s gone forever!

Update: Google About-Turn

Google have capitulated to feedback and decided to keep Google Video alive and migrate the videos to YouTube, by which point Archive Team had 40% of the content and were well on track to save it.

Google Video will be shutting down within the next few weeks, and for some stupid reason, they’re not just transferring the videos to YouTube (as Google owns both) – instead they’re just pulling the plug and it’s all going to be lost. To fix this rubbish state of affairs, Archive Team are in a race to scrape as much Google Video content as they can before the viewing deadline (29/04/2011) and the download deadline (13/05/2011) – and you can help! have kindly donated 100TB for storage, but first we need to index the videos and scrape them.

If you have a lot of bandwidth you can help scrape the videos themselves, but even if you don’t you can help with the indexing effort by running a simple, resource and bandwidth light Linux script and just leaving it running!

Why save it?

YouTube has a 15 minute video length limit – and Google Video doesn’t. This means there are large amounts of video that might be on Google Video and nowhere else, so when they’re gone – they’re gone. A lot of this might not be fantastic material – but a lot of it will be unique and the only copy on the Net. There’s documentaries, films, and all sorts of good stuff, and even personal video blogs will be a snapshot of the times we live in. In short, it’s stuff we as a species, should not throw away.

It’s like the BBC scrapping their archives when they didn’t want to pay to save them – we won’t know what’s been lost until it’s gone, and by then it’ll be too late. So let’s not let that happen, eh?

How you can help if you have ~200GB bandwidth/storage or more: Scrape the videos

Update: pentium ported the video download script to Windows (you still need python and aria2, which can be downloaded separately). Script location:

Head on over to ArchtiveTeam Google Videos wiki, sign up, pick an un-taken section of videos and add your initials/handle to it, then go for your life! Full instructions at the site.

How you can help if you don’t have a lot of bandwidth/storage

Update: nstrom ported the indexing script to Windows – all you now need is a pre-compiled version of the phantomjs browser. Full instructions come with the script, which is located here:

Note: This will only work on Linux machines with X running – you can’t run it on headless servers due to phantomjs requirements. Instructions are for Ubuntu 10.10 or later and might need a little modification if you’re running an older or non-Debian based distro.

  1. Get and build phantomjs (a headless web browser) by doing the following:
    • Install build-essential, curl git, libqtwebkit4and libqtwebkit-dev if necessary, for example using:
      sudo apt-get install build-essential curl git libqtwebkit4 libqtwebkit-dev
    • Create a directory called phantomjs
    • In the terminal go into your new directory and run the following command to get the phantomjs source code:
      git clone
    • Build phantomjs by issuing the command:
      qmake && make
    • Move the phantomjs binary somewhere in your path by issuing the command:
      cd bin && sudo mv ./phantomjs /usr/bin
  2. Create a folder called gvscript or such and download the file with the list of Google Video related pages to scrape: google_video_related.tar.gz
    • Extract the above downloaded file (Right-click and Extract To.. or use tar -zxvf ./google_video_related.tar.gz)
  3. In a terminal, navigate to the folder where you extracted the google_video_related file (above) and run the following command to help scrape Google Video:
    while : ; do ./ ; done
  4. Simply leave the script running, and head on over to #ggtesting on EFnet (IRC) if you need any assistance or in case the script has any issues (p.s. kill the script with Ctrl+Z if it misbehaves – though mine’s been running for about 7 hours solid with no complaints so I doubt you’ll have any).

The script scrapes each page for related videos and sends them off to an archiveteam server. It takes very little processing and bandwidth on your end (a couple of kb/sec, if that) and seems to work just fine.

Every little helps

I’m sure anything you can do to pitch in will be appreciated by Archive Team, the Internets, your future self, your kids, your kids kids, your kids kids kids… you get the picture ;)


How To: Convert a Directory of MP3s to WAVs in Linux

I went to burn a couple of audio CDs for the boy today, and bod-frickn-dammit if Brasero / GnomeBaker and K3B didn’t all threw their hands in the air in dismay that I might actually have the nerve to want to convert mp3s to CD-audio on the fly. Very poor.

So, as wav files come in less flavours than mp3s and tend to work first time, I knocked up a quick script to convert a directory of mp3s to .wav files using mpg123, it even handles spaces correctly after I’d given it a stern talking to… Anyways:

gotmpg123=$(which mpg123)
# If there's no copy of mpg123 we're not going to be doing any converting. Abort!
if [ "$gotmpg123" = "" ]; then
    echo "No copy of mpg123 found on system! Try running: sudo apt-get install mpg123"
    exit 0
# Otherwise, if we're all set, make a directory to dump our wav files into
mkdir WAV
# For each file in the current directory that ends with .mp3, convert it to .wav and place in our new WAV folder
for file in ./*.mp3
  mpg123 -w ./WAV/"${file}".wav "$file"

To use the script:
– Copy and paste the above code into a new text file called mp32wav or something
– Make it executable with chmod +x mp32wav, then
– Copy it to /usr/local/bin for easy access with sudo cp mp32wav /usr/local/bin/

With that all done you can just go into a folder of mp3s in the terminal and fire it off. It’ll create a folder called WAV inside whatever directory you’re in and stick the converted wav files there with the original filename but with .wav tacked on the end.