Sunday, January 6, 2008

Vim: Find and replace text across files

Programmers almost always need the ability to find and replace certain text across multiple files in their projects. Most IDEs provide find and replace functionality. And there are many free tools online to just find and replace text across multiple files. Because I am using Vim as my favorite editor for over 15 years now, it came natural to choose Vim for such job, and not depend on another tool.

For a real life example, in one Rails project, I used caching in my view templates. I used a plugin to set an expiration time for the cache. I needed the cache function to look like:
<% cache("sidebox_latest_news", :expire => 15.minutes.from_now) do %>
<% end %>
I used this technique in about 25 of my templates. Then i decided not to use timed expiration of the cache. So I needed to remove the expire argument of the cache method. I needed to remove it from all of my 25 templates. I used Vim to find the needed piece of text to be removed.

In Vim, when we need to replace text in a file, we use the substitute command.
This should replace the first occurrence of "old_text" with "new_text" for the current cursor line. To substitute all matching occurrences in the line, just add the flag 'g' which refers to global. And to substitute all the file occurrences of the pattern, add the '%' before the substitute command.
Vim alerts if it didn't find the pattern, to disable the alert just append the flag 'e'.
For a complete documentation for the substitute Vim command, use vim help.
:help substitute
Now, after we knew how to substitute text in Vim, we need to substitute the expire argument. we cannot do without using regular expressions. That's because the argument is not the same in all the files, it carries different time values, and may differ in spacing.
<% cache("sidebox_related_reports", :expire => 1.hour.from_now) do %>
<% end %>
So we need to match the whole argument from the comma at the start till the right parentheses and replace it with empty text. the regular expression we should use to match the argument is;
/, *:expire.+)/
I will quickly explain the pattern above, but please refer to any of the many tutorials online about Regular expression for more details. The pattern starts with a comma which is the start of the text we need to remove. Followed by a space and a '*' which means a comma followed by zero or more spaces then ':expire' followed by "one or more" "character". And finally the pattern ended with a right parentheses. So our pattern starts with a comma and ends with a right parentheses. Any text matching this pattern in all templates files should be replaced with empty text, or to be specific, with a right parentheses, as the parentheses is already part of the text to be removed.

To apply Vim command to all the templates files, I will use the 'args' command. I will pass all the templates files in the app/views folder to 'args' using wildcards, and because Rails store the templates files grouped in folders named after their controllers I will use double wildcard 'app/views/*/*' to list the files 2 levels deep under app/views folder.

After I passed the required files to 'args', I can apply whatever command I like to all these files using the command 'argdo'. First I will apply the substitute 's' command and then 'update' which will only save the modified files.
:args app/views/*/*
:argdo %s/, :expire.*)/)/ge | update
Finally, another neat option is adding the flag 'c' which will ask confirmation before each substitution.
:argdo %s/, :expire.*)/)/gec | update


peterm said...

This is a very nice overview of this technique. Thanks for posting it!

Chris Roos said...

Cool. This is pretty useful, thank you.

Crunch said...

Yes, thanks for this. I've been using vim for years. Perhaps I should be embarrassed for not knowing how to do this already, but mostly I'm just happy that I now can. Yay!

Sander said...

Very useful, thanks!

ravikiran said...

Great! Pretty useful thing

Daniel said...

Fantastic - very good description, just saved me hours. Thanks!

antipode said...

I've noticed that you can run any arbitrary shell command with args.

For example, this coughs up all rb files in your current directory:

:args `ls *.rb`

So, if you have a script like this:

if [ -d app ] && [ -d config ] && [ -d script ] && [ -d vendor ]
find . -name '*.sql' -prune -o -name '*.csv' -prune -o -path './log' -prune -o -path './vendor' -prune -o -path '*/.svn' -prune -o -type f -print
echo "Oops! `pwd` doesn't look like a Rails directory!"
exit 1

... and supposing you call it find-rails and you put it in your path, then you can run:

:args `find-rails`

... and you get your whole project tree for handy search-and-replace. The script excludes vendor, svn, log, plus sql files and csv files (if you have any). Also, the script will choke if you're not currently in a rails directory (if you try to run it, vim will produce the cryptic message "E79: Cannot expand wildcards"). This is by design, since a find in the wrong directory can come up with a colossal number of files.

Run :pwd in vim to see your current directory. Do :Rcd (if you're using vim-rails) or whatever to switch to the appropriate rails directory.

Salt to taste.

antipode said...

Here's a slightly more maintainable version of the find-rails script.

if [ -d app ] && [ -d config ] && [ -d script ] && [ -d vendor ]
find . \
-name '*.sql' -prune -o \
-name '*.csv' -prune -o \
-name '*.swp' -prune -o \
-name '*.txt' -prune -o \
-name 'schema.rb' -prune -o \
-path './tmp' -prune -o \
-path './log' -prune -o \
-path './vendor' -prune -o \
-path '*/.svn' -prune \
-o -type f -print
echo "Oops! `pwd` doesn't look like a Rails directory!"
exit 1

Nicholas Evans said...

Thanks for this! Well explained. I've been using vim for over a decade now, but there are many corners and abilities I've never really delved into before. This article came up after a simple google, and now I'm a better vimmer for it. Thanks again!

Hatim Hegab said...

السلام عليكم

لكم انا سعيد برؤية مدونتك هذه - هي اول مدونة بهذا الشكل الإحترافي التي اجدها مكتوبة بواسطة عربي. كنت ابحث عن معلومة حول "في أي إم" ووجدت مدونتك. اتمنى لك التوفيق

حاتم - اميريكا

Sergey Miryanov said...


Sandman said...

Thanks for this post. I've been using Vim for only about a month now but everyday I try to learn something new about it. Today I stumbled upon this post and i'm excited to try it out. Also, thanks for @antipode for the great script.


Michael said...

I was looking for a better S&R method over files. Vim user of 6 years and have never used argsdo... Excellent!

Phil said...

is there a way to revert the argdo changes, if its not possible to apply the inverse regexp to the files?

Dan Goldstein said...

This is very useful. Thank you very much.

James O'Beirne said...

Excellent; thank you. Your post seems like the only place that the args/doarg search and replace technique is expounded on.

Anders said...

Very nice! Thanks for sharing.

Yonah said...

Great timeless article, thanks for posting. This article helped me solve a problem today and is likely to save me a lot of time in the future.

Thank you!

jkff said...

Thanks! Another priceless tool in my vim toolbox - :argdo. Oh, what more is to come :)

maasha said...

:bufdo s///

Daniel Williams said...

Great example - thank you!

Paul Stewart said...

Thanks mate, that was an excellent, clear explanation.
I appreciate it.

Micah said...

This was a huge help to me today. I had to change a method name in 100s of files and this turned it into a 30 minute job instead of 30 hours.

Technocom said...

To Find & Replace Multiple Words in Multiple Word files use a simple tool.

It can replace Headers & Footers along with Fonts, Special Characters, Formatting, Colors etc

Joel Marchesoni said...

Another useful thing when doing this is to use the :set nomore command to disable the prompt to advance pages when performing the replace on a larger number of files than you have screen lines for.


From market data to usable insights Gekko is an open source platform for automating trading strategies over bitcoin markets. Get started!C...