Seo Tricks

My personal experience in content web site building, and how I make money online with it. Search Engine Optimization Tricks, articles and tutorials about Adsense and the Webmaster community.

Wednesday, February 01, 2006

Explaining how to mod rewrite, 301 redirects, and optimizing apache

I would like to report here an article that appeared over Digital Point forums, located here, in which mod rewrite is discussed and explained in detailed. I have never find on the net a guide more clever than this one, so I thought to report it here for the convenience for who wouldn't never read it. But I must point that the credits for this article are for Nintendo, a poster on DP forums, a mod rewrite master! :D

Introduction.


Welcome to mod_rewrite, the Swiss Army Knife of URL manipulation! Despite the tons of examples and docs, mod_rewrite is voodoo!

This module uses a rule-based rewriting engine (based on a regular-expression parser) to rewrite requested URLs on the fly. It supports an unlimited number of rules and an unlimited number of attached rule conditions for each rule to provide a really flexible and powerful URL manipulation mechanism. The URL manipulations can depend on various tests, for instance server variables, environment variables, HTTP headers, time stamps and even external database lookups in various formats can be used to achieve a really granular URL matching.

This module operates on the full URLs (including the path-info part) both in per-server context (httpd.conf) and per-directory context (.htaccess) and can even generate query-string parts on result. The rewritten result can lead to internal sub-processing, external request redirection or even to an internal proxy throughput.

This module was invented and originally written in April 1996. [1]


How to change your URLs from dynamic to search engine friendly static URLs using mod_rewrite.


Get an example of the dynamic URL and the way you want it. For example

http://www.domain.com/cgi-bin/store.cgi?section=Nintendo&id=4867635&item=Pokemon
and
http://www.domain.com/store/Nintendo/4867635/Pokemon.html

Now that you got both URLs, make a domain.com/.htaccess file starting with...

Options +Indexes
Options +FollowSymlinks
RewriteEngine on
RewriteBase /
RewriteRule ^

Depending on the server, you might not need the first two lines.

Right after RewriteRule ^ enter the static URL, then a $, a space, and then original URL (with out the domain part for both URLs).

You now got...

Options +Indexes
Options +FollowSymlinks
RewriteEngine on
RewriteBase /
RewriteRule ^store/Nintendo/4867635/Pokemon.html$ cgi-bin/store.cgi?section=Nintendo&id=4867635&item=Pokemon

In the first URL, the static URL code, where ever the URL will change, replace it with a (.*) (Nintendo, 4867635
and Pokemon in the example above).

Then after .html add a $ and add a \ before the .html
If you have a hyphen (-) in the new static URL, add a \ before the hyphen, for example...

RewriteRule ^store\-(.*)\-(.*)\.html$ cgi-bin/store.cgi?section=Nintendo&id=4867635&item=Pokemon

If you don't add the \, you might get an Internal Server Error message, depending on the servers Apache version.

Now in the static part of the URL where the URL changes, in the first change, change it to $1, then $2 and so on. Then add an [L] at the very end, with a space before the [L].

You now got...

Options +Indexes
Options +FollowSymlinks
RewriteEngine on
RewriteBase /
RewriteRule ^store/(.*)/(.*)/(.*)\.html$ cgi-bin/store.cgi?section=$1&id=$2&item=3 [L]

Save the .htaccess file and upload it at domain.com/.htaccess and your static URLs will now work.
http://www.domain.com/store/Nintendo/4867635/Pokemon.html

Here's some other examples...

http://www.domain.com/cgi-bin/store.cgi?section=Nintendo&id=4867635
RewriteRule ^store/(.*)/(.*)\.html$ cgi-bin/store.cgi?section=$1&id=$2 [L]

http://www.domain.com/cgi-bin/store.cgi?section=Nintendo
RewriteRule ^store/(.*)\.html$ cgi-bin/store.cgi?section=$1 [L]

http://www.domain.com/cgi-bin/store.cgi
RewriteRule ^index\.html$ cgi-bin/store.cgi [L]

In this last example domain.com will show the index of the script. If the page shows nothing, try

RewriteRule ^$ cgi-bin/store.cgi [L]


With all the examples combined, you got...

Options +Indexes
Options +FollowSymlinks
RewriteEngine on
RewriteBase /
RewriteRule ^store/(.*)/(.*)/(.*)\.html$ cgi-bin/store.cgi?section=$1&id=$2&item=3 [L]
RewriteRule ^store/(.*)/(.*)\.html$ cgi-bin/store.cgi?section=$1&id=$2 [L]
RewriteRule ^store/(.*)\.html$ cgi-bin/store.cgi?section=$1 [L]
RewriteRule ^index\.html$ cgi-bin/store.cgi [L]

Notice the order. if you list it as...

Options +Indexes
Options +FollowSymlinks
RewriteEngine on
RewriteBase /
RewriteRule ^index\.html$ cgi-bin/store.cgi [L]
RewriteRule ^store/(.*)\.html$ cgi-bin/store.cgi?section=$1 [L]
RewriteRule ^store/(.*)/(.*)\.html$ cgi-bin/store.cgi?section=$1&id=$2 [L]
RewriteRule ^store/(.*)/(.*)/(.*)\.html$ cgi-bin/store.cgi?section=$1&id=$2&item=3 [L]

then mod_rewrite will freak out and it won't work! List the line with the most variables first, then the second most and so on.

Can I have the .htaccess in a directory?

Yes.

In the above example, for having it at domain.com/store/.htaccess, change the code to...

Options +Indexes
Options +FollowSymlinks
RewriteEngine on
RewriteBase /store/
RewriteRule ^index\.html$ /cgi-bin/store.cgi [L]
RewriteRule ^(.*)\.html$ /cgi-bin/store.cgi?section=$1 [L]
RewriteRule ^(.*)/(.*)\.html$ /cgi-bin/store.cgi?section=$1&id=$2 [L]
RewriteRule ^(.*)/(.*)/(.*)\.html$ /cgi-bin/store.cgi?section=$1&id=$2&item=3 [L]

You moved store/ up to the RewriteBase line and added / before cgi-bin. If the script was in /store/store.cgi
you would of had store/ instead of cgi-bin/ and then just got rid of it, to look like...

Options +Indexes
Options +FollowSymlinks
RewriteEngine on
RewriteBase /store/
RewriteRule ^index\.html$ store.cgi [L]
RewriteRule ^(.*)\.html$ store.cgi?section=$1 [L]
RewriteRule ^(.*)/(.*)\.html$ store.cgi?section=$1&id=$2 [L]
RewriteRule ^(.*)/(.*)/(.*)\.html$ store.cgi?section=$1&id=$2&item=3 [L]

The URL to the index of the store will be domain.com/store/

Ack!!! Now it's messing up the rest of my site.

If you have domain.com/index.html for example, make sure your mod_rewrited URLs use another extension, like .htm or .shtml.

The original script URLs don't have the product name in the URL. Can I add the product name to the URL?

Yes! If you can change the script to put the product names in the URL, or edit the links to link to them, yes you can. Here's an example. Notice there are two (.*)'s and no $2.

RewriteRule ^(.*)/(.*)\.html$ cgi-bin/file.cgi?Item=$1 [L]

Just edit the script links, or links in the static page to link to domain.com/whatever/PRODUCT_NAME.html have the product name show up where the last (.*) is in the .htaccess code.

But how can I get rid of special characters or spaces?

For perl, you can do search and replaces, for example...

$value =~ s/ /_/g;
$value =~ s/?//g;
or
$value =~ s/[^\w\d\-_. ]//g;

which gets rid of almost everything but letters and numbers. Just make sure it only changes the URL and not the content. As for php or asp, I don't know how to do it there.

Can I rewrite a sub-domain to a directory?

Yes. Here's the code [url=http://forums.digitalpoint.com/member.php?find=lastposter&t=20416]mnemtsas came up with...

xxxxx.domain.com

to

www.domain.com/XXXXXX/

RewriteCond %{HTTP_HOST} ^[www\.]*xxxxx.domain-name.com [NC]
RewriteCond %{REQUEST_URL} !^/XXXXX/.*
RewriteRule ^(.*) /XXXXX/$1 [L]

Does .htaccess increase server load?

I have yet to ever see it increase server load on my dedicated server. IMO, that's just a rumor. I got about 30 domains with about 54 lines in the domain.com/.htaccess file and have yet to ever see it effect the server. The only effect I've ever got is getting GoogleBombed (Google chomping away at the static URLs so much that the server almost crashes or does crash!!!). Don't panic. This is why you have static URLs, to help search engines crawl your site.

If you ever see high server loads or a slow server, try optimizing Apache.

How do I optimize Apache?

You have to have access to the actual server through telnet as root.

Edit your httpd.conf file.

Here's the best settings I've found.

Timeout 50
KeepAlive On
MaxKeepAliveRequests 120
KeepAliveTimeout 10
MinSpareServers 10
MaxSpareServers 20
StartServers 16
MaxClients 125
MaxRequestsPerChild 5000

and then restart apache. Even when I have massively HIGH server loads, the sites are fast. Once I had the server load above 100, which is EXTREMELY high, and the static pages loaded as if nothing was high!!

Don't ask me how to do it. If you don't know what you're doing, don't mess with it. Ask your web host. Mess up and your sites can 'die' until it get's fixed! For example, simply pressing return can crash your sites until you go back and undo the return, geting it back to how it was before.

How can I do a 301 redirect?

at domain.com/.htaccess

Options +Indexes
Options +FollowSymlinks
RewriteEngine on
RewriteBase /
RewriteRule ^whatever/(.*)$ http://www.domain.com/$1/ [R=301,L]
or
RewriteRule ^index.htm$ http://www.domain.com/ [R=301,L]

The second example only changes one URL.

(.*) and $1 work the same way here as in mod_rewrite, so you can easily change a lot of URLs with one line. The only change with redirects and mod_rewrite is the R=301 (Redirect 301).

Conclusion.

Yes, mod_rewrite is voodoo, and it may look hard to learn, but it's not that hard. When I first tried to figure it out, I spent a day over at apache.org and hardly got any where (hence there is only one link there as the source to the introduction.) I then posted over on the Amazon Associate board, some one gave me a few lines of code, I changed it a little and with in a day I had a completely search engine friendly Amazon store using MrRats script, and my mod_rewrite hack, which as you may know by now, it completely revolutionized the Amazon AWS industry, until it drove Google insane! mod_rewrite rocks, if you got any URLs that have ?, =, or &, do mod_rewrite!

2 Comments:

At 2:32 AM, Blogger Crazy Dan said...

Great blog. Found your blog while searching for more information at yahoo about this post . Your blog has quite a lot of interesting thoughts. Keep up the good work, Explaining how to mod rewrite, 301 redirects, and optimizing apache

Thanks,

search engine marketing tutorials

 
At 1:14 AM, Blogger Tito Maury said...

I was searching the web and found your entry Explaining how to mod rewrite, 301 redirects, and optimizing apache I really like your site and found it worth while reading through the posts. I am looking to publish a comprehensive site reviewing many different articles and blogg. Please feel free to take a look at my blog at search engine internet marketing strategy and add anything your want.

 

Post a Comment

<< Home