What Is Robots.txt?

For a search engine to keep their listings up to date, and present the most accurate search engine results, they perform an action known as a ‘crawl’. This is essentially sending a ‘bot’ (sometimes known as a ‘spider’) out to crawl the internet. The bot will then find new pages, updated pages or pages it did not previously know to exist. The end result of the crawl is that the search engine results page is updated, and all of the pages found on the last crawl are now included. It’s simply a method of finding sites on the internet.

However, there may be some instances where you have a website page you do not want included in search engine results. For example, you may be in the process of building a page, and do not want it listed in search engine results until it is completed. In these instances, you need to use a file known as robots.txt to tell a search engine bot to ignore your chosen pages within your website.

Robots.txt is basically a way of telling a search engine “don’t come in here, please”. When a bot finds a robots.txt file, it will ‘read’ it and will usually ignore all the URLs contained within. Therefore pages within the file do not appear in search results. It isn’t a failsafe; robots.txt is a request for bots to ignore the page, rather than a complete block, but most bots will obey the information found within the file. Some “nasty” bots may actually ignore your robots.txt file and index everything they find. However, for the nice bots, when you are ready for the page to be included in a search engine, you simply modify your robots.txt file and remove the URL of the designated page.

Why Is SEO Important?

Think back to the days before the internet, when everyone looked for business listings in the Yellow Pages or a similar hard copy directory. Now, imagine you’re trying to find a plumber. You go to the “P” page, flick through, and only see three plumbers listed.

It’d be a massive advantage for the three plumbers, wouldn’t it? To effectively be all that potential customers see. Well, that’s the power of SEO.

When someone uses a search engine, they type in a set of words to bring up results that are relevant to them. Once the applicable websites appear on the results page, they are rated in terms of ‘relevancy’. Any website which has correctly used search engine optimization will be judged by Google to be ‘relevant’. So if you have correctly used SEO, your website will appear somewhere near the top of the search engine results page – hopefully in the top three.

Why is that so important? Well, studies have shown that the vast majority of those using search engines only click on websites listed in the top three results of any search engine results page. This is where it compares to being only one of three businesses listed in an old-style paper directory like the Yellow Pages. Master SEO, and your website will appear in high in search results, in the slots (one to three) that users click. That naturally leads on to more people visiting your website, and that in turn means more business, more customers, and ultimately more money.

What Is SEO?

“SEO” stands for “search engine optimization”, and as the name would suggest, it’s all about utilizing your website so that search engines can find it easily.

When someone searches for a website, they will key a few choice words in to their chosen search engine. For example, if they’re looking for a restaurant in New York, they’ll probably type something like: “good new york restaurant” or even just “new york restaurant”. SEO is all about making sure your website appears at the top of the search results for any given phrase like this.

Obviously the phrase is dependent on your niche, what you’re selling or what services you offer – but good SEO practices are about reducing your site down to those key words that people type into Google. There would be no point, in the example, for a great New York restaurant to have their website online but not have it come up under search results for “new york restaurant”.

However, with hundreds of thousands of websites online, there are literally thousands of pages that could turn up results for (again, using the example) “new york restaurant”. SEO is the method used to beat all the other websites, and get yourself as high in the search results as possible. As most users will tend to visit a website from the top three search results and completely neglect the rest, the efficacy of your SEO really can make or break the success of your website. And if your business in general is dependent on your website, then SEO can be the difference between success or failure in the business world.

« Previous Page