HOW SEARCH ENGINES WORK

Lesson - 2

HOW DO SEARCH ENGINES WORK?

Search engines work by crawling hundreds of billions of pages using their own web crawlers. These web crawlers are commonly referred to as search engine bots or spiders.

Web crawlers follow links from page to page in order to find new content to add to the search index.

When you use a search engine, relevant results from the index are extracted and ranked using an algorithm.

If that sounds complicated, it’s because it is. But, if you want to rank higher in search engines and drive more traffic to your website, you must first understand how search engines work, index, and rank content.

Be relaxed, we will go into details step by step.

AIM OF SEARCH ENGINES?

You will be amazed to know search engine handles more than 2 trillion searches per year and it aims to provide the best and most relevant results to its searcher’s as fast as possible.

To achieve this, the search engine keeps crawling and indexing the web pages 24/7 and saves the HTML version of a page in a gigantic database.

And when a search performs, the search engine scours its entire database and returns only those results which are most relevant to the searcher’s query within a fraction of a second.

how search engines work

For Example:

If you search for the term ‘off-page SEO benefits’, Google will come up with approximately 36 million results within 0.66 seconds.

search result example

To accomplish this entire sequence, from holding billions of web pages to promptly displaying the relevant results, the search engine company have set up their data centre all over the world.

That does three things:

  • Crawling
  • Indexing
  • and Ranking

WHAT IS SEARCH ENGINE CRAWLING?

Crawling is the process in which search engines send computer bots (referred to as crawlers or spiders) to find new and updated content.

Crawlers use various techniques to visit each website and try to determine how many pages it has, whether it is textual content, videos, or images; but regardless of the format, content is discovered by links.

It also examines the web page’s internal and external links in order to discover more and more pages.

Spider do it continuously and check if there are any changes made in the website (adding or deletion of content, images, links etc.) and keep track of it.

So make sure crawlers will discover and scan your website without any issue.

HOW DO YOU TELL SEARCH ENGINES TO CRAWL YOUR WEBSITE?

By telling search engines how to crawl your site, you can gain more control over what appears in the index.

If you used Google Search Console or the “site:domain.com” advanced search operator you can easily find how many pages are indexed, missed or unimportant pages have been mistakenly indexed.

site operator image

The number of results Google displays isn’t exact, but it does give you a solid idea of which pages are indexed on your site and how they are currently showing up in search results.

Use XML sitemap to list all the important pages of your website (It’s a one time process) — One of the easiest ways to ensure Google will find your highest priority pages. While submitting a sitemap does not replace the need for good site navigation, it can help crawlers find all of your important pages.

IMPORTANCE OF GOOD SITE NAVIGATION

A crawler needs a path of links on your own site to direct it from page to page, just as it needs links from other sites to find your site.

Many website owners make the critical mistake of not linking some of their pages to the rest of their website pages. Don’t do it; if you do, your unlinked page will become invisible.

unlinked page example

That is why your website must have clear navigation, clean information architecture and helpful URL structures.

WHAT IS INFORMATION ARCHITECTURE?

The practice of organizing and labelling content on a website to improve efficiency and findability for users is known as information architecture.

It means users shouldn’t have to think very hard to navigate or to find something on your website.

DEFINING URL STRUCTURE

A well-crafted URL provides both humans and search engines with an easy-to-understand indication of what the destination page will be about.

Example of best URL practices:

https://www.example.com/coffee

https://www.example.com/products/shirt

From the above example, users and search engines will easily understand the topic by looking at the URLs.

Example of bad URL practices:

https://www.example.com/455de/sh45

https://www.example.com/rtgdfd/5879

From the above example, users and search engines will not understand the topic by looking at the URLs.

However, there are chances where you might not want the search engine to index things like old URLs that have thin content, duplicate URLs (such as sort-and-filter parameters for e-commerce), special promo code pages, login pages, staging or test pages, and so on.

To direct Googlebot away from certain pages and sections of your site, use robots.txt.

Robots.txt

Robots.txt is a text file webmasters create to instruct web robots (typically search engine robots) how to crawl pages on their website.

You can use almost any text editor to create a robots.txt file. For example, Notepad, TextEdit, and emacs can create valid robots.txt files.

Key Point of Robots.txt file:

1. Robots.txt is case sensitive: the file must be named “robots.txt”

2. If Googlebot can’t find a robots.txt file for a site, it proceeds to crawl the site.

3. If Googlebot finds a robots.txt file for a site, it will usually abide by the suggestions and proceed to crawl the site.

After you’ve ensured that your site is crawlable, the next step is to ensure that it can be indexed.

HOW SEARCH ENGINES BUILD THEIR INDEX?

Indexing is where processed information from crawled pages is added to a big database called the search index.

Search engine stores all the important details of crawled pages. Like when it was created, freshness, title, description, keywords, type of content, alt attribute, internal and external links and other parameters that are needed by their algorithms.

When you type a query into a search engine, you’re not directly searching the internet for matching results. You’re searching a search engine’s index of web pages.

If a web page isn’t in the search index, search engine users won’t find it. That’s why getting your website indexed in major search engines like Google and Bing is so important.

TELL SEARCH ENGINES HOW TO INDEX A WEBSITE

You can tell search engines how you want your web page to be treated by using Meta directives (or “meta tags”).

ROBOTS META TAG

index/noindex tells search engines whether a page should be crawled and stored in their index for retrieval.

If you use “noindex,” you’re telling crawlers that you don’t want the page to appear in search results. Because search engines assume that they can index all pages by default, using the “index” value is unnecessary.

follow/nofollow tells search engines whether or not to follow links on a page. “Follow” causes bots to follow the links on your page and pass link equity to those URLs. Alternatively, if you use “nofollow,” search engines will not follow or pass any link equity through to the links on the page.

noarchive is used to restrict search engines from saving a cached copy of the page. If you run an e-commerce site and your prices change frequently, you should think about using the noarchive tag to prevent searchers from seeing outdated pricing.

HOW SEARCH ENGINES RANK PAGES?

Actually, a search engine is an answering machine and it will only survive if it shows the proper results.

Days have gone, when search engines used to rank a website as per the keywords used in it. It doesn’t mean keywords are not important, it’s still important only if it is used in a proper way.

The selection of keywords and their implementation is a crucial part of SEO. That we will see in our next chapter.

Nowadays people are seeking more accurate and complete information on their queries and in order to fulfil their needs, search engines have updated their entire algorithm.

Today hundreds of SEO factors determine which web page is more relevant to the search queries.

Typically the long contextual post performs best because it contains In-Depth information about the topic.

And this is what people and search engines are looking for.

So, if you really want to rank your webpage on the first page of SERPs then start writing valuable In-Depth content.

Along with this, the website having more relevant backlinks and popularity also drags search engine attention to rank it.

Ok, Let’s dissect even more

We all know websites are ranked on the basis of relevancy.

For example:

If you search for the term ‘best insurance policy’, first of all, Google looks for the pages that are closely related to the keyword and bring all of them on the SERPs.

In this case, Google came up with 7.54 billion results within 1.08 seconds.

page ranking in search results

and out of 7.54 billion results, Google picks up the best web pages and put them on the top of the search result based on:

  • Backlinks
  • Relevance
  • Freshness
  • Content
  • Page speed
  • Mobile-friendliness

BACKLINKS

Backlinks are one of Google’s most important ranking factors. However, it’s not all about quantity because not all backlinks are created equal.

The more natural backlinks you have from high-authority (trusted) websites, the more likely it is that you will rank higher in search results.

It is entirely possible for a page with a few high-quality backlinks to outrank a page with a large number of low-quality backlinks.

RELEVANCE

Search engines especially Google has invested in many technologies to help understand the relationships between entities like people, places, and things.

The search engine even understands what kind of information the searchers are looking for.

For example: When you type “How to start an online store”, the search engine will understand that you are looking for instruction on starting an e-commerce website thus it returns all the web pages that contain the guide on making an online store.

They even know that the meaning of ‘How to change a light bulb’ is the same as this ‘How to replace a light bulb’.

google related query

Source!!

It means relevance goes way beyond keyword matching. Now Google uses interaction data to assess whether search results are relevant to queries. In other words, are searchers finding the page useful?

FRESHNESS

Freshness is a query-dependant ranking factor. It mostly depends on what people are searching on the web.

For example, in the case of a query like “new movie,” freshness is important because searchers want to know about newly-added movies. That is most likely why Google prioritises newly published or updated search results.

impact of fresh content in search engine

There are some queries where the freshness of the results is mostly irrelevant, such as “how many continents are there?” The number of continents hasn’t changed in decades, so it doesn’t matter if the search results are from yesterday or 1990.

CONTENT OF A SITE

A web page can be authoritative, fresh, and relevant but it doesn’t mean it will be useful.

And if the content is not useful for the searchers then Big G is not going to rank it anywhere.

For example, let’s say you searched for ‘benefit of yoga’.

And opened the first result (Result A) which is written by the foremost expert on Yoga.

Since it was written by an expert it covers almost everything related to the topic, because of that, it got lots of backlinks.

but…

When you started reading, you found the content is unorganized, images are unrelated, topics are not systematic and lots of Jargon are used that you don’t understand.

organized vs unorganized webpage

What will you do?

Most probably you will leave the website immediately.

Now, contrast that with another result sitting on the second page of Google (Result B) which have relatively fewer links pointing toward it.

You found that web page HELPFUL…

…because it was well organized with great presentation, relevant images, and proper subheading.

Well, that page is going to rank higher on the basis of the ‘usefulness scale’. Even though the backlinks and authority of Result B are less than Result A.

Here Google measures ‘Users experience’. It analyzes how searchers are interacting with the search result and then rank them accordingly.

It’s a machine learning algorithm called RankBrain.

PAGE SPEED

Google has indicated site speed as an SEO signal used by its algorithm to rank the web pages (and recently they have given even more importance to Page Speed).

So make sure your website loads in under 2 seconds because a slow site completely hampers user experience which results in a high bounce rate.

You can check the speed of any web page in PageSpeed Insights, which also generates suggestions to make the page faster.

10 pagespeed insights

The hosting I use helped me a lot in improving my site speed from 4.3 seconds to nearly 1 second.

Along with the best hosting, you should also consider using CDN, Compressed images, Minify CSS, JavaScript, HTML and the fastest theme like Astra.

MOBILE FRIENDLY

Today 60% of Google searches are coming from mobile devices and mobile searchers are growing fast.

People love to access the internet while they are on the go, which means local SEO is going to become even more popular.

You can test the mobile-friendliness of any web page using Google’s Mobile-Friendly Test tool or the Google Search Console’s Mobile Usability report.

mobile friendly website

HOW DO PEOPLE INTERACT WITH SEARCH ENGINES?

Over the years human interaction with search engines has been evolved so much, you can say it is getting more specific.

However, the primary principles of conducting a search are still the same and it goes as:

  1. People experience the need for some information or help.
  2. Formulate that need in a phrase or question or even words, commonly known as ‘the query’.
  3. Enter that query in the search box of search engines like Google or Bing.
  4. The search engine will show some relevant results.
  5. Searchers’ will click on the result shown in SERPs and scan for the solution.
  6. If the searcher’s got what they were looking for then well and fine or else they will go back to the SERPs and click on another link to get the solution.

Generally, people have two types of queries:

1. Informational query: When people are seeking some information.

Such as the Best University of the world, Biography of Steve Jobs, Best SEO company, How to rank my website etc comes under informational queries.

2. To do something: When people are willing to do something.

Such as buying a movie ticket or buying a domain name comes under transactional queries.

In both types of queries, searchers look for the relevant result.

That is why I am repeating this, again and again, Build your website for users not only for search engines.

Even the search engines themselves are working after providing the relevant results to their users as fast as possible.

  1. Make your webpage for users, not only for search engine
  2. Create useful In-Depth content
  3. Make a site with easy navigation
  4. Don’t misguide users and search engine
  5. Use keywords to create descriptive meaningful content, titles, and URLs
  6. Avoid unnecessarily keyword stuffing
  7. rel = “nofollow” and rel = “canonical” tag should be used properly

HOW SEARCH ENGINES PERSONALIZE SEARCH RESULTS?

Open a new tab in your browser (an incognito window will be better ) and type “SEO Company”

what did you get?

List of all the SEO companies near your location right?… YES.

This is how search engines have evolved themselves.

They will automatically analyze your search history, browsing behaviour, your location, your interest and then combine all the data in order to provide you with the most accurate result within a second.

As I have mentioned earlier, people nowadays are getting more specific. Like if someone wants to start an online business, then their queries will be as:

  • How to start an online business in India?
  • How can I make a website to start an online business?
  • What are the requirements to start an online business?
  • Can online businesses make money?
  • Where to register an online business?

and so on…

So if you start writing In-Depth content which covers almost every detail about the topic will definitely get ranked because people like the website which provides almost every detail in one place.

And if they (people) found it worthy, your post will definitely get shared, resulting in driving more traffic to your website.

It indicates to the search engine that your web page has valuable content and your page will get ranked on the first page of SERPs.

It’s cool right…? Yeah..!!

Now, if you are wondering how you will find all the questions people can think of, here is a great website you can go with.

You just have to enter the word and you will get hundreds of questions related to that keyword. Choose some of them, analyze them and start writing your article.

But how you will choose a proper keyword and how you will properly implement it? Check it out in the next chapter.

FAQs and Most ASKED BASIC QUESTIONS ABOUT SEARCH ENGINES

The five elements of a search engine are documents, index, user query, ranking model and results page (SERP).

A browser (firefox, internet explorer, chrome) is a program to display websites. A search engine (google, bing, yahoo) is a particular website that provides you with search results.

Next Lessons

PREVIOUS

WHAT IS SEO

A comprehensive guide to SEO, website traffic, basic content optimization, on page and off page SEO. This lesson will also go over the concepts of quality and quantity traffic.

COMPLETE INTRODUCTION TO SEO

7 subtopics - 15:00 minutes
LESSON 1

Next

ADVANCE GUIDE ON KEYWORD RESEARCH

In this post, we'll define keyword research, why it's important, how to conduct research for your SEO strategy, and how to select the best keywords for your website.

A detailed guide on PROPER KEYWORD RESEARCH

9 subtopics - 20:00 minutes
Lesson 3

4 thoughts on “How Search Engine Works”

  1. pushpendra singh

    Great…!! that’s great advice good work, I read and also saw your every post, nice artical very usefull your post Thank you so much for sharing this and the information provide.

  2. I’m really enjoying the design and layout of your site.
    It’s a very easy on the eyes which makes it much more pleasant for
    me to come here and visit more often. Did you hire out a developer to create your theme?
    Superb work!

Comments are closed.

Sign up for our Newsletter

Fastcomet

Award winning hosting company for your website

Elementor

Best page builder for creating a stunning website

Astra

The most popular theme among webmasters