How do search engines work? Indexing, crawling, and ranking

How do search engines work?

It goes without saying that various features of search engines have made our lives easier. From checking the weather to setting mobile alarms, search engines are being used everywhere. People around the world perform an average of 3.8 million searches every minute on Google alone. As of today, it has reached 5.6 billion. The amount of traffic and challenges a search engine has to face on a regular basis can be estimated from Search Tribune’s statistics. But have we ever thought about how this search engine works?

Also Read: What is the keyword ? Why is keyword important?

In today’s post we will try to know, “How search engines work”? But before that, we will discuss some important terms for the readers to understand which they often do not understand. Our writing topics are well organized. If you have prior knowledge of a subject, you can easily skip it. So hopefully there will be no cause for irritation. We write content with each of our audiences in mind. Everyone holds Google universally as a search engine. So Google will dominate our discussion.

How do search engines work?

Search engines have three primary functions. They are – crawl, index, and rank. A crawling system uses a search engine to first find information that is already indexed. Then Googlebot decides who should be shown first and who should be shown later. It is better to clear another thing, for the convenience of the reader. Every search engine has its own crawling robots. They rank content based on pre-defined ranking factors. One such bot is Google Bot. For example, Moz.com has Rogerbot, Bing has Bingbot.

We understand that some terms may confuse readers. For that we will return to the detailed discussion about all of them.

What is Search Engine Crawling?

The direct Bengali meaning of crawl is – to crawl. The meaning of the word is much the same in the case of search engines, that is, when you search for a keyword, first of all, Google starts looking for the results that will be good for you from the information it has already indexed. This searching process is called crawling. And the one who crawls is called ‘Crawler’ in many cases is called ‘Spider’. The one that crawls for Google is called ‘Google Bot’. Crawling can be anything. It depends on your search.

Google bot first targets some web pages by Fetching. Especially those whose URLs are brand new. Who first identified them and stored them in caffeine. Caffeine is a huge URL repository where new URLs are stored. Google Caffeine was created in 2010. That story can be told another day. I don’t want to be the cause of the reader’s annoyance by saying all together. Caffeine saves their URLs beforehand so they don’t take long to show up in search results. For this reason, caffeine is saved by crawling in advance.

What is Search Engine Indexing?

Search engine indexing is the process of storing search engines in advance to facilitate the display of their search results. It’s like arranging books in a library. Just as it is convenient to know where a book is if you organize the books in advance, it is very convenient for the search engines to show the results later if they are indexed in advance. Every search engine pre-indexes for this. Hope the reader has understood what search engine index is.

What is Search Engine Ranking?

When a user searches for a keyword, the search engine refines, expands and provides the most relevant data according to the keyword. Most search engines, including Google, have rules about how they will show your content first. And these rules are called ranking factors. Generally Google has 217 ranking factors.

But it is very variable. Sometimes less and sometimes more. But speaking from my vantage point, there are more than 200 ranking factors. And after going through this process the whole process of who will show first and who will show last is called ranking factor. Another thing to say in this regard is that-

The earlier the search engines show the results, it means that Google considers the most relevant results for you.

We will discuss some of the top ranking factors in one of our next posts. For those who are more interested in SEO, good things are waiting for them.

How do search engines organize information?

The search engines to search for something have already prepared and indexed the information for you. For the convenience of all of us, we have arranged today’s post by following Google. In other words, we have shared the information according to the Google search engine. Although I’ve said it before, I’ll say it again. Many may have come here directly without reading from the beginning. I told them again. According to Google, they crawl over 100 billion web pages and pre-organize the search results for you.

Search basics

The crawling process starts with pre-listed web addresses and sitemaps provided by website owners. I feel the need to add another information for those who are new. Sitemap is like a file manager. The file manager in your mobile is set like -where video files, where audio files. Similarly, the file name of the web site is sitemap. Through this, search engines get an accurate idea of where your web site has any data, which posts there are. Which is later matched according to keywords and shown in search engines.

If the reader pays attention, one of the ‘means’ for search engines to know about your web site or web pages is the sitemap. Through this sitemap, search engines are able to keep an impression on your website. The main purpose of talking so much is – if any information on your web site is updated, the only way to know it is this site map.

After that, the computer program determines what information the search engines will find. Accordingly, they collect your website information. Now if you say that my web site has some very valuable information. Which I don’t want search engines to read. What should I do in that case?

Thank you very much if you have any questions like this. Now to your answer. Yes, the search engines have set some rules for you the reader. They named it – ‘robots.txt’. With this ‘robots.txt’ you will be in full control of your website. That is, you can tell the search engines in advance which web pages can be searched and which web pages cannot be searched.

Since we have kept Google as the search engine of today’s discussion – it is better to say one more thing. That is the search console, through which you will get some more important controls of your website.

For example, suppose you have made a new change to your website. or moved some web pages elsewhere. In this case, the problem is that – since Google has already indexed a structure of your website, now when it is necessary to show this same result, Google will not find the previous result. Two reasons can be put forward.

First, we discussed earlier the idea of showing Google results from a pre-indexed or sitemap. But Google does not know when you have changed the structure of your sitemap. Second, you haven’t told Google that you’ve changed your website. That is, Google knows nothing about the new sitemap. You also didn’t ask Google to re-crawl.

The solution to all these problems is to notify Google whenever you make changes to your website by updating the sitemap. Or if the site structure doesn’t change drastically and everything is sorted, it doesn’t cause problems. And you can resort to ‘Google Search Console’ to control this whole process. Another thing to note is that Google doesn’t charge you extra for re-crawling the sitemap over and over again.

Finding information by crawling

The number of web pages is constantly increasing in the world. According to Citify data, an average of 54,7200 web pages are created every day. Readers can understand how much the need to create web pages is increasing. It seems that the number of books in the library is growing exponentially. Think for a moment, readers – if suddenly a large number of books are sent to a library, then the librarian does not monitor them or register them according to the ISBN number, then how much chaos will be created? It would be wise not to dwell on that vain thought.

But Google is not like the librarians of the government libraries of our country. They index the right information at the right time unless there is a block from the site. They named the software they developed for this – “web crawlers” whose job it is to index the ever-creating public accessible web pages. You will understand better the day I discuss about ‘robots.txt’ why I said public accessible.

In this case, I can’t help but add something more. When Google crawls a specific web page of your website, it doesn’t just crawl that specific web page. Rather, as many links and backlinks, inbound links are added in that web page, they are scrolled. Yes, but only those that are ‘robots.txt’ allowed. Thus they fill their database by scrolling new web pages with their “web crawlers”.

Organize and index the information

Let’s take a look at what Google says in this regard-

When crawlers find a webpage, our systems render the content of the page, just as a browser does. We take note of key signals — from keywords to website freshness — and we keep track of it all in the Search index.

Note an important information here ‘We take note of key signals’ means they are collecting some comments about your web page. And they also say that in that note, they collect all kinds of data from keywords to the freshness of the website, so that they can easily show the most relevant results when someone searches later. They are saying more strictly that – ‘we keep track of it all in the search index’ means they are keeping all kinds of information in consideration and monitoring.

Google searches with web scrollers as if they read every word by word.

I’m keeping it here for now. In the next post we will discuss in detail – the topic of ‘How Search Algorithms Work’. Until then stay well and stay healthy. And if you like our content, please comment. Let me know if you don’t like it. Because we respect comments. Good luck reader. I am leaving today with the hope that your life will be beautiful.