Search Engine
Basics How Search Engines Crawl, Index and Rank the Web
Search engine basics describe how a search engine discovers, organizes and ranks web content to answer a query. Crawlers find pages by following links and sitemaps, then the index stores what each page says. When someone searches, ranking algorithms sort that index to return the most relevant pages in a fraction of a second. This free guide explains each stage through live demos.
Written by Usman AliGrowth and SEO specialist · 7 years in SEO
Reviewed by Amir ShahidSenior SEO strategist · 9+ years in SEOSearch engines discover pages through links before their bots crawl each one. The words on every page go into an index, which the engine uses to rank matching pages for each query.
Search, end to end
The Whole Journey, in One Screen
Pick a stage or let it play. Each tab shows what a search engine is doing to one real page at that exact moment.
Google finds out your page exists
Google cannot visit a page it has never heard of. It learns about new pages mostly through links from pages it already knows, a bit like following road signs to a new shop.
In short: if no link leads to your page, Google may never find it.
Google's robot visits the page
A program called Googlebot asks your website for the page, the same way your browser does when you open a link. It only reads what it is allowed to open.
In short: the page has to open properly, or the robot moves on.
Google sees the page the way a visitor does
Many modern pages are assembled by code after they load. Google opens them in a browser of its own first, so it can read the finished page rather than the raw code.
loading recipe…
In short: text that only appears after a click can be missed.
Google files the page in its library
The important words on the page go into a huge index, much like the index at the back of a book. Later, Google can find every page about a topic in a split second.
In short: a page that is not in the index cannot show up in results.
Google puts the most helpful page first
When someone searches, Google compares every page that matches the words. It orders those pages by how well each one answers the question.
AI can quote your page in its answer
AI Overviews write a short answer from pages that are already in Google's index. Each answer links back to the pages it used as sources.
Feed a starter by mixing equal weights of flour with water. Discard part of it each day to keep it healthy. The bakery guide shows the full schedule.
In short: clear, well-organised pages are easier for AI to quote.
Definition
What Is a Search Engine, and What Is It Not?
A search engine is an automated software system that discovers, organizes and ranks web content, so people can find information by typing keywords or a question. It keeps a processed copy of those pages in an index, which it uses to return a ranked list for each query. Every lesson in search engine basics starts with this definition, because each later step builds on it.
Three abilities, working together
That definition is the short search engine overview most textbooks on information retrieval agree on. Three abilities have to work together for a search engine to function. Crawling is what finds pages across the web. The index stores what was found, while ranking decides which results come first. Without any one of them, you have a search box instead of a search engine.
The difference matters because it tells you where to put your effort. When a product shows results licensed from another company, being indexed by that company is what actually counts.
Search engine vs web browser
Google Search is the search engine, while Chrome is the browser Google makes. A browser runs on your device to request a page you already know. It then shows that page’s HTML on your screen. A search engine runs on the provider’s servers, where it builds a list of addresses from its index.
You reach one through the other, which is why the two get mixed up so often. Safari is another popular browser that works the same way. Every browser lets you choose Google or another provider as its default search engine.
| Tool | What it does | Examples | Own index |
|---|---|---|---|
| Search engine | Builds its own index to rank pages for a query | GoogleBingBrave Search | Yes |
| Web browser | Opens a page you already have the address for | ChromeFirefoxSafariEdge | No |
| Web directory | Lists sites that editors picked by hand | Early Yahoo DirectoryDMOZ (Open Directory)Curlie | No |
| AI answer engine | Writes an answer from pages it retrieves | AI OverviewsChatGPT searchPerplexity | Usually borrows |
The pipeline
How Search Engines Crawl, Index and Rank Pages
Every search runs as a pipeline, with each stage passing its work to the next. Each stage can only use what the stage before it handed over, so a page that fails early never reaches the later steps. This search engine workflow is the most useful idea for fixing any visibility problem.
A new page is born
A small bakery publishes a new guide to keeping a sourdough starter. A link from its homepage is the only way a search engine can learn that the new page exists.
link found · added to the visit listGoogle's robot comes knocking
Googlebot follows that link to the new page. Before asking for the page, it checks that robots.txt allows the visit. The page loads without an error, so it can move on.
page opened · no errorsThe page is read like a visitor would
Google runs the page in a browser of its own, so it sees the same text that a visitor sees on screen.
finished page · text visibleWords go into the index
Google picks out the important words on the page. Each word is filed in the index with a note pointing back to this page.
3 words filed in the indexFrom fourth place to first
When someone searches, Google compares every matching page. A clear answer, backed by links from trusted sites, lifts the guide to the top.
position #4 → #1Discovery: the engine learns a URL exists
Before anything can be fetched, the engine needs the address, a step often called URL discovery. Most pages are found through a link on a page that was already crawled. An XML sitemap can also add a URL to the queue, as can a submission in Google Search Console. A page with no links pointing to it is not penalised; it is simply unknown.
Crawling: a bot requests the page
Web crawlers, also called spiders or bots, are automated crawlers that take URLs from the queue before requesting them. Googlebot first reads robots.txt to see whether it may fetch the page. Only a 200 HTTP status code passes content forward to the next stage. Repeated server errors slow the crawler down across the whole site, which is why crawling basics matter to site owners.
Rendering: the engine runs the code
Many modern pages build their content with JavaScript, so the engine runs that code in a headless browser to see the final text. Loading the HTML, CSS and scripts together costs far more than reading plain source code. Google queues this work separately, which means text that appears only after scripts run can reach the index later than text in the source. Content placed inside an iframe belongs to another URL, so it may be indexed there instead.
Indexing: the words are stored
The rendered text is split into words, a step called tokenization, before those words are written into the index alongside metadata such as language. Images, videos and structured data are analysed too, so each type of content can surface in its own kind of result. Whenever it finds duplicate content, the engine picks one canonical URL from each group. Google also uses mobile-first indexing, so the version of a page its smartphone crawler sees is the version that gets stored. Because the search engine indexing process is selective, a page can be crawled cleanly yet still be left out.
Retrieval and ranking: results are ordered
When someone searches, the engine first fixes spelling. Query interpretation then expands the words to work out the intent. The engine pulls matching pages from the index. It then ranks them, mainly on relevance and quality. Results arrive in a fraction of a second because the heavy work of crawling and indexing happened long before.
Find the stage that failed first
When a page is missing from search, start at discovery and move forward until something breaks. A page blocked from crawling needs a very different fix from a page that was indexed but outranked, so guessing at content changes usually wastes time.
Worked example
One New Page, from Publish to Page One
Follow a single bakery page through every stage of the pipeline, including the one small mistake that would keep it out of search entirely.
The page goes live
A bakery publishes a page called “sourdough starter guide”. Nothing happens yet, because no search engine knows the address exists.
Googlebot finds the link
The homepage links to the new page. Googlebot recrawls the homepage, where it spots the new link. It requests the new page, which returns a 200 status.
The words are stored
Google renders the recipe card to read its full text. It then splits the text into single words called tokens, such as sourdough or starter. Each token is then added to its own posting list in the index.
It reaches page one
Someone searches “how to feed a sourdough starter”. A clear answer near the top, backed by two links from local food blogs, lifts the bakery onto page one.
One tag stops everything
With a leftover noindex tag, the first three stages still succeed. Indexing stops the page cold, so ranking never sees it.
Components
What Are the Main Components of a Search Engine?
The pipeline above shows what happens to a page, while these four parts are the machines doing that work. Every search engine architecture uses them, from Google down to a weekend coding project. Knowing these search engine fundamentals turns a vague problem into a specific one.
engine
Crawler
spider · bot · Googlebot- Runs at
- Stage 02 · Crawl
- In → out
- A queue of URLs in, fetched HTML pages out.
- When it fails
- Robots.txt blocks it or the server errors, so the page is never fetched.
Indexer and index
inverted index · database- Runs at
- Stages 03 and 04 · Render and index
- In → out
- Rendered pages in, a searchable inverted index out.
- When it fails
- A noindex tag or duplicate content keeps the page out of the index.
Query processor
query parser- Runs at
- Every search, before ranking starts
- In → out
- Raw typed words in, a structured request out.
- When it fails
- A misread intent means the right page is never even considered.
Ranking algorithm
relevance scoring- Runs at
- Stage 05 · Rank
- In → out
- Matching candidates in, an ordered results page out.
- When it fails
- The page is stored but weaker than rivals, so it sits too low to be seen.
// Keep scrolling to move through all four parts.// Auto-advancing. Tap a part to hold it.
A fifth layer, the user interface, is the search box plus the results page you see. It sits outside the retrieval machinery, yet it decides how answers are presented. These search engine components map neatly onto failures. A missing page usually failed at one part, such as the crawler never visiting or the indexer dropping it.
Query processing
How Search Engines Process Queries Before Ranking
Ranking cannot start until the engine understands what you typed. In a fraction of a second, query processing turns a few loose words into a structured request that the index can answer.
Parse the query
query parserThe query parser splits your text into tokens, keeping any phrase in quotes together. Operators such as site: or a minus sign are read at this step. Spelling is checked here as well, which is why a typo still finds the right page. A spoken question from voice search is first turned into text, then parsed in exactly the same way.
Normalize the words
tokenization, stemming, stop wordsTokenization breaks your text into separate terms the engine can match. Stemming maps words like running or runs back to one root, so a page about running still matches. Very common stop words such as the or of carry little weight on their own.
Work out the intent
search intentThe engine guesses what kind of answer you want. A query like weather tomorrow needs a quick fact, while best laptop for students suggests a comparison. Your location can shift that guess, as can the language you search in.
Match against the index
search query matchingThe processed terms are then looked up in the inverted index. The engine collects every page that holds those terms, often thousands of them, before the ranking algorithm puts them in order.
This is how search engines process queries for every search you run. It also explains a puzzle that confuses many people. Two queries that look different can return nearly the same results, because after parsing they ask the index the same question.
Index lab
Inside the Index: Why Search Is So Fast
A search engine never reads every page when you type a query. Try the trick it uses instead: look words up in an inverted index.
Type words from the three tiny pages on the right. Google never opens each page to check. It reads a word list instead, much like the index at the back of a book. Only the pages that contain all of your words are kept.
That huge scale is the reason the system is designed this way. Google reports that its Search index is well over 100,000,000 gigabytes (Google Search), so reading every page for every query would be impossible. The word the shows the other half of the idea: it appears in every document, so it tells the engine nothing about relevance. That observation led to inverse document frequency, which modern search engine technology still uses.
Ranking
How Do Search Engines Decide Which Pages to Show?
Retrieval narrows billions of pages down to a few thousand candidates. Ranking then puts those candidates in order, using signals that Google groups into five families.
| Signal family | The question it answers | Example signals |
|---|---|---|
| Meaning | What is the person really asking for? | Spelling fixesSynonymsSearch intent |
| Relevance | Does the page cover the topic of the query? | Keyword matchingHeadingsEntitiesFreshness |
| Quality | Can the page be trusted? | BacklinksAuthorityE-E-A-TContent quality |
| Page experience | Is the user experience good on every device? | Core Web VitalsWebsite speedMobile layoutHTTPS |
| Context | Who is searching, and from where? | LocationLanguageDeviceSearch historySettings |
The context family is where personalization happens. Your location, language, device and past searches can all shift the order, which is why two people typing the same query rarely see an identical page. Signed-in Google users can switch off results based on their activity, although location still shapes local searches.
Ranking mixer
Change the Weights, Watch the Order Change
Each type of query leans on different signals. Pick a query or drag the sliders to see why the same four pages swap places.
Change the weights, watch the order change
Each type of query leans on different signals. Pick a query or drag the sliders to see why the same four pages swap places.
These weights are illustrative and meant for learning only. No search engine publishes its real formula.
- 10.00guide.example.org › indexingA complete guide to search indexes
- 20.00newsdesk.example.com › core-updateCore update: what changed this week
- 30.00localseo.example.net › your-citySEO consultant in your city
- 40.00forum.example.com › threadForum: how does an index actually work?
Algorithms
Search Engine Algorithms in Three Layers
People talk about the algorithm as if it were one formula. In practice, search algorithms are layers stacked on top of each other, with the cheapest checks running first.
Keyword matching
Classic formulas such as TF-IDF or BM25 measure how well the words on a page match the words in the query.
Link analysis
PageRank, the idea Larry Page and Sergey Brin built Google on in 1998, treats each link as a vote. Modern systems still use descendants of it.
Machine learning
Learned ranking models such as RankBrain or BERT rely on semantic matching to compare meaning, so a page can match a query that shares none of its exact words.
Why nobody can give you exact weights
No outsider knows the precise weighting, which also changes from one query to the next. Freshness matters a great deal for news yet barely at all for a definition, so lists of search engine ranking factors with fixed percentages deserve some doubt. The ranking systems guide on Google Search Central describes real systems instead of guessing at a formula. That makes it a better starting point than any list of ranking factors.
Results
Inside the Search Engine Results Page
A SERP is no longer ten blue links. Organic results now share the page with ads, while answer boxes take up even more space above them. Ranking first does not always mean being seen first.
SERP features you will see
- Organic results are free listings ordered purely by the ranking system.
- Sponsored ads are pay-per-click listings bought through an auction, then labelled as ads.
- Featured snippets lift a short answer out of one organic result.
- People also ask expands into related questions.
- Local packs show a map with nearby businesses.
- Knowledge panels summarise facts about a well-known person or brand.
- Video results show clips from sites such as YouTube, often with key moments marked.
These SERP features change how many people scroll down to the blue links, which is why position alone is a weak measure of visibility. Click-through rate, the share of people who see your listing and then click it, tells you far more about how a result performs.
AI search
How AI Search Changes the Picture
AI answers sit on top of the same crawl and index, so the foundations in this guide still decide which pages get cited.
AI answers still start with the index
AI tools such as AI Overviews, ChatGPT search or Perplexity write generative answers in their own words. Even so, they still pull those answers from indexed pages. Google describes this as grounding the response in web results. If a page cannot be crawled or indexed, it cannot be quoted as a source either.
No special files required
Several popular claims about how AI search works deserve caution. Google’s guide to AI features says no special markup or AI text file is required to appear in them. Clear answers near the top of a page still do most of the work, backed by real expertise along with a layout the crawler can read.
The landscape
Types of Search Engines, and How They Differ
Search engines differ in how they collect pages and how they present answers. Most fall into one of six types.
Crawler-based
Bots crawl the web automatically, building the index without human help.
Google, BingDirectory
Editors sort websites into categories by hand.
Early Yahoo DirectoryHybrid
Crawled results mixed with curated listings or extra features.
Most modern enginesMeta search
Sends your query to several engines before combining their results.
DogpileVertical
Searches one kind of content only, such as papers or jobs.
Google ScholarAI answer engine
Writes an answer from pages it retrieves, with links to sources.
PerplexityWho owns the index
Popular Search Engines, and Who Actually Owns an Index
Hundreds of search engines exist, yet only a handful run their own index. The rest license results from one of them, which decides what you can find.
Pick a search engine to see whose index answers you
| Search engine | Index | Known for |
|---|---|---|
| Own | The largest index in most countries | |
| Bing | Own | Also powers Yahoo and much of DuckDuckGo |
| Brave Search | Own | Independent index launched in 2021 |
| Mojeek | Own | Small UK engine running its own crawler |
| Baidu, Yandex, Naver | Own | Market leaders in China, Russia or South Korea |
| DuckDuckGo | Mostly Bing | Search without personal tracking |
| Ecosia | Partner indexes | Funds tree planting from ad revenue |
Google and Bing lead most people’s list of the five best search engines. DuckDuckGo suits people who want privacy, while Brave Search or Ecosia fill the last spots. When you pick one, ask whose index sits underneath, because that decides what you can find.
History
A Short History of Search Engines
Search went from listing file names to writing full answers in about 35 years. These eight moments shaped how it works today.
Archie indexes file names on public FTP servers.
Yahoo launches a web directory that people build by hand.
AltaVista offers full-text search of the web.
Google launches with PageRank as its core ranking idea.
Microsoft launches Bing to compete with Google Search.
RankBrain brings machine learning to Google ranking.
BERT helps Google read whole sentences in context.
AI Overviews add written answers above results.
Everyday use
7 Common Uses of Search Engines
Search engines matter because the web has no shared table of contents. They are the main way a page written by a stranger reaches someone who never knew it existed.
- 01Find a quick fact or definition
what is an inverted index - 02Learn how something works
how do search engines work - 03Follow breaking news
google core update - 04Compare products before buying
best laptop for students - 05Find a place nearby
library near me - 06Reach a site you already know
bing webmaster tools - 07Research for school or work
site:edu search techniques - Next stepSearch better with five operators→
Search skills
Search Better with Five Operators
Knowing how an engine reads your query helps you find things faster. These operators work in Google and in most other engines.
"how search engines rank"Quotes tell Google to match the exact phrase only.site:gov crawlingLimits results to one site or domain type.robots.txt -wordpressA minus sign leaves out a word.filetype:pdf indexingThis operator finds a specific kind of file, such as a PDF.ranking OR algorithmThis operator matches pages with either word.It also helps to judge the quality of whatever you find. Before you trust a page, check who wrote it. A recent update date is a good sign, as are clear links to its sources. University libraries teach the same habits in their research guides. These habits work just as well for everyday questions.
For site owners
Get Your Own Site Found in 15 Minutes
Most visibility problems come from a single failing stage. Work through these checks in order before you change any content.
- Open the page and confirm it returns a 200 status, not a redirect chain.
- Read /robots.txt and make sure no Disallow line blocks the page.
- Check that your XML sitemap lists the page and loads without errors.
- Inspect the URL in Google Search Console to see its index status.
- Compare your declared canonical with the one Google selected.
- View the rendered HTML to confirm the main text appears without clicks.
- Search for an exact sentence from the page to see whether it shows up.
- Make sure at least three indexed pages link to it with clear anchor text.
- Add the site to Bing Webmaster Tools for a second opinion.
- Review impressions and clicks in the Performance report after two weeks.
SEO basics
Search Engine Optimization Basics: How to Rank Higher
Search engine optimization works with the pipeline rather than against it. It is the part of digital marketing that earns unpaid visits, so patience counts for more than budget. Each habit below supports one stage, which is why the order you fix things in matters more than any single trick.
Keyword research
Find the words people actually type before you study the results page for each one. Keyword research shows which questions deserve their own page and which ones are already answered well.
Helps retrievalSearch intent
Match the page type that already ranks. If the top results are step-by-step guides, a sales page will struggle however carefully it is written.
Helps rankingOn-page SEO
Write a clear title tag for each page. Add a meta description that answers the query. Use one H1 at the top, with clear subheadings below it. These meta tags shape how your listing looks on the search engine results page.
Helps the SERP listingTechnical SEO
Each page should return a 200 status. Link to it from at least one other page. Make sure it also appears in your XML sitemap. Then check that your robots.txt file does not block it. A desktop site crawler reveals the same problems a search bot would meet.
Helps crawlingOff-page SEO
Backlinks from relevant sites still act as votes of trust. The dependable way to earn a backlink is to publish something specific enough that other writers want to cite it.
Helps rankingMeasure the results
Organic traffic shows whether the work paid off. Google Search Console shows how many clicks each page earns. It also flags any indexing errors it finds. Bing Webmaster Tools gives you the same view for Bing.
Covers every stageBetter search visibility rarely needs anything exotic. Search results reward pages that are easy to reach. The page must also be clearly useful for the exact query. Fix the earliest broken stage before you polish the later ones.
Four controls
Robots.txt, Sitemaps, Noindex and Canonicals Compared
Each of these tools controls exactly one stage of the pipeline. Mixing them up is one of the most common reasons a page disappears from search, or refuses to leave it.
| Control | What it controls | What it cannot do | Use it when |
|---|---|---|---|
| robots.txt | Whether a crawler may fetch a URL | Remove a page that is already in the index | You want bots kept away from admin areas or endless filter pages |
| XML sitemap | Which URLs you want the engine to discover | Force a page to be crawled or indexed | You publish new pages often or run a large site |
| noindex tag | Whether a fetched page may be stored in the index | Work on a page the crawler is blocked from fetching | A page should stay live for visitors but out of search |
| Canonical tag | Which version of duplicate pages should rank | Bind the engine, since Google treats it as a strong hint | The same content lives at several URLs |
The classic mistake is blocking a page in robots.txt while also adding a noindex tag. The crawler never fetches the page, so it never sees the tag, which means the URL can linger in results for months.
Problem finder
Why Isn't My Page Showing Up?
Pick the problem that sounds most like yours. You will see where the pipeline usually breaks. Each stage lists its most common causes, along with what you should check first.
The engine never reached the page
Common causes
- No internal link or sitemap entry points to it
- A robots.txt rule blocks the path
- The site is new, so crawlers have not visited yet
Check first
- Inspect the URL in Google Search Console
- Link to it from a page that is already indexed
- Test the path against your robots.txt
Google fetched it, then chose to leave it out
Common causes
- A noindex tag left over from staging
- Google picked a different URL as the canonical
- The content looks thin or duplicated
Check first
- Search the page source for noindex
- Compare your canonical with the one Google selected
- Make the page clearly different from similar ones
The page is stored, but it is outranked
Common causes
- It targets a phrase people rarely search
- Stronger pages already answer the query well
- Very few links point to it
Check first
- Search the exact page title in quotes
- Look at impressions in the Performance report
- Add internal links with clear anchor text
Something changed in the comparison
Common causes
- A core update re-weighted ranking signals
- A site change broke crawling or redirects
- Competitors refreshed their content
Check first
- Match the drop date against Google update history
- Re-test status codes on your key pages
- Review what now ranks above you
Google rewrote how your listing looks
Common causes
- The title tag does not match the main heading
- Google adapted the title to fit the query
- The meta description is missing or generic
Check first
- Align the title tag with the page heading
- Keep titles specific, roughly under 60 characters
- Write a description that answers the query
The answer is hard to quote
Common causes
- The direct answer sits deep in the page
- Facts have no named source
- AI crawlers are blocked in robots.txt
Check first
- Put a short answer right under each heading
- Name primary sources next to key facts
- Allow the AI crawlers you want in robots.txt
Try it yourself
Test a Page Before Google Does
Switch conditions off to see exactly where a page would stop. You can run the same checks on your real site with these free tools.
Break a page on purpose
Turn off any switch below to see where Google would get stuck. It stops at the first problem it meets, exactly like the real thing.
- 01Discoverfinds it
- 02Crawlvisits it
- 03Renderreads it
- 04Indexsaves it
- 05Rankshows it
Does it open? ✓ yes
Can Google read it? ✓ yes
Allowed in results? ✗ no, noindex tag
→ Google visits it but will not show it
Pipeline simulator
Run a page through each step and see exactly where it gets stuck.
"exact words" this wording only
site:gov government sites only
filetype:pdf PDF files only
-word leave a word out
Search shortcut builder
Make precise searches without memorising any special codes.
Your rule: keep out of /private/
/blog/post ✓ allowed
/private/notes ✗ blocked
robots.txt checker
Paste in your robots.txt file to see which pages its rules block for search robots.
Choose a path
Two Ways to Learn, One Free Course Built by Working SEO Experts
Track one is a search engine explained from first principles. Track two turns the same ideas into practical steps for your own website. We built this course from real, hands-on SEO work, then reviewed every lesson before it went live. Both tracks teach the same search engine basics, each from a different starting point.
Understand how search works
For students, teachers or curious minds who want to know what happens after they press Enter.
- Search engine basics
- What is a search engine?
- How search engines work
- Components of a search engine
- Crawling
- Web crawlers
- Indexing
- Search engine ranking
- Search engine algorithms
- The SERP
- AI search
- Search skills
Get your website found
Built for site owners and new SEOs who want their pages to appear in search results.
- SEO basics
- How to learn SEO
- Robots.txt
- XML sitemap
- Googlebot
- Check if a page is indexed
- Canonical tag
- Noindex tag
- Google ranking factors
- PageRank
- Search engine positioning
- Positioning example
Myths
Common Search Myths, Checked Against the Source
Blocking a page in robots.txt removes it from Google.
A robots.txt block only stops crawling, not indexing. To keep a page out of the index, use a noindex tag on a page the crawler can still fetch.
Submitting a sitemap gets every page indexed.
A sitemap is only a hint that helps Google find your URLs. Google still decides whether each page is worth storing.
Structured data is a direct ranking factor.
It can make a page eligible for rich results, which changes how a listing looks rather than where it ranks.
Longer pages always rank higher than short ones.
Google says there is no ideal word count for a page. Length only helps when the topic genuinely needs it.
Running paid ads improves your organic rankings too.
Ads and organic results run on separate systems, so buying one does not lift the other.
Glossary
Search Engine Terms Explained Simply
These are the words you will meet in every guide to the basics of search engines. Each definition is short enough to remember, yet precise enough to use when you read documentation or talk to a developer.
| Term | What it means |
|---|---|
| Web crawler | A program, also called a spider or bot, that follows links to fetch pages. Googlebot is the best-known example of a web crawler. |
| Crawl budget | How many URLs an engine is willing to fetch from one site within a given period. |
| Index | The engine’s stored copy of processed pages, which it searches instead of the live web. |
| Inverted index | A lookup table that lists, for every term, the pages that contain it. |
| Tokenization | Splitting text into separate terms that the index can store. |
| Stemming | Reducing words to a shared root, so search and searching match each other. |
| Stop words | Very common words, like the or a, that carry little meaning on their own. |
| Retrieval model | The method that decides which indexed pages become candidates for a query. |
| Relevance scoring | Measuring how well a page answers a query, for example with the BM25 formula. |
| PageRank | The link-based scoring method Larry Page and Sergey Brin published in 1998. |
| Backlink | A link from another website to yours, often read as a vote of trust. |
| Canonical URL | The preferred version of a page when duplicates exist. |
| SERP | The search engine results page you see after running a query. |
| Organic search results | Unpaid listings that the ranking algorithm orders, as opposed to ads. |
FAQ
Questions People Ask About Search Engines
What do the basics of search engines include?
They cover the steps every engine follows. An engine first has to find pages before it can crawl them. It then stores those pages in an index, ready to rank them for each query. A good search engine 101 lesson also explains the results page, because that is where the work finally shows up.
How do search engines work in simple terms?
Bots follow links from page to page to find new content. The engine stores the words from those pages in an index. A ranking system then picks the most useful matches when you search.
What is the difference between crawling, indexing and ranking?
Crawling is when a bot fetches a page from your server. Indexing then processes that page so it can be stored. Ranking compares indexed pages for a specific query to decide the order.
What are the four parts of a search engine?
The crawler fetches pages from across the web. The indexer then stores what the crawler found. When you search, the query processor reads your words. The ranking algorithm then decides the order of the results.
Is Google a search engine or a web browser?
Google Search is a search engine, while Google Chrome is the web browser. That is why you can still use Bing or DuckDuckGo inside Chrome.
What does SERP stand for?
SERP stands for search engine results page, the screen you see after running a query. Today it mixes organic links with ads and other features, such as AI Overviews.
What is query parsing in search?
Query parsing is the first step after you press search. The engine breaks your words into separate terms. It also reads any operators you used, such as quotes. Spelling gets corrected before the engine checks the index.
What are the 5 best search engines?
For most people the shortlist starts with Google and Bing. DuckDuckGo usually comes next, followed by Brave Search with Ecosia close behind. Only some of them keep their own index, so results can overlap.
How many search engines are there?
Hundreds exist, yet only a handful run their own large web index. Google and Bing run the largest ones. Brave Search keeps its own index too, as do regional engines like Baidu in China. Most other engines license their results from these.
What is the difference between a browser and a search engine?
A browser such as Chrome opens a page you already have the address for. A search engine such as Google finds that address for you by searching its index.
What are 7 common uses of search engines?
Most people search to look up facts or learn how something works. Many also follow the news or compare products this way. Finding places nearby is another big use, as is research for school.
How do I rank higher on Google?
First make sure Google can reach the page. Then match the type of page that already ranks. Answer the question fully, close to the top of the page. Links from relevant sites help once those basics are in place.
Why is my website not showing up on Google?
Usually Google has not found the page yet. Other common causes are a robots.txt block or a leftover noindex tag. The URL Inspection tool in Google Search Console shows which stage failed. New sites can also take a few weeks before pages appear.
How long does it take for a new page to be indexed?
It ranges from a few hours to several weeks. Pages linked from an established site tend to be found sooner. A sitemap entry helps the engine notice the URL, although it does not guarantee indexing.
Is Google the only search engine that matters?
Google handles most searches in many countries. Bing still matters because its results also power Yahoo. Some countries prefer a local engine over Google. Baidu leads in China, while Naver is strong in South Korea.
How do I learn SEO as a beginner?
Start by learning how a search engine crawls pages before it ranks them. Then practise on a real site with Google Search Console open beside you. Track two of our free course follows exactly that order.
Do AI Overviews change the way search works?
They change how answers are presented, not how pages are found. AI Overviews draw on pages that are already crawled and indexed, so the same foundations still decide who gets seen.
How do search engines personalize results?
Search engines adjust results using context such as your location, language and device. Google can also use your past searches when you are signed in, so two people may see a different order for the same query.
Recap
Search Engine Basics in One Page
If you remember one idea, make it this: search runs as a pipeline. Each stage has to succeed before the next one can start. So every fix begins with finding the stage that failed. These search engine fundamentals hold for Google and Bing, as well as for AI answers.
Free PDF
Take the Whole Guide with You.
Get every diagram from this page in the free search engine basics PDF, along with a one-page cheat sheet. It is ready to use for study or with your team.