Queue for indexing. How to request pages to be rescanned. Foreign search engines

For the convenience of webmasters and search engines, a special sitemap format has been developed - sitemap. This is a list of links to internal pages of the site, presented in XML format. This format is also supported by Yandex. In a special section of the service you can download a sitemap for your website. This will allow you to influence the priority of the robot crawling some pages of your site. For example, if some pages are updated much more often than others, you should indicate this information so that the Yandex robot plans its work correctly.

Yandex indexes the main types of documents distributed on the Internet. But there are restrictions that determine how the document will be indexed, and whether it will be indexed at all:

• A large number of cgi parameters in a URL, a large number of duplicate subdirectories, and too much overall URL length can lead to poor document indexing.

  The size of the document is important for indexing - documents larger than 10MB are not indexed.

  Flash indexing:

  1. *.swf files are indexed if they have a direct link or are embedded in html with object or embed tags;

     if the flash contains useful content, the original html document can be found by the content indexed in the swf file.

In PDF documents, only text content is indexed. Text presented in the form of pictures is not indexed.

Yandex correctly indexes documents in the Open Office XML and OpenDocument format (in particular, Microsoft Office and Open Office documents). But keep in mind that implementing support for new formats may take some time.

It is acceptable to use tags And , the Yandex robot indexes the content loaded into them and allows you to find the source document by the contents of the frames.

If you have overridden the server behavior for non-existent URLs, make sure that the server returns a 404 error code. When the search engine receives a 404 response code, it will remove the document from its index. Make sure that all necessary pages on the site display the code 200 OK.

Yandex Webmaster allows you to:

• add sitemap;
• update;
• delete;
• analyze.

How to add a Sitemap to Yandex Webmaster

To add a Sitemap file to the queue for indexing, you need, as always, to go to the Webmaster service - webmaster.yandex.ru. Further:

1. In the Indexing section you will find the Sitemaps option.
2. Add the full address of the Sitemap files, for example, https://site/sitemap.xml.
3. Click Add.
4. Wait while Yandex Webmaster checks the file.

After adding, the file is queued for processing. The robot will add it within two weeks. Each added file, including those attached to the Sitemap index, is processed by the robot separately.

After loading, opposite each sitemap you will see one of the statuses in Yandex Webmaster:

Update Sitemap

When changes occur to the Sitemap, it does not need to be deleted from Yandex Webmaster and added again. The search engine itself checks it for changes and errors from time to time.

If your Sitemap is not indexed, then this can be speeded up. You need to go to the “Indexing” section, then to the “Sitemap files” item. Click on the update icon opposite the site map. Yandex will download the data within three days. You can use the function up to 10 times for one host.

Once you have used up all attempts, the next one will be available 30 days after the first. The exact date is displayed in the Webmaster interface.

Remove Sitemap from Webmaster

In the Yandex.Webmaster interface, you can delete those files that were added on the “Sitemap Files” page:

If a directive was added to the Sitemap in your robots.txt file, remove it. After making changes, information about the Sitemap will disappear from the robot and Yandex.Webmaster database within a few weeks.

Formats supported by Yandex Webmaster

Yandex accepts Sitemap in the following formats:

XML Sitemap format

It is preferable to transmit in XML format, because it can contain additional data.

Yandex supports the Sitemap protocol. Use this format if you want to report:

• date of last page update (lastmod element);
• page change frequency (changefreq element);
• page significance (priority element) - the robot loads pages one by one, taking into account the presence and value of the coefficient from 0.0 to 1.0. Specify the coefficient for those URLs that are most important to the site.

https://сайт/ssylki-v-yandex-webmaster/ 2018-06-06 monthly 0.8 ...

TXT format Sitemap

In this format, you can only transmit page addresses. That's all. Yes, sadness and sorrow, better use XML :)

Use this format to report page addresses only.

https://site/ssylki-v-yandex-webmaster/ https://site/zerkala-sajta-v-yandex-webmaster/

To transmit information about videos, specify links to XML feeds in the Sitemap (see details in the Ya.Video service Help).
Search does not support sharing image, RSS, and Atom feed information using a Sitemap.

When is a Sitemap needed?

Yandex is developing special algorithms by which the indexing robot learns about the site. For example, using internal and external links - moving from one page to another. Sometimes the robot may skip pages. Use Sitemap if your site:

• a large number of pages;
• individual pages without navigation links;
• deep nesting.

How to create a file

Add Sitemap.xml to Yandex Webmaster - video

Typically, the average period for indexing by Yandex is 1–2 weeks.

Some blogs are indexed much faster. The posts that I publish on my blog are indexed by Yandex within two hours, or even within two minutes. But there are exceptions.

This happened with two pages " How to write texts to get to the top?"(04/14/2011) and " iForum 2011. What happened this year"(04/18/2011). What's wrong with them? Why aren't they in the index? So much time has already passed.

I have already reposted it on Twitter and posted links on two other blogs and used the add form //webmaster.yandex.ru/site/add.xml.

Hello.

Please tell me what could be the reason.

Sincerely yours, Ilya.

Plato's answer soon came:

Hello, Ilya!

The specified page has already been indexed by the robot and should appear in the search results after the next update of the search databases. Please wait a few more days.

—
Sincerely, Platon Shchukin
Yandex support service
help.yandex.ru

Honestly, I was comforted by this answer. But after updating the Yandex search results, which occurred on April 29, 2011, I became concerned again. These pages never appeared in search results. But why? What's wrong with them? Promotion is inevitable only with indexed links. Otherwise, they are simply not taken into account, although they are posted on the page.

Google, by the way, indexed both documents.

Well, okay, while Yandex is updating its search results database, I bring to your attention a wonderful video about future technologies made on glass. In principle, I think websites will also develop in this direction. In particular, interfaces and navigation Already partly looks like this now.

P.S. I checked on April 30 - the pages are already in the Yandex index. A copy of these pages was saved on April 24, 2011. Hooray!

Things don't always look good with new sites. Yandex indexes them very slowly. Yandex.Webmaster says:

Your site is not indexed by Yandex. (The site has been successfully added to the indexing queue)

If the site is new and was not previously known to the robot, then after adding it to the Yandex.Webmaster service it will begin to be indexed and then appear in the search. This usually happens within two weeks from the moment the site is added.

Yandex.Webmaster writes that the site is not indexed because it is new, but at the same time it can index 2 pages. That is why, for the “Promotion Is Inevitable” competitions, competitions are held in Russian Google.

When adding a new site to Yandex.Webmaster, a message appears that the site has been added to the queue for indexing. After two or three days there are already changes:

The site has started to be indexed. Pages should appear in search after 1-2 updates to the search database. The search may contain pages indexed by a fast robot.

Due to the delay of the latest updates, Yandex is asked questions to which it gives equally informative answers

As for the statistics of ups, it is displayed in the table.

How to speed up site indexing

And finally, a small life hack on how to speed up site indexing. You just need to get a couple of bold external links to the page you need (for example, a new article on your blog). It is important that the links come from trusted sites and popular (thematic) groups of social networks, where

In this instruction we will consider the issue of adding our new site for indexing to various search engines.

I decided to mention both popular search engines and those that you may not have heard of.

Site indexing in Yandex

To add a site for indexing, just enter the url of the main page and the captcha. Captcha is a few numbers that protect against automatic registrations. After you click the “add” button, several options for the development of events are possible.

1) The message “your site has been added” signals the successful addition of a resource to the queue for indexing in Yandex.
2) If the message “Your hosting is not responding” appears, it means that your server is down at this moment. You can try adding a site later or find better hosting.
3) But if a message appears that “the specified URL is prohibited from indexing,” then things are bad. This indicates that sanctions have been imposed on your resource in the form of a site ban. It is quite possible that the domain you purchased already had a website that received sanctions. Using addurl, webmasters often check sites for bans in Yandex.

Site indexing in Google (Google)

The next most important search engine for our site is Google. The process of adding a site to indexing in Google is exactly the same as on Yandex. Google also has its own adurilka, which is located at: https://www.google.com/webmasters/tools/submit-url.

You also need to enter a captcha when adding. But there are two differences. If in Yandex you can simply add a URL without unnecessary movements, then in Google you need to be logged in to your account. Otherwise it won't work. Accordingly, if you don’t have an account there yet, you will have to create one. The second difference between Google and Yandex is the speed of indexing. Google indexes websites very quickly.

Site indexing in Rambler (Rambler.ru)

Of course, Rambler is not what it used to be, as many will say, and provides very little traffic. And anyway, why neglect them? The process of indexing a site in Rambler is the longest among other domestic search engines. And his adurilka has not been working for a long time, it was at: robot.rambler.ru/cgi-bin/addsite.cgi

He has been using the Yandex search database for a long time. Therefore, in order to get into the Rambler.ru index, it is enough to add to Yandex.

Site indexing in Mail.ru (Mail)

The Mail.ru search engine also has a webmaster's account. Adding a site for indexing in Mail.ru occurs through the addurl form, which is located at: go.mail.ru/addurl

Moreover, like Google, in order to apply for indexing, you need to create your own account and log into it. It won't work out otherwise. Mile has recently been trying to develop his own tools for webmasters.

Above we looked at the main domestic search engines in which we would like to index our site. I will give the following PS more for your general SEO erudition than for specific actions.

Search engine Aport.ru (Aport)

Aport.ru was once a search engine, with its own index database and adurilka. Now it has been turned into a product search engine in which you can compare prices for goods and services.

Search engine Nigma.ru (Nigma)

Nigma.ru is our Russian intelligent search engine. The total volume of its traffic is about three million requests per day. Obviously, traffic from Nygma should not be neglected. You can add your site for indexing in Nigma on the page nigma.ru/index_menu.php?menu_element=add_site.

Media navigator Tagoo.ru

The Tagoo.ru system is a media search engine that searches for media data. This includes music content, videos and programs. In order for your site to be indexed by the Tagoo system, you need to use the add form: tagoo.ru/ru/webmaster.php?mode=add_site.

Search engine Turtle.ru (Turtle)

The international search engine Turtle (Turtle) searches across the CIS countries in any language. For indexing, resources located in such domain zones as: ru, su, ua, am, az, ge, by, kz, kg, uz, md are accepted. To add a site for indexing in Turtle, you need to use Aduril: http://www.turtle.ru/add.html. It is advisable to wait for the message that your site has been accepted. It may not be added, but you won’t even know.

Foreign search engines

If the above were domestic search engines, then below there will be a list of foreign search engines.

Search engine Yahoo.com (Yahoo)

Search engine Bing.com (Bing)

The Bing search engine is owned by Microsoft and was created to replace Live Search. Microsoft hopes that the new brainchild will be much more popular than its predecessor. If you want your Bing.com site to be indexed, you can do so at http://www.bing.com/toolbox/submit-site-url.

Ukrainian search engines

And in conclusion of my review, I will cite two popular search engines in Ukraine.

Ukrainian search engine Meta.ua (Meta)