HTML programming lessons from scratch. Modern HTML lessons for beginners. What do you need for work?

Good afternoon friends! For a very long time now, blog readers have been asking me to make a post about how to get rid of replytocom. This topic is relevant, because this results in a huge number of similar pages with comment parameters in the index. The thing is that many blogs are created on the WordPress CMS, which suffers from duplication of comments. Therefore, I decided to give all the information not only about replytocom. In this article, I explain the concept of duplicate content, give a method for finding duplicate pages on a site, and explain why you need to get rid of them. And in the next post I will show how to remove them.

Duplicate content on the site

Takes can be different - quiet and contagious. 🙂 Seriously, a duplicate page is a site document that may have completely or partially the same content. Of course, each take has its own address (page url). A clear duplicate page may occur for the following reasons:

• artificially created by the site owner for special needs. For example, printable pages that allow the user of a commercial site to copy information on a selected product/service.
• are generated by the resource engine, since this is built into their body. Some modern CMSs display similar pages with other URLs, located in their own different directories.
• appear due to errors of the webmaster who manages the site. For example, a resource may have two identical main pages with different addresses (for example, site.ru and site.ru/index.php).
• arise due to changes in the structure of the site. For example, when creating a new template with a different URL system, new pages with old content receive completely different addresses.

Fuzzy duplicates are obtained from pages that partially duplicate content. Such pages have a common part of the site template, but differ in small content content. For example, such small duplicates could be resource pages that have search results or individual elements of an article (for example, pictures - this very often happens in many blog templates).

In the following figure, I have collected the most common variants of duplicate pages that are inherent in a blog with the WordPress engine (without taking into account various plugins and template features, which also often create duplicate documents):

The presence of duplicate pages on a site may indicate that either the webmaster is not aware of their presence, or he simply does not know how to deal with them. But you need to fight them, as they lead to various errors and problems in terms of search engine promotion. Let's talk about this now.

Why do you need to remove duplicate pages?

But no matter how passive the duplicate pages are, they can play a real role on the owner of any website or blog. So, why is duplicate content so dangerous? Here are the main problems that arise when there are duplicates:

Deterioration of site indexing. I call this problem #1. The fact is that, depending on the source and the webmaster’s errors, a site page can have from one to several or more duplicates. For example, the main one may have two or three duplicates, and blog posts, due to the ubiquity of replytocom, will be copied according to the number of comments. And if the blog is popular, then the number of “left” pages will be huge. And search engines (especially Google) do not like duplicates and therefore often lower the position of the site itself.

Replacement of the relevant page. The search engine algorithm may consider the duplicate more relevant than the original page being promoted (). Thus, the search results will not contain the target page, but its clone. But this clone has completely different insignificant parameters (link juice, ), which over time pessimize the site in the search engine crescent.

So, let's summarize the above. Duplicate pages interfere with the normal indexing of the site, become a source of incorrect choice by the search engine of a relevant page, and take away the influence of natural external links. Also, duplicates incorrectly distribute internal link juice, taking away power from promoted pages and changing behavioral indicators:

In general, duplicates are a big evil in search engine promotion and an optimizer’s nightmare. 🙂

How to identify and check duplicate pages

There are different ways to find and check duplicate pages. They require varying levels of CMS knowledge and understanding of how the search index works. I'll show you the easiest way. This does not mean that just because it is simple, it is not accurate. With its help, duplicate site documents can be easily found. And most importantly, this method does not require special knowledge and will not take much time.

In order to find and check your website for duplicate pages, you need to either use the advanced Yandex (Google) search or immediately enter a special query into the search bar. Let's look at the first option first.

Checking the site for duplicates using advanced search

Yandex advanced search allows you to get more accurate results thanks to clarifying query parameters. In our case, we need only two parameters - the site address and a piece of text of the page on which we are looking for duplicates. First, we select the text that is on the page of our website (for example, a completely foreign resource is taken) that we will check for duplicates. Then we go to the Yandex advanced search and enter a piece of content and the site address in the appropriate positions:

Next, we click the treasured word “Find” and the Russian search engine begins to generate results. It will not be the one we usually see - it will consist entirely of headlines and snippets from our resource only. Moreover, if there is only one search result in the search results, then everything is fine - there is no duplicate content on this page. If there are several results in the resulting sickle, then you will have to take out a shovel:

In my example, the search engine found several unclear duplicates - a number of pages from pagination for some categories. It is immediately obvious that on this site the page with highlighted text about the salad recipe was clearly placed in different sections at once. And since there was no ban on indexing pagination pages for this resource, all sorts of duplicates appeared in the index.

Now let's look at the same steps for a foreign search engine. We go to the Google advanced search page and perform the same actions:

Having filled in everything necessary in the advanced search, we get the indexing of the site under study according to the specified piece of text:

As you can see, Google also indexed inaccurate duplicates of the page under study - all the same category pages appear in the results.

In principle, you can get the same results without using an advanced search. To do this, you need to enter the following query in a simple version of Google or Yandex. And this is the second way to determine duplicates.

Finding duplicate pages using a special query

Using the advanced search, you can easily find all duplicates for a given text. Of course, using this method we will not get duplicate pages that do not contain text. For example, if a duplicate is created by a “crooked” template, which for some reason shows on another page a picture that is on the original page. Then such a duplicate cannot be found using the method described above. In this case, you will have to use another method.

Its essence is simple - with the help of a special operator, we request indexing of our entire site (or a separate page) and then manually look at the results in search of duplicates. Here are the syntax rules for this request:

When we specify the address of the main page in a request, we simply receive a list of pages indexed by a search robot. If we indicate the address of a specific page, we get a list of indexed duplicates of this page. In Yandex they are immediately visible. But in Google everything is a little more complicated - first they will show us those pages that are in the main search results:

As you can see in the picture, in the main search results we have one page of the site and it is also the original. But there are other pages in the index that are duplicates. To see them, you need to click on the “Show hidden results” link:

As a result, we are given a list of duplicates that Google has indexed and linked to the original page (number 1 in the picture). In my example, these duplicates were pages with positions 2 and 3.

In the second position there is a double, which is the trackback of this document ( automatic notification for other sites of this publication). The thing is certainly necessary, but its presence in the index is not desirable. The owner of this site understands this very well and therefore has prescribed a ban on indexing trackbacks from the site. This is indicated by the inscription “Web page description is not available due to restrictions in the robots.txt file.” If you look at the instructions for search engines (robots.txt), you will see the following picture:

Thanks to the latest directives, the author of the site indicated a ban on indexing trackbacks. But unfortunately, Google puts into its index everything that comes to hand. And here you just need to delete duplicates from the database. We’ll talk about this in the second article on duplicate content.

The third position shows the beloved replytocom by many bloggers. It is obtained by using commenting on blogs and websites. And thanks to him, there is simply a huge pile of duplicates - usually this number is approximately equal to the number of comments on the resource. In our example, this attribute, as well as the trackback, are closed for indexing. But Google put him in its index too. It also needs to be cleaned by hand.

By the way, if we slightly change our query, we can get the same results, which gives an advanced search for duplicates using a piece of text:

So, friends, in this article I lifted the curtain on the concept of duplicate pages and their successful search and verification. To consolidate the material covered, I suggest watching my video on this topic. In it, I clearly showed not only today’s material in two parts, but also added other examples of duplicate content:

In the next article you will find out. See you!

Sincerely, Your Maxim Dovzhenko

They almost always occur. This is due to the fact that the site developers did not take into account many nuances related to SEO. Therefore, usually issues with duplicates are resolved by the optimizer together with webmasters.

1. What are duplicate pages on a website?

Duplicate pages- these are pages with different URLs (address), but with the same content

For example, the same page is available at the following addresses (response code 200)

/category/razdel.html /category/razdel/

There are a lot of similar examples with duplicate pages. Moreover, even if the content of the page is slightly different, it is very important that the name of the page, which is written in