Why Your Pages Aren’t Indexed by Google – Confirmed Indexing Issues

Google is facing serious indexation issues these days, affecting countless businesses all over the world.  Despite the official announcements that the indexation issues have been fixed, the problems seem to persist.

With great power comes great responsibility. Therefore, we can only hope these issues will be solved as fast as possible for all webmasters and company owners that are already facing a difficult year for their businesses. 

Google is undoubtedly one of the largest and clearest monopolies in the world. Bing, its closest search competitor, has just 2 percent of the market — hardly a significant threat to Google’s 90 percent. So, what happens when Google’s SERP is not working accordingly? 

Let’s see!

Google Indexing Problem – Confirmed on October 1, 2020

It all started at the end of September when a number of complaints related to pages being dropped from Google’s search index appeared. Lots of people started reporting index coverage issues and crawl errors on forums, Facebook groups, etc..

Twitter was the most popular channel for this type of issues, as usual, as it’s the only place where there is a chance that a Google representative might actually respond to the issue. 

The same happened in this case, with the mention that this time, Google’s representative John Mueller wasn’t aware of the issue, as stated in the screenshot below.  

Of course, this leaves a lot of room for speculations on this matter, yet we prefer to leave those thoughts on you and present you only the facts for now. 

Lots of other discussions were generated on this matter all over the places, Search Console Help and Black Hat World to mention just a few. Below you can get an idea of the vibe that was on Twitter regarding this matter.  

On October 1st, 24 hours later, Google officially announced that some indexing issues do exist and they are working on solving them. 

We are currently working to resolve two separate indexing issues that have impacted some URLs. One is with mobile-indexing. The other is with canonicalization, how we detect and handle duplicate content. In either case, pages might not be indexed….

— Google SearchLiaison (@searchliaison) October 1, 2020

For the following three weeks, they kept on communicating updates on the problems and how they were fixing them. The most recent and important updates on this matter are the following:

• October 9: the mobile indexing was effectively resolved yesterday, with about 99% of the URLs restored. 

• October 20: the canonical issue was effectively resolved last Wednesday (Oct. 14), with about 99% of the URLs restored. We expect the remaining edge cases will be restored within a week or two.

Meanwhile, Google communication revolved around this type of messages:

There’s no action to take with these issues on the part of site owners.
We apologize for the issues here and are working rapidly to resolve them.
We’ll update this thread as each is corrected.
It may take days to fully resolve both these issues completely, but we have restored many URLs already and are working quickly to process more.

All good in the hood, you might be tempted to say. Google had a bug, they fixed it three weeks after.

Just that things are not exactly like that. Beside the fact that three weeks of not indexing new content or de-indexing old one is a very big amount of time in Internet days measure, the problem is still on, as several webmasters reported. 

We have two examples of our own that confirm the issue is still on. Not sure if you know, but the same hard-working team that developed cognitiveSEO, crafted the brand monitoring tool BrandMentions.com as well. Both examples will be from this website.

The first one is a really cool research on best time to post on Instagram (I might be subjective here, but you can check it out to see for yourself). 

It doesn’t seem that the URL has crawl issue, yet, it is not indexed by Google. It is a unique content piece, with no special indications (it doesn’t have noindex directive). 

The article was published on the BrandMentions’ blog on October 13.  I don’t believe it was a “Tuesday, the 13th” bad luck. Most likely, Google didn’t manage to fix the issues for 99% of the URLs as they were claiming. And yes, there might be the chance that the article is within that 1% of not-fixed URLs. But this is not the only example. 

The second example is a page that presents info on brand tracking, also belonging to brandmentions.com. This time, the page seems to be crawled (on the same unlucky Tuesday, 13) but today, 20th of October, is nowhere to be found within the SERP.

Of course, we hope that this will change and by the time you’re reading this article the page will be indexed and ranking pretty well. Yet, for the moment, the Google Indexing issue seems to be persisting. 

What Is the Google Canonicalization Issue 

Google stated in a Tweet from October 2nd that “ […] If the canonical issue is involved, URL Inspector may show the URL as a duplicate & the Google-selected canonical will be different from it.  The issue with canonicals impacted roughly about 0.02% of our index, beginning around Sept. 20 until late yesterday around 4:30pm PT. We’ve since restored about 10% of those URLs and keep reprocessing more.” 

What does this exactly mean?

It’s somehow a duplicate content issue.  Let’s say that you’d publish a fresh article and soon after a scraper copies your entire article and publishes it just the same. 

The canonical tag added on your site would tell Google that your article is the original one and that it is the one that should be ranked first. You can basically tell Google “Instead of this page, show this page.” So, if you have page B ranking on some keyword, you can basically tell Google to show page A instead.

Yet, once the canonicalization issue appeared, Google would not show the original content, but the syndicated one (the one that it’s a copy of the original one). 

Canonical tags are Google’s recommendation for several situations, as you can see on their page or on the image below. 

There are billions of web pages, and Google has the job of ranking them. Ideally, all websites should have unique pages. Each and every page should contain original content. In reality, however, duplicate pages are to be found more often than you’d think. 

So, Google crawls your website and finds 3 pages trying to rank for the exact same keyword. Not only that Google has to pick between billions of different websites, it now also has to choose between duplicate pages on the same website. That seems tough indeed, so that’s why the canonical tag is there, to tell Google what is the original piece that should be ranked.

And when an issue of this size appears, you remember how important it is for the search engine to do their job correctly, but also for the webmasters to have everything set up correctly. 
Yes, it seems that even with even if you have everything alright on your side, issues can still appear. 

At the time you are reading this post, this type of issues shouldn’t appear anymore as Google stated that they have been fixed. Yet, given the current situation and the fact that webmasters are still complaining about it, let’s hope for the best and expect the worst. 

What Is the Google Mobile Indexing Issue

The Mobile indexing issue caused new web pages to take a very long time to get indexed and appear in Google’s index. Some of the new published content is still not indexed. 

Even the big sites where affected by these issues. Below you can see some screenshots made by Ewdison Then. He reported this issue on September 29, as he found it really strange that all these sites posting news every couple of minutes have nothing indexed from the past hour.  

Google stated that “if a previously indexed page has gone, it might be the mobile-indexing issue, where we’re failing to select any page at all to index.”

As you could see in the screenshots above, lots of webmasters complained that they simply vanished from the index.  As you can see in the screenshot below, Google stated that the issue impacted just a few websites (roughly 0.2%).

This may sound like a small number but remember that Google’s index is huge and 0.2% of it is still a big number.  The Google Search index contains hundreds of billions of webpages and is well over 100,000,000 gigabytes in size. Therefore, 0.2% of all this enormous amount of data doesn’t feel that small of a percentage anymore. 

The mobile indexing issue was the first to be fixed according to Google, with almost two weeks before the canonicalization one. Yet, hopefully this issue won’t appear anymore in the future as this is impacting the search and people’s businesses big time. 

Google faced indexing issue in the past as well. Last year, for instance, a similar problem happened, with people facing indexing issues from all domains and countries. Let’s hope that this time the fix is for good and these types of issues won’t appear anymore. 

We’re aware of indexing issues that impacted some sites beginning on Friday. We believe the issues are mostly resolved and don’t require any special efforts on the part of site owners. We’ll provide another update when the issues are considered fully resolved.

— Google SearchLiaison (@searchliaison) April 7, 2019

How Does Google Indexation Issue Affect Your Business

The short answer to this would be: BIG TIME. 

Home pages, articles, product pages were de-indexed all of sudden. Imagine your business’s site not appearing in searches anymore, at all. 
The errors in Google search will cause massive drop in organic traffic, your content marketing strategy will not be useful anymore 

It’s already a difficult year, and the indexing issue is making business lose even more. 

It’s easy to forget that, at the end of the day, Google is just a private business, like any other. One that has a worldwide monopoly on search engines, with a staggering 92.27% market share that we all depend on, and that has managed to insert its namesake as a common word in the dictionary. But a private company, nonetheless. So, while we would all like to understand what has and is happening and move on with our virtual lives, it might not be as easy as that.

Yet, with great power comes great responsibility. Or at least, it should. 

Whom should the businesses losing their income for over two weeks blame? Yes, bugs happen all the time and all apps, be them big or small have them. And yes, we tend to misuse the Google engine as a “reality interface”. But Google has their reponsibilites as well.

For instance, Google is influencing economy by the way ads are ranked right now: the more a company pays, the more often the add will be visible. Google answers that result from queries are also already ranked when searches are conducted. And Google is responsible (on a certain percentage, of course) for a company’s value. Google is of course doing this already for ads, but it does that for the whole SERP already. 

A sudden disappearance, or more generally a dramatic and sudden change event is an often-used narrative device in much of literature and film. It is sometimes left unexplained, but always manages to elicit a strong emotion. Such is the mass disappearance of 2% of the world’s population in TV series “The Leftovers”, or the sudden stop of any death for an entire country in Jose Saramago’s “Death with Interruptions” novel.

In real life, such events are usually much smaller and less dramatic in scale, but they are not without consequence.

We will probably never know the real number of impacted businesses.

What we do know is that an issue of this amplitude translates in very stressful situations for business owners and their employees, for companies’ general well-being and profit, in a year that is already difficult enough for so many. 

Can You Do Anything in This Situation?

The short answer is no. We just need to wait and hope for things to get back to normal, a phrase we are sure you’ve heard way too many times lately.

On September 30th, Google announced it had suspended the request indexing feature temporarily from their webmaster tool.

There were no detailed explanations as to why, just a vague mention of infrastructure changes.

As days went by, company officials insisted that things were fine, while complaints around Google indexing issues persisted as more and more businesses seemed to be impacted. 

Google is so ubiquitous nowadays that most of the time we do not even think of it as a private company. Both the Oxford English Dictionary and the Merriam-Webster Dictionary added “to google” as a transitive verb back in 2006. It is so common that it is just… there. And with an over 90% market share, this way of thinking is no surprise.

The problem with all of this is that we tend to think of common nouns as more trustworthy, for some reason. Like Google were no longer a company, but some public utility service, whose only intention is to help its users, rather than trying to balance that and make a profit as a market-leading private company.

And companies sometimes will be more interested in profit than in other things, like fairness or truth. Which is not to say that Google has purposefully lied in the past or did not put enough effort in fixing their bugs. But it has, at times, told it in ways that were profitable: not telling the whole truth about the impact of links from press releases, not disclosing anything about the Hummingbird algorithm update except long after its effects had already taken place, or not mentioning the current submit URLs and indexation bugs from the very beginning.

Even if all of this is true, though, it is what companies tend to do. They keep a little to themselves, they over or undersell, as it fits their purpose. But due to Google’s quasi-monopoly, they are also able to bend the playing field, not just accuracy or truth. Of course, the search engine giant is under no obligation to play nice. As a private company, they do not need to tell anyone how they update their algorithms, when they have bugs, what is the intended impact of their actions or what their strategy is. And yet there is a whole industry that is trying to figure out these exact things.

With the hope that this article will be indexed and you’ll get to find it easily within the SERP, we’re looking forward for your opinion on it.