Google News: Update for 2019 Robots.txt and GMB Profile Updates

Judith Lewis

Jul 03, 20198 min read
Google Update on Robots.txt

The last two weeks were relatively calm in the search department, but certain things are going away and many new features were rolled out in Google Ads and Google My Business. Also, the search giant has started several new sites that are supposed to make small business owners' and creative makers' lives easier. So, keep reading to see all the updates, and please share any insights you have in the comments below.


Google Says Goodbye to Unsupported Rules in Robots.txt

Google announced on Twitter yesterday that they are saying goodbye to "undocumented and unsupported rules in robots.txt." 

In their blog post, they state, "While open-sourcing our parser library, we analyzed the usage of robots.txt rules. In particular, we focused on rules unsupported by the internet draft, such as crawl-delay, nofollow, and noindex. Since these rules were never documented by Google, naturally, their usage in relation to Googlebot is very low."

Google's Gary Illyes chimed in and explained sites are hurting themselves, so this part of the reason the decision was made:

What you should know is if you are using crawl-delay, nofollow, and noindex in your robots.txt file, they will be ignored by Google starting September 1, 2019. If you have relied on this method, Google has provided the following alternative options:

  • Noindex in robots meta tags: Supported both in the HTTP response headers and in HTML, the noindex directive is the most effective way to remove URLs from the index when crawling is allowed.
  • 404 and 410 HTTP status codes: Both status codes mean that the page does not exist, which will drop such URLs from Google's index once they're crawled and processed.
  • Password protection: Unless markup is used to indicate  subscription or paywalled content, hiding a page behind a login will generally remove it from Google's index.
  • Disallow in robots.txt: Search engines can only index pages that they know about, so blocking the page from being crawled usually means its content won’t be indexed. While the search engine may also index a URL based on links from other pages, without seeing the content itself, we aim to make such pages less visible in the future.
  • Search Console Remove URL tool: The tool is a quick and easy method to remove a URL temporarily from Google's search results.

Source: Google Webmasters TwitterA note on unsupported rules in robots.txt

New Search Features Tests

As usual, users spotted (and Barry Schwartz documented) numerous signs of new features tested on SERPs. Here are the most noteworthy of them:

Separate tabs for Google Shopping and other resellers in Google Shopping Box:

New "refine search" dropdowns for mobile search:

New mortgage calculator design:

New local listing tabs, such as Product, Last Models, Video Game, Menu, etc.:

"People also considered" ad carousel:

Sources: New Google User Interface Updates: Refine Layers, Mini-Carousels, Shopping Toggle, Mortgage Calculator & More, Twitter: SEMrushGoogle Testing New Branded Local Pack UI With More Tabs

New Ad Types: Live Stream and Swirl

Two new ad types are being tested in Google Display and Video 360 (formerly Doubleclick), Swirl and Live Stream.

Swirl ad is a 3D object that users can interact with, rotating it and zooming in/out. To make creating such models easier, Google plans to add a new editor to its 3D platform Poly

Live Stream ads allow advertisers to include live streams in their display ads. These streams feature regular YouTube controls and can be watched in full screen.

Both new ads types are now in limited beta; it is unknown when they will be available for all advertisers.

Source: Deliver more interactive ad experiences with Display & Video 360

Changes in Portfolio Strategies

Google announced a couple of changes to help simplify portfolio bid strategies. 

The first change is removing the ability to create Enhanced CPC portfolio strategies. They provided no additional benefits compared to individual campaigns and were not widely used, according to Google's statement. All existing Enhanced CPC portfolio strategies will be reverted to individual Enhanced CPC campaigns later this year.

Second, the target spend setting is no longer available for new maximize clicks portfolio. Google said, "Average daily budgets will be the way to determine spend for campaigns using Maximize clicks portfolio. Later this year, our systems will remove legacy target spend settings and instead use your average daily budget to manage spend."

Source: Simplifying portfolio bid strategies

Responsive Ads Creation in Google Ads App

Google Ads mobile app users are now able to create and edit responsive search ads (RSA). The new functionality includes creating and editing a list of negative words and recommendations on conflicting negative keywords. Thus, the Google Ads mobile app is gradually turning from a reporting and analytics instrument into a full-fledged tool for creating advertising campaigns. Now users can launch new promotions while on the go.

Source: Create and edit responsive search ads in the Google Ads mobile app

Shopping Campaigns Data in the Landing Pages Report

You can now review the performance of your landing pages for Shopping campaigns by viewing the “Landing pages” page in Google ads. It provides users with the primary metrics for each specific page, such as clicks, impressions, CTR, conversions, etc. With this report, advertisers will be able to detect issues with their landing pages and fix them quickly.

Source: See landing page performance from Shopping campaigns on the "Landing pages" page


Preferred Domain Setting is No More

Google has removed the option to choose your preferred domain in GSC. Previously, users could choose whether they wanted Search Console to display data for a WWW or non-WWW version of their site. Now the tool relies on canonical, sitemap, and redirects data to pick the default site version. There are still a few options to make GSC take a specific version of the site into account; you can find them here.

Source: Bye Bye Preferred Domain setting

New GSC Features

A couple of small but handy features appeared in Google Search Console. The first one is Markup search in URL testing tools, such as AMP Test, Rich Results, and URL inspection tool. You can see how it works in this tweet:

Also, you can now check the crawler type for each report (desktop/mobile) and see the annotation to learn when your site was switched to mobile-first indexing.

Sources: Twitter: Google Webmasters

Top Queries Change Alerts for GSC

Google has started to inform site owners of top queries changes. These emails contain information about top queries and how they changed in a week. It can help you track your site's most important keywords performance.

Source: Google Search Console Alerts: Change In Top Queries For Your Site

Google My Business Updates

Several new tools for local optimization were released in the last couple of weeks:

  • Welcome offers allow businesses to reward users who start following them with special rewards, such as discounts, coupons, etc.

  • Short names and URLs can now be claimed by businesses. Soon users will also be able to perform short name searched in Google Maps.
  • Business owners now can manually pick their profile cover photos.
  • Logos: company logo becomes visible at the top-right corner of the business profile. You have to complete core information to make the logo appear;
  • New Dynamic photo module is updated immediately after an image is uploaded. Soon businesses will be able to add captions to their photos.
  • Creating offline materials will now be easier with the help of a special website launched by Google. It allows businesses to quickly create stickers, posters, and other assets.
  • Local Favorite is a new label that will be shown for the top 5% of businesses in a particular category. The goal, according to Google, is "to recognize those businesses that consistently deliver a great experience for people."

Source: Helping businesses capture their identity with Google My Business

Google for Small Business Portal

To make small business owners' lives easier, Google has launched a new website called Google for Small Business. It contains information about various tools for small business.

By answering a few questions about their business, users can get a personalized step-by-step plan with a list of Google products to use to perform better online, attract more customers, and increase work efficiency. "The recommendations will include products to help with all three, but with a special focus on the goal that’s most important to you," said Google spokesperson Kim Spalding. Google for Small Business publishes news about small business tools and service updates and will provide readers with information on finding Grow with Google offline events in their area.

Source: A new way to discover Google tools to grow your small business

A New Site for the Creative Community 

Create with Google is a new site designed to inspire creative makers and to help them find ideas for presentations, speeches, and marketing campaigns.

In the Inspiration section, you can find examples of innovative and imaginative work, and you can " filter by format, vertical and platform, or search by keyword to inspire your creative thinking."

The Audience Connect "transforms creative presentations from broadcast into conversation, by tracking client engagement in response to any video." You can  sign up for access here.

YouTube Mockup "lets you upload and showcase work to clients and colleagues in an authentic YouTube environment, simulating your creative across desktop, tablet and mobile."

The site is currently available in English, Spanish, Korean and Japanese.

Source: Create with Google: Inspiration, resources, and tools to fuel your next big idea

Suspicious Site Reporter Extension — Chrome

Two new features were released to help protect users from deceptive websites is now available for Google Chrome. "The Suspicious Site Reporter Extension will improve security for Chrome users by giving power users an easy way to report suspicious sites to Google Safe Browsing." 

Google has also launched a new warning designed to protect users about shady sites and deceptive URLs. They state, "This new warning works by comparing the URL of the page you’re currently on to URLs of pages you’ve recently visited. If the URL looks similar, and might cause you to be confused or deceived, we’ll show a warning that helps you get back to safety." Example: vs.

Apart from blocking you from visiting suspicious sites, the extension also allows you to send a report about such sites to Google.

Source: New Chrome Protections from Deception

There is a quick recap of the most important recent news from Google. Stay tuned for our next digest in two weeks! 

Author Photo
Judith is a renowned writer, trainer, blogger and a digital and media consultant specialising in applying strategic understanding of digital technologies to help businesses innovate and optimise their effectiveness within the new, networked communications environment. She is a regular speaker around the world on SEO, content strategy, link building, PPC, and digital strategy, a contributor to the Huffington Post, and has been recognised by her peers as one of the most influential people in the UK digital industry. Judith judges the MENA, EU, US and UK Search Awards, an industry leading award and has every year since their inception. She has worked with market-leading global businesses including Google, Orange, COI, NatWest/RBS, National Gallery, CIPD, Fidelity, GalaCoral, NBC Universal, Readers Digest, Bayer, Family Search, Amadeus, AMD, AmEx, TotalJobs,, Virgin Startup & more. She has over 20 years of experience, predating Google in her entry into the market and now runs her own consultancy. She has worked at Centaur Publishing, i-level, & Beyond before setting up her own consultancy. Judith also blogs about chocolate, helps with London Girl Geek Dinners, was a founding Modern Muse, and has contributed to the book “Pimp My Site” and the Econsultancy guide to Paid Search. She is currently working on updating the Econsultancy Best Practice guide to SEO and teaches the advanced SEO course as well as the Paid Search courses.