English Español Deutsch Français Italiano Português (Brasil) Русский 中文 日本語
Submit post
Go to Blog

6 Technical On-site SEO Hacks to Improve Crawlability and Increase Organic Traffic

The Wow-Score shows how engaging a blog post is. It is calculated based on the correlation between users’ active reading time, their scrolling speed and the article’s length.

6 Technical On-site SEO Hacks to Improve Crawlability and Increase Organic Traffic

Guy Sheetrit
6 Technical On-site SEO Hacks to Improve Crawlability and Increase Organic Traffic

SEO is all about improving organic traffic to generate more sales. We often spend more time increasing the number and quality of backlinks (a major component of OFF page SEO) and forget to improve the technical aspects of the website.

This article will shed light on some extremely useful technical SEO hacks that can gain more qualified inbound traffic and improve the crawlability of a website. Let’s start! 

1. Optimize the Google Crawl Budget

Googlebots regularly crawls the existing and new pages on your site in the same manner as a regular human searcher might. This helps Google to understand the performance of the website as a slow loading time or a 404 page might degrade the user experience. 

What is a Google Crawl Budget?

The number of pages that Google visits on your site during a single connection is referred to as the crawl budget. This crawl budget is different for different sites. An increased crawl budget means Google is interested in knowing more about your site which in turn can improve your search ranking positions (remember, rankings have over 200+ factors and crawl budget is just one of those).

Here is how Google defines crawl rate:

“Crawl rate limit Googlebot is designed to be a good citizen of the web. Crawling is its main priority, while making sure it doesn’t degrade the experience of users visiting the site. We call this the “crawl rate limit” which limits the maximum fetching rate for a given site. Simply put, this represents the number of simultaneous parallel connections Googlebot may use to crawl the site, as well as the time it has to wait between the fetches. The crawl rate can go up and down based on a couple of factors: Crawl health: if the site responds really quickly for a while, the limit goes up, meaning more connections can be used to crawl. If the site slows down or responds with server errors, the limit goes down and Googlebot crawls less.”

We can say that crawl limit is an excellent way to estimate the performance of a website in the search results as a better crawl budget leads to more organic traffic because it increases the importance of a website in the eyes of Google. In the words of Google ”An increased crawl rate will not necessarily lead to better positions in Search results.” The use of the word necessarily means that crawl rate indeed has an impact on the search performance and can be considered as a ranking factor.

How to Check Google Crawl Rate?

Login to the Webmasters search console and click on crawl stats under the crawl menu as displayed in the below screenshot:


This will give you a clear idea about the number of pages that Google crawls per day along with the amount of time Googlebot spends in downloading the page.

Here are some of the ways through which you can optimize the Google crawl budget:

  • Increase the speed of the site as making a site faster improves the user experience and also increases the crawl rate. Efficient crawling automatically leads to better indexing and improved rankings.
  • Regularly monitor the crawl error report and keep the number of server errors to as low as possible.
  • Ensure you have proper AMP pages on your site so that it takes less time for Google to crawl such pages to improve the mobile performance of the website.
  • Reduce the excessive page load time for dynamic page requests. Dynamic pages take too much time to load resulting in time-out issues.
  • Make use of virtual private servers to improve the server response time.
  • Optimize images and reduce unnecessary JS and CSS.
  • Ensure to take the mobile-friendly test and fix any mobile crawlability or design issues that your site might be having.

2. Use HTML Tables for Direct Answer Queries 

Google loves to give direct answers to the searchers.

In this context, if your web pages provide direct answers to the commonly searched user questions related to your niche then you have an excellent chance of diverting lots of traffic from the direct answer box results returned by Google.

Here is what Google returns when the user types the query: MacBook Pro price


The site macprices.net is ranking in the answer box on top of the regular search results because it uses proper HTML tables to give direct answers. Google loves HTML tables likes these, and this is the reason Apple gets defeated and wasn’t included in the answer box.

The crux of this experiment is, you need to first identify the question-based search queries then you can prepare answers in a tabular form by making use of HTML tables and structured data so that it becomes easier for Google to understand the content of the page and display it directly in the search results.

3. Leverage the Power of Internal Links

Internal linking still remains one of the most powerful SEO tactics. An internal link connects one page of a website to a different page on the same website. 

Proper internal linking helps to pass link juice more efficiently across the inner pages. Following the below tree structure for directories, subdirectories, and pages helps. 


 Here are some of the ways through which you can improve the power of internal links: 

  • Create lots of actionable, authoritative, and high-quality content. When you have lots of content, then you can easily create a lot of internal links.
  • Do not always link to the home page. Instead, link out to deeper pages that otherwise have fewer links. The more internal links you have to your important pages, the better chances you will have to get them ranked higher up in the search results.
  • Follow Wikipedia if you are looking for an inspiration for relevant internal links. Contextual internal linking is far better than linking done at the bottom of the content.
  • Use breadcrumbs, as it aids in navigation and also raises the importance of inner category pages especially in the case of an e-commerce site.
  • Use relevant, natural, and do-follow links. Unnatural links that are not beneficial to the users but created for the sole purpose of internal linking will have a reduced CTR and time on page thereby reducing the overall importance of the website.
  • Diversify your anchor text when doing internal linking to remain safe from the penalties of Penguin. You can make use of tools like Internal Link Analyzer to check the current status of your internal links. 

4.Increase the Word Count of Blog Pages to 2500-3000 Words 

If you want to rank for an informative search query like “how to improve WordPress SEO”, then make sure that you have at least 2500 words of content in your page.

Snapagency did a survey and found that blog posts having a word count of 2500 words or more received the maximum social shares.


Similarly, pages having a word count between 2200-2500 words received the maximum organic traffic.


Hence, the best blog post length that you should always aim for is 2500 words. Simply find out all the informative search queries based on Google Micro Moments that will help the customer to reach the end of the funnel. After that, create content that provides actionable and interactive text to help the user find answers to their problems or confusions. 

5. Optimize Your Site for Mobile Web Crawling 

Google has now moved to a mobile first index. While doing a technical site audit, it is crucial to review whether Google Smartphone crawler is able to properly index the contents of the website.

Here are some of the ways to have a mobile friendly site: 

  • Conduct a mobile friendly test and use the “Fetch as Google” option from Google to check for any issues.
  • Check the mobile usability report under the search console to check any mobile usability issues detected over time.
  • Make use of Tools like Screaming Frog to simulate mobile bot search behavior and ensure that the mobile pages are rendered properly.
  • Perform a log analysis and block spam bots so as to allow easy access to search engine crawlers. 

6. Regularly Update Your Sitemap 

As per Google,

“A sitemap is a file where you can list the web pages of your site to tell Google and other search engines about the organization of your site content. Search engine web crawlers like Googlebot read this file to more intelligently crawl your site. Also, your sitemap can provide valuable metadata associated with the pages you list in that sitemap: Metadata is information about a web page, such as when the page was last updated, how often the page is changed, and the importance of the page relative to other URLs in the site.”

Here are some ways to ensure that your sitemaps are proper and crawlable:

  • Update your XML sitemap every time you add a new page on your site.
  • Remove duplicate pages, non- canonicalized pages and improper redirected pages.
  • Use consistent and fully qualified URLs.
  • Do not include session ids.
  • Ensure the sitemap file is UTF-8 encoded.
  • Check the sitemap errors regularly under the Search Console.
  • Keep the sitemap size to less than 50,000 URLs so that the important pages gets crawled more frequently. If you have multiple sitemaps, then use a sitemap index file.
  • Have an optimized mobile sitemap. You can download the elements specific to mobile from here.

Follow the above technical SEO tips and you can instantly notice the difference.

Remember, both on-page and off-page are integral parts of an effective SEO strategy. Which on-page SEO hack has given you the maximum benefit? Please let me know in the comments below.

Guy Sheetrit

A bearer of digital marketing wisdom.

Founder and CEO of Over The Top SEO. He holds broad expertise in the areas of SEO, Social Media, Digital Marketing, B2B, B2C Brand Development and online lead generation.
Share this post


Guy Sheetrit

A bearer of digital marketing wisdom.

Andy, thanks for taking the time to reply with such detail but you're barking up the wrong tree my friend, what you said now on linking internally is exactly what I literally just said in my first reply, diversify means using creative ways of showing your desired anchor to get attraction to your destination internally or externally.

so when you want to feature a product in on your own property you can showcase it in many different ways instead of writing the same anchor over and over again (thats what i mean by diversify internally) in your home page thinking it would help you rank.

As for credentials this is not the place to try and outdo each other so I will simply say if you question my ability take some time and search on Google for various keywords like SEO Experts or Top SEOs, etc.
Andy Drinkwater

Provides valuable insights and adds depth to the conversation.

Guy Sheetrit
That literally makes no sense I'm afraid Guy. That isn't what I said - I don't advocate diversifying anchor texts for internal links. Diversify means to make use of varied ways (or texts in this case), which is not what you should do when creating internal linking for hub pages.

However, you just mentioned external links, which is a very different ball game and one where you wouldn't employ the same tactics as you would for internal.

Glad to see you have read up a bit more about Penguin as well but as I said earlier, do stop by Twitter and have a look at the type of work I rank 1st place by correctly using internal links.
Andy Drinkwater

Provides valuable insights and adds depth to the conversation.

Guy - I still can't reply to comments for some reason...

Hop over to Twitter and have a look at this:

Nothing but internal linking helped get this to where it is.

More than happy to discuss this in more detail away from here if you wish.
Alex Tsygankov

SEMrush employee.

Andy Drinkwater
Hi Andy, could you please clarify what happens when you try to reply to comments? We will fix this bug ASAP. Please get in touch with me at blog-editors-en@semrush.com. Oh, and thank you for starting this great discussion!
Andy Drinkwater

Provides valuable insights and adds depth to the conversation.

Alex Tsygankov
It worked this time around Alex - it kept just popping a red box at the top right of the screen telling me something had gone wrong.

And you're very welcome - I love to debate SEO :)
Andy Drinkwater

Provides valuable insights and adds depth to the conversation.

I would honestly go and do lots of reading up on these. You should never diversify internal links to a page - you send mixed signals to Google about what the page is all about then. Google uses this to help them understand. You wouldn't have a page about Cat Flea Treatments and then use just 'Treatments' as an anchor text as this is too ambiguous. People want to see a link about Cat Flea Treatments.

You need to be descriptive - craft your keywords carefully and Google will use these to help you rank better. How do I know? I do this day in and day out for clients - follow me on twitter if you want to see some of the results from these that I share.

However, never over-optimise either - you need to do this carefully.

Create hub pages that target a phrase and enhance the understanding of that phrase for Google by using a strong anchor.

This is a little bit of required reading that I would suggest you go and have a look at:

And please, do go and have a look at some SEO basics - Penguin has NEVER looked at internal links - even Matt Cutt said this some yeas ago: https://www.webpronews.com/matt-cutts-on-penguin-and-internal-links-2013-04/

In the past some have said that over-optimisation of pages can trigger a penguin penalty, but no-one has ever been able to show anything like that. Perhaps in the early days of Penguin, but certainly not in the recent past.

Penguin is all about link spamming and was originally the Farmer update and was always targeted at external links by looking for anything unnatural - never internal and you will be hard pressed to find anything to the contrary.

Been doing this for 18 years now and have a stack of results to back up everything.

I don't want it to look like I am coming down all heavy on these points as there is some good advice in here - but when it's wrong, it's wrong.
Guy Sheetrit

A bearer of digital marketing wisdom.

Andy, thank you for your input. as for the content of your reply i would have to strongly disagree with you.

1. "When you do internal linking, you don't diversify" this is probably the most misleading thing i heard in a while buddy, no idea where or who you heard it from but its terribly wrong, you should diversify your anchor texts internally and externally, even natural looking, branded keywords such as: YourWebsite.com, www.yourwebsite.com, your business or brand name, your website name, ‘click here’, ‘here’, etc get diversity so i am not following your point here.

2. "Secondly, Penguin was never about internal links - only external" again, wrong and misleading information, Penguin DOES takes into account internal links in its algorithms. External links gets most of the attention when it comes to Penguin but you should never ignore the signals your sending to Google with internal linking. I never said its the MAIN thing or "if you don't do this you'll end up dead or penalized" I merely gave tips to what you should pay attention to and what you can do to make it better.

on another note, Penguin is still very much alive and kicking, and as you may or may not know since websites are being evaluated in real-time and rankings impacted in real-time the less chances you take the better.
Andy Drinkwater

Provides valuable insights and adds depth to the conversation.

I would like to point out that there are a couple of big mistakes on here.

"Diversify your anchor text when doing internal linking to remain safe from the penalties of Penguin"

When you do internal linking, you don't diversify - you keep your internal anchor tags as descriptive as possible. If you were targeting one particular page, you would keep each of those internal links very specific and there is absolutely no issue with using the same one over and over to that page. Google uses these signals to help understand what a page is about.

Secondly, Penguin was never about internal links - only external. Even now there is no Penguin as that is now part of the main algorithms, rather than something stand alone.
Jyoti Thapa

Provides valuable insights and adds depth to the conversation.

A wonderful articles. I would say, updating your database with new contents every week will increase the overall Google crawl rate. I would also like to use GWT's fetch as google to re-index and render at least using all the 10 quota per month. I think using once a week is better.
Gabka Koscova

Provides valuable insights and adds depth to the conversation.

Great article with some good tips :) I appreciate mostly the second one, which I did not know about. Thanks.
Alex Wood

Occasionally takes part in conversations.

Awesome article for increasing search engine rank!

Either just recently joined or is too shy to say something.

That's a great information on technical SEO for organic traffic, Which really give us one step forward toward the knowledge.
Great thanks

Send feedback

Your feedback must contain at least 3 words (10 characters).

We will only use this email to respond to you on your feedback. Privacy Policy

Thank you for your feedback!