Have you ever found yourself in unfamiliar territory without a map? Perhaps you’ve visited a city on a weekend break and thought you could around just by following the signs.
It is possible to navigate like that but it’s definitely not the most efficient way of getting from A to B. Sometimes the signs stop.
Sometimes they are just wrong.
Wouldn’t it be a lot easier if you had a map?
Of course it would.
Maps show us a birds eye view of an area we want to navigate so we can quickly see how to get around.
Google thinks the same. It likes maps, they help Google understand your site and how to get around it.
In particular, google likes XML sitemaps.
What is an XML sitemap
An XML sitemap shows Google and other search engine bots what interesting content is on your site and how it links together.
Don’t get too hung up on the ‘XML’ bit at the start. That just identifies how the sitemap has been structured.
Why do you need an XML sitemap?
You want Google to crawl all your important pages. If you don’t have a sitemap and a page doesn’t have a link going to it, then Google won’t find it.
If Google can’t find then Google can’t crawl and index it so no-one else will find it either.
You will miss out on search traffic, which is one of the reasons you have a website, right?
If you have an XML sitemap then you can be sure that Google will find your page.
If your site is small or welloptimised, you could argue that you don’t need a sitemap. However, if you submit a sitemap to Google you are then made aware of problems that could be affecting your SEO by analysing your sitemap data – see Sitemap audit at the end of this articl
What does an XML sitemap look like?
An XML sitemap is just a list of all your content presented in a ‘bot’ readable fashion (that’s the XML bit).
To you and I, the sitemap looks like this.
But to the bots, who read XML, it looks like this;
You can see that the date the content was last modified is included in the sitemap. This tells Google what your most recent content is, so it will crawl that content first.
Want to take a look your sitemap?
How do you view your xml sitemap?
To view your XML sitemap (as a human) and to verify it exists go to www.yourwebsite.com/sitemap.xml and your sitemap will load.
Now, depending on how the sitemap has been generated you may see either one file containing all of your content or you may be redirected to a sitemap index file that contains a list of sitemap files.
On my WordPress website I have a sitemap index file because I used a plugin to create my sitemap.
When I go to sitemap.xml I get redirected to sitemap_index.xml and I see the following list.
I have different sitemaps for my my posts, pages and categories. If I click on one of the links I see all the content within that sitemap.
Google loves this type of structure. I can easily understand how the content is grouped on your website.
If your website is new or you haven’t really paid too much attention to SEO in the past, you might not have a sitemap yet. In which case, you need to create one.
How do you create an XML sitemap?
Creating an XML sitemap is not an automatic process. Just because you have a website don’t assume you have a sitemap. If you have a WordPress website, creating an XML sitemap is really easy.
You can download a plugin to create it for you. There are many plugins around. You may have heard of Yoast or RankMath, these are the two I recommend.
In the plugins I have mentioned you can decide what gets included in the sitemap. Most of the sitemaps I have audited contain everything possible. This is not a good way to manage your sitemap. You need to strategically manage the contents of your sitemap.
What content should you include in your XML sitemap?
To answer this question, let’s start by thinking of the end user, the person who searches in Google and ends up on your website.
You only want to include content that is relevant to the users search intent.
You want to include at least;
- Your best blog articles
- Your main pages
- Your category pages, only if they have been optimised for SEO
- Images if you want them to found in image search
If your content isn’t really relevant or helpful to them then you should remove it from search.
What content should you exclude from your XML sitemap?
Keeping unhelpful content in your sitemap and thus in the Google index is damaging to your brand.
You want to remove anything you don’t want people to find when they search.
Removing content from the sitemap only stops Google crawling it.
Google can still find content, even if it’s not in the sitemap, by following links.
If you want to remove content from the search engine results you need to add a ‘no-index’ tag to your pages also.
So, back to what should you remove…
You also want to remove anything that can be classed as low quality content. If Google sees too much low quality content it will think your site, overall, is low quality.
- Thank you pages
- Gated content pages where the content resides. You will be surprised how many people get you to opt-in to something then they give that ‘something’ away on the thank you page, only, the thank you page can be found on Google!
- Thin pages (pages with a small amount of content) that is in no way helpful to the searcher. e.g Some author pages and tag pages
- Category pages if you haven’t optimised them for SEO,
- Individual testimonials (if you are using a plugin to create them)
- Calendar booking pages
I really want you to understand this issue so let me give you an example.
On my WordPress website I am using a testimonials plugin. For each testimonial WordPress creates a page. That page only contains my testimonial content. There are three things wrong with this.
- No body is searching for this individual testimonial. They might be searching for ‘Weider Web Solutions testimonials’ but the intent behind that search search is to find all my testimonials which is why I have one page dedicated to all my testimonials. This dedicated page pulls in each individual testimonials. It is not necessary for the individual testimonial to be in the Google index.
- It’s thin content. Even if people were searching for this specific content, it’s never going to rank. It only has a few words!
- It’s out of context. A testimonial really needs some context. I think they perform best when on a sales page. Even on my testimonials page someone can click on ‘Conversion rate optimisation’ so they understand that is what the testimonial is related to.
So depending on how your website has been coded you may have a lot of pages like this being index. You need to remove them from the index where possible.
How do you exclude pages from the sitemap and the Google index?
If it’s a specific data type that you want to exclude like ‘tags’ or ‘testimonials’ then you can use your SEO plugin to do this.
If it is a specific page, e.g a thank you page, then you need to add a ‘no-index’ tag to the page. Again, you can do this using an SEO plugin for this.
Just to re-iterate; Removing content from the sitemap doesn’t always stop it from being found. To be really sure they are out of the search engine results you need to add a ‘no-index’ tag to the page too. This ensures the page won’t be indexed.
How to submit your sitemap to Google
It takes just a few seconds to submit your sitemap to Google using Google search console.
Open up a Google search console account.
Go to the Sitemaps tab from the left hand panel.
Enter the URL to your sitemap.
Click on ‘SUBMIT’.
Once your sitemap is uploaded and Google is using it you need to regularly check all is well with the sitemap and how Google is crawling your site. You need to audit your sitemap.
What is a sitemap audit?
A sitemap audit is a detailed comparison of your sitemap (what you want indexed) and the Google index (what is actually indexed). It can reveal the following information;
- What URLS are in the sitemap
- What URLs are being indexed that are not in the sitemap
- Orphaned URLs – URLs that are in the sitemap but have no links to them
- Non indexable URLs that are in the sitemap
- Sitemaps that may be too big for Google to crawl
Why is a sitemap audit important?
A sitemap audit often identifies problems that will be affecting your sits ability to rank well in the search engine.
A clean sitemap and a clean index can give your SEO traffic a real boost.
How do you carry out a sitemap audit?
First you need to look at your sitemap. Does it contain only the important and valuable pages you want Google to see?
If it contains a lot of pages that are no use in the search index then take steps to remove these pages from the sitemap.
Once you are happy you have a clean sitemap you have your ‘map’ for cross checking what is actually being indexed.
In your Google search console go to Index -> Coverage. Click on Valid.
Next scroll down and look for the following two entries.
Submitted and indexed pages should match the actual number in your sitemap. This is where you can spot a lot of website misconfiguration. If the numbers do not match then click on ‘excluded’ in the coverage report and start to look for the reason why some of your pages have been excluded.
This can be tedious, if you would rather someone else do this check for you then let me know. I do this as part of my SEO audit.
Indexed, not submitted in sitemap can be quite an eye-opener.
Here you will see a lot of random pages that Google thinks you want in the index. You need to look over them and for most of them you will need to stop Google from indexing them. I have so many because I changed my URL structure. Google has stopped crawling these and with time they should drop out of the index If they don’t I will manually request they are removed. I will be monitoring the situation.
Your XML sitemap is the map for search engines of the pages you want to be indexed.
Keep the file updated and check it regularly. Small changes in website design can affect your sitemap so be sure to check it after any updates.
Cross check your sitemap with what is actually being indexed.
This will alert you to any website misconfigurations. You can then take steps to removing unnecessary pages from the index.
If you want help with your sitemap audit then take a look at my SEO services page.