How to analyse URLs and Indexed Pages in the SISTRIX Toolbox

The SISTRIX Toolbox gives you the possibility to analyse a URL exactly as you would do with a domain, discovering its visibility and rankings on Google both for mobile and desktop devices. This feature is useful to find pages that are ranking particularly well on Google, or check the online performance of specific URLs after a Website migration. In this tutorial we'll show you how to use the Toolbox to evaluate the number of indexed pages for your domain, which is fundamental to knowing if Google has problems indexing the domain.

Where do I find information about the URLs of my website in the Toolbox?

Almost every keyword table in the SISTRIX Toolbox also shows the ranking URLs for those keywords.

If you want to see a detailed overview of all the ranking URLs for your website (both desktop and mobile), the section “URLs” is the best place to look.

To reach it you just have to type the domain you want to evaluate in the Toolbox search bar 1 and then click on the entry “URLs” in the left hand navigation 2.

Note that the data contained in this section comes from the entire database of the SISTRIX Toolbox, allowing you to work with the most complete dataset possible.

Overview of the available functions

The section “URLs” of the Toolbox offers a lot of different options to let you find and evaluate the data as quickly and efficiently as possible.

Before beginning your evaluation, let’s focus on the options at the top right corner of the page, which allow us to define the URLs data according to:

  • Date: by default the Toolbox shows the data of the present week. Click on this button to select a date in the past.
  • Options: select different options for the table (more on this below).
  • Mobile/ Desktop: choose between mobile and desktop data as well as the indexed pages.
  • Export: export the table as a CSV file. To do this you’ll need to use some credits.
  • Shortlink : share the table on Social Media.

By clicking on the option menu, you’ll be able to change some settings of the URLs table or to personalize some filters:

  • Select Columns: you can change the columns in the table, choosing only the ones you’re interested in. The Toolbox will save your preferences as well as the default design.
  • Save filter: if you made a filter combination you really find useful, you’ll be able to save it for the future, saving some time.
  • Load filter: select one of the filter combinations you saved or delete it.

When you click on the downwards arrow besides the mobile/desktop toggle, you will find the link to the indexed pages section.

There, you’ll be able to see the amount of pages that Google has indexed for the domain. We’ll look into this section later on in this article.

Remember that you can add all the elements contained in this section into a Dashboard or a Report. Read the following articles for more information about these two features:
  • How to create and configure Reports in the SISTRIX Toolbox
  • How to Use Dashboards for SEO Monitoring and Workflows
  • General information about the URL data

    Right under the header you’ll see four boxes that contain general information about the URLs of the domain.

    The first box will show you the number of URLs for this domain which rank for at least one keyword in Google’s Top-100 result. The arrow shows the % change compared to the previous week.

    A sudden change in this number could be a wake-up call for a possible indexing problem of the Website.


    The second box shows you the average amount of keywords within the Top-10 and Top-100 for which the individual URLs are ranking, giving you an easy and quick overview. A domain with a good performance should have at least 20% of its URLs ranking on the first result page.


    This data lets you discover how many of the ranking URLs use the HTTPS protocoll (SSL) and how many only use HTTP.

    Remember that those pages that still don’t use an encrypted protocol are marked by Google as “not secure”, quickly convincing the user to leave and search for competitor. Moreover, HTTPS can give a slight boost in the domain’s rankings.


    Finally, this box will show you the average length, architectural depth (directories) and number of added parameters for all ranking URLs.

    Filters

    The filters allow you to further refine your list of URLs and eventually discover features that could be optimized.

    Some filters are immediately available on the interface (Quick Filters), while others can be selected by clicking on the green button “Filter now“.

    Here’s a quick explanation of every avaliable filter for the URLs:

    URL: finds the URLs that contain (or not contain) a text. For example, the quick filter “Contains underscores” lets you quickly find all the URLs with this feature, which is particularly important because Google recommends using hyphens instead of underscores in URLs.

    Number of parameters: filters the URLs according to a specific number of parameters. Note that URLs which have more than 3 parameters are to avoid, because they are extremely vulnerable for Duplicate Content problems: that’s why we created the quick filter “At least 3 parameters“.

    Number of directories: filters the URLs according to a specific number of directories. Thanks to the quick filter “At least 3 directories” you can find those URLs that are deep in the website architecture, which run the risk not to get crawled as often by Google (or at all) as URLs higher up in the structure.

    Top-10 ratio: indicates how many Top-100 keywords are also ranking in Google’s first result page. The quick filter “Few top rankings“, for example, shows you all those URLs that are ranking on the first organic page with less than 3% of their Top-100 keywords: you might want to check if you want these URLs in Google’s index.

    Containing Session IDs: filters the URLs that contain (or not contain) Session IDs. Note that Session IDs can easily cause Duplicate Content problems, so it could be a good idea to check why your system is providing Session IDs to users and Googlebot.

    Containing uppercase characters: shows only the URLs which have uppercase characters. It is always better to avoid using uppercase characters in URLs, as upper and lower-case letters are technically different URLs which may get you in trouble when it comes to setting up 301 redirects.

    SSL encryption: shows the URLs that use (or don’t use) HTTPS protocol: this is particularly useful for those URLs that are still not secure (quick filter: “Without SSL“).

    AMP: filters all the mobile URLs that are AMP pages.

    Table of ranking URLs

    Now we can move on to the most important part of this section: the list of the different URLs for the evaluated domain.

    For every URL 1 the Toolbox shows the number of keywords ranking within the Top-10 2 and Top-100 3 results on Google.

    By default, the table is sorted by the amount of Top-100 rankings. You can change this with a click on the column heading. If you want to visit a specific URL, you can click on the icon and the Toolbox will open that page in a new tab.

    Detailed analysis of a single URL

    If you want to analyse a single URL in the Toolbox you can either type it into the toolbar or select it from the URL table.

    For the later case you just have to move your mouse to the end of the URL you want to analyse and then click on it.

    Depending on where you click, the Toolbox will open the host, path or entire URL. You will always see how much of the URL is selected.

    After clicking on the URL, you’ll find yourself on its overview page containing the most important KPIs, like the Visibility Index, the number of ranking keywords, links and social signals.

    Note that these KPIs cover desktop, mobile, weekly and daily data. To know more about this section of the Toolbox, please read our tutorial: “Use the Domain Overview to get a quick summary of how a domain is doing“.

    If you prefer to evaluate another “level” of the URL (for example its directories), you can click on the links contained in the yellow notice at the top of the page.

    Use the menu on the left to further analyse the URL. The following evaluations are useful, if you don’t know where to begin.

    Visibility Index of a URL

    In the section “Visibility Index” you can evaluate both the daily and weekly visibility for desktop and mobile: are there some technical problems that are damaging its online performance? Is its mobile visibility at the same level of the desktop one, or are there some improvements to do?

    Note that many individual URLs will have a Visibility Index of zero. In that case it is better to evaluate the directory that the URL is in.

    If you want to know more about the Visibility Index please read the following article: “What is the SISTRIX Visibility Index?

    Ranking keywords of a URL

    A URL can (and should) rank for more than one keyword in Google: the section “Keywords” will let you see them all in order to begin your keywords research analysis.

    You can also visit the section “Ranking Changes” to see if the URL has gained or lost keywords, or if there have been any changes in its organic rankings in Google.

    Finally, you can take a look at the “Ranking Distribution” of the URL, which will show you how the Top-100 ranking are distributed within Google’s result pages.

    Links of a URL

    It can also be interesting to know how many backlinks the URL has, where they come from and if they are actually a benefit to the domain. You will find this information in the Link Module of the Toolbox.

    How to see the Indexed Pages of a domain

    Thanks to the Toolbox you’ll be able to evaluate the number of indexed pages of a Website which is extremely useful to find out if Google is having problems with the indexing process. In this section of the Toolbox you’ll be able to discover the historical development of the indexed pages for a domain in the past. This feature can be activated not only for a domain, but also for a directory, a subdomain or a host.

    As we said before, you can reach the indexed pages of a domain from the “URLs” section: after writing the domain in the search bar 1, select the entry “URLs” 2 from the menu on the left. Now click on the grey arrow on the top right corner of the screen, near the button for the desktop data 3 and then on “Indexed Pages” 4 .

    Alternatively, you can also simply click on the entry “Indexed Pages” in the overview page of the domain, inside the box called “Domain Overview”.

    The Toolbox won’t show the number of indexed pages if the domain has a visibility score inferior to 1 point. However, you can click on “Collect data” and, from that moment on, the Toolbox will evaluate them for you.

    In the section “Indexed Pages” you’ll see a graph containing the amount of pages that Google has indexed for the domain you are evaluating.

    These values are gathered through a site:-query for the domain on Google – for the above screenshot that would be site:screwfix.com.

    These values are, by Google’s own admission, very rough estimates which can vary. To catch large outliers, we will query this data multiple times per week and calculate the average.

    However, this chart is very helpful to notice if Google is having problems indexing the website. If you notice that the number of indexed pages drops suddenly, you should immediately check your Website in order to find out whether GoogleBot is being blocked from indexing it.

    You can also use the icon at the top right corner of the graph to modify the event-pins, download the graph as a PDF, PNG or CSV file, compare it with another chart or add it in a Dashboard or a Report.

    It’s not possible to download or export a list of all the indexed URLs of a domain. However, you can download the entire page as a PDF or export the chart and the number of indexed pages.

    The table beneth the graph shows you the maximum, minimum and current value of indexed pages for the domain and the last crawl date.

    Finally, when you use the “Add to Watchlist” function for the domain, we will query the data for the indexed pages at least once a week, even for small domains.