Learn how connections are made in netRivals.
Firstly, it is worth distinguishing Smart Connections from automatic connections. The first ones are basically suggestions that are offered by comparing titles, pictures and references with those of your competitors’ available in our database. On the other hand, automatic connections, as the name indicates, are automatically established whenever netRivals finds matching EAN codes on the pages of your competitors or in the regular scrapings done in Google Shopping.
When you ask how often netRivals crawls competitors for connections, that’s actually more than one question. Processes involving finding a product, including it in our database and generating connections with it, are multiple from the technical point of view.
The first step of such process is crawling the complete website of your rivals. This can’t be done that often, as it supposes many requests and can thus get us banned from the servers in which your rival’s website is allocated. For this reason, we do it on a monthly or bi-monthly basis.
Once we have a complete crawl, we generate a database with all the products of your competitors. This can take a day or two, and it’s only then when netRivals can offer suggestions and automatic connections.
Once we’ve completed the process of building a database, we need to rely on our Google Shopping scraping in order to figure out the EAN codes, in case such codes have not been found in your competitors’ websites. It works as follows: netRivals does a general and random scraping for a specific amount of products in Google Shopping on a regular basis, then it matches all the products it finds in it to the ones existing in your rivals database. Whenever we find matches (by applying a set of rules we have in place), we assign the matching product the EAN form the one we find in Google Shopping. As you may guess, this is a complicated process that takes a while and the fact that it’s random means that we may now find any of your rivals products on a specific scraping. netRivals will surely get the EAN code of your rivals’ products if they’re available in Google Shopping, but we can’t guarantee when that will happen.
Just to clarify! Once a product is connected with a competitor, we update it on a daily basis. What we’re talking about here is only related to the process of generating new matches, not to the matches that have already been put in place.