Listening to 4 Billion Search Queries per day
One finds these insights in forums and investment boards—no need for stalking or stealing private data. You have to know where to look for it and how to automate your search process. – with watchdogs.
But there is more: when and where someone is looking for information, social support, inspiration, hard facts, or help tells us what they think and who they think is qualified as a trusted source.
How do search engines technically work?
Search engines follow links. Links connect pages and documents, much like roads connect villages and cities. By following these links, search engines collect data to show to their users. But how do they do it?
Search engines consist of three parts: the crawler, the index, and the algorithm. The process begins with a software program that travels the web through links and pages. The crawler mainly looks for content headings and site structure. The crawler goes around the internet 24/7, and when it passes through a website, it saves the HTML version of a page in an enormous database – called the index.
The index is updated every time the crawler comes around again and finds new information. In Google’s case, the crawler revisits your site more or less often, depending on how often you change things and how important Google thinks your site is.
How do search engines get their results?
Search engines use an algorithm for ranking results. The algorithm takes the data from the index and calculates many factors that predict whether a result will be helpful for the searcher. Like we maintain our websites to say relevant, Google never stops re-evaluating their indexing database and algorithm. While we’re talking about links, both internal links (links from one part of your website to another part of your website) and external links (links from a different website to yours) have a tremendous effect on search engine rankings.
Site speed, site security, and quality of content are essential. But Google makes changes to the importance of all these different factors almost every day. Therefore, crawlers follow all the links on the web that they have access to continuously. This way, they can find new sites to index and change outdated information.
Why are vertical search engines different?
Like any generic search engine, vertical search engines consist of crawlers, an index, and an algorithm. But vertical search engines are trained to optimize in their exceptional direction. Vertical search offers greater precision due to limited scope. Vertical search engines leverage domain knowledge, including taxonomies and ontologies. You cannot find the same level of information in generic search engines as in vertical search engines.
Watchdogs monitor the decision process
Watchdogs gather data from multiple sources
We combine multiple data sources and identify implicit signals. Here is an example of an implicit indication. If I search for a specific category or brand, I learn about my brand awareness, attitude, and buying intent.
Technical Data can be an important signal and indicate a business impact. For example: maybe you heard of drones and small satellites monitoring physical car traffic patterns. They can tell you how many cars leave the factory or how many vehicles use the Tesla Supercharger Network, including the time of day and frequency. You can learn about Tesla’s 🚕 adoption rates, charging interval, and you can analyze these insights over time and compare different stations of the Tesla Supercharger Network.
Compound queries run in the background
For a long time, we have been using searching engines like this:
- You type in the first query, the Google Algorithm does its little magic, and you get a response. If you don’t like the outcome, you start the same process. You type in a second independent revised query. Again the Google algorithm does its magic, and you get a new response.
- But now, let the Google algorithm learn from the first query – so it’s becoming a compound query, meaning it combines more than one component query into a single SQL statement via a set operator. In our first example, let the algorithm know that we are on a smartphone with GPS enabled. That’s not a big deal in 2020.
- Today, your smartphone has a built-in superpower. It has sensors and it knows if you are walking, running, or driving. It knows where it is, where you have been, who your friends are. The system manages the complexity in the background. The user experience is seamless. 🚀
Watchdogs use cases
Watchdogs find trigger events and black swans
When unexpected events cause economies and capital markets to tumble, politicians, economists, and asset managers like to speak of a “black swan.” These animals are scarce and, therefore, also stand for improbable events in economic theory, which throw the familiar out of balance. I took a picture of the two black swans on the photo in Knysna, South Africa, 2017, a beautiful place with amazing animals. As we know, these beautiful animals are rare, but sometimes you meet even more than one of them.
New York Professor Nassim Taleb used the term in his book “The Black Swan to outline the meaning of this in Innovation and Crisis Management. The power of highly improbable events”. Nassim Taleb uses a Thanksgiving Turkey to describe why expectations built on historical events can fail dramatically.
“Consider a turkey that is fed every day. Every feeding will firm up the bird’s belief that it is the general rule of life to be fed every day by friendly members of the human race ‘looking out for its best interests,’ as a politician would say. “On the afternoon of the Wednesday before Thanksgiving, something unexpected will happen to the turkey. It will incur a revision of belief.”
The black swan event has killed the turkey – or in a business context, it can stop your competitors´ ability to execute strategy. | Nassim Nicholas Taleb
Watchdogs indicate crisis
The worldwide Covid19 crisis shows us in 2020 how vulnerable our complex globalized social and economic system with its interdependencies is. We realize that the pandemic disrupts global supply chains and little capacity reserves and redundancies in the system.
Pure software companies are less exposed to supply chain disruptions and might even see an exciting market opportunity. Traditional companies have banned all non-essential travel. Travel companies and companies that depend on in-person meetings are affected.
The pandemic was foreseeable.
In a guest article for the “Neue Zürcher Zeitung” (NZZ), he and his co-author Mark Spitznagel explain why the corona crisis should by no means be considered a Black Swan event, “for which being unprepared is excusable.
Nassim Nicholas Taleb and his co-author Mark Spitznagel explain why the corona crisis should by no means be considered a Black Swan event. On the contrary: The global pandemic is a white swan – an event that will undoubtedly occur in the future.
“A global pandemic will undoubtedly occur in the future. The government of Singapore, which called on us as consultants at the time, had already been preparing for a pandemic since 2010 with a detailed plan.”
Nassim Nicholas Taleb, NZZ 2020
With change comes opportunity because certain events will damage your competitor’s ability to execute strategy. The challenge is to look out for when these events will occur. The black swan event has killed the Thanksgiving Turkey – and it can stop your competitors’ ability to execute strategy.
There is always a risk in the business plan, and at the same time, a massive business opportunity. Each shift in spending creates a foreseeable market opportunity. Taleb writes that Singapore took action and already started projects in 2010. The opportunity lies in defining these scenarios and execution models before they occur. Search listening and social listening helps us understand when the time is right – and what we have to do. When the event arrives, you should be ready.
👉 How to find out about the next battlefield, understand its impact, and prepare for it?
👉 How to spot a company worth buying or sell your own company before it´s affected?
Watchdogs improve market sizing
Professional preparation makes sure that you enter the right niche. Target Market Sizing helps you refine business model canvas hypotheses. Growing international comes with language and cultural barriers, different tax codes, business regulations, and packaging standards in other countries.
Watchdogs find a correlation between opinions and voters
You may not believe what the unbiased perspectives of millions of people can tell us. Search Listening is less subjective to unreliable responses, which can happen in surveys, or the desire to project a specific image rather than reality, which is a risk with social data.
Book: Everybody Lies
The book has fascinating, surprising insights into everything from economics to ethics to sports to race to sex, gender, and more, all drawn from the world of big data. Google knows more. Seth Stephens-Davidowitz demonstrates the extent to which all the world is indeed a lab. On sensitive topics, every survey method will elicit substantial misreporting.
- What percentage of white voters didn’t vote for Barack Obama because he’s black?
- Do violent films affect the crime rate? Can you beat the stock market?
- Who is more self-conscious about sex, men or women?
- Do parents secretly favor boy children over girls?