Key Takeaway
We spoke with Dan Mitchell and Henrik Grunditz, the CEO and CRO respectively of Hivemind who illustrate how the team leverages the power of human decision-making and AI together to help business leaders unearth valuable insights that are hiding within the world’s unstructured data sources, enabling them to make better and more informed decisions.
The unstructured world in which we live
Over 80% of the world’s data today is unstructured. It exists in formats that are not easily digested by a machine – locked in text documents, images, videos, audio files, or online in news articles, blogs and social media.
However, most tools available – dashboards, machine learning algorithms and more – require this data to be structured in order to pull relevant insights. That is, data which is standardized, organized (often in tables with rows and columns) and computationally searchable. As a result, much information remains locked within our unstructured files and documents. Platforms like Fairhair.ai look to apply enrichments like Natural Language Processing to make sense of this unstructured data and set parameters that can enable algorithms to distill trends, patterns and signals.
Similarly, the team at Hivemind has set out to give business leaders a tool that will allow them to ask more questions of their unstructured data and take advantage of the potentially transformative information hiding inside, by combining the power of human and artificial intelligence.
Unlocking the value of proprietary unstructured data
As part of their efforts to build rich and varied data sets, in 2014, the R&D team at leading quantitative investment management fund Winton Capital created a platform designed to collect, clean, and enrich unstructured data at an industrial scale. This came to be known as Hivemind, and in 2018 the company was formally spun out as a separate business, offering its services to clients not just in finance but across multiple sectors. Today Hivemind specializes in solving the complicated problem of unstructured data.
“Initially Hivemind was conceived as a capability that would allow firms like Winton to pursue more ambitious and varied research initiatives, specifically moving beyond conducting research on data that was available commercially,” Hivemind Co-Founder and CRO Henrik Grunditz tells us. The idea was that “if a platform could be developed that would allow Winton to collect, clean and enrich their data sets on the basis of primary sources such as news, filings websites and annual reports, the research wouldn’t be limited by the availability, quality and pertinence of what data vendors had packaged up.”
To be useful, that platform would need to be able to turn large volumes of unstructured data into valuable data assets within meaningful timescales (i.e. days, weeks and months, rather than years) and with high attention to detail.
How it works
Artificial + human intelligence explore unstructured data, identify insights
Hivemind works by taking a man plus machine approach. “Complicated, unstructured data projects or processes are broken down into chains of simple tasks that can either be automated or solved with human judgement, as appropriate,” said CEO Dan Mitchell.
This combined approach is at the core of how Hivemind works. Automated methods – for instance natural language processing or OCR – are used where they can be, but despite advances in machine learning, unstructured data is challenging to deal with computationally.
“After all, unstructured data is essentially data designed for human consumption. Dealing with it requires intuition, flexibility, sometimes emotional intelligence and always the ability to adapt to unexpected circumstances. Humans cope seamlessly with inconsistencies, inaccuracies and idiosyncrasies in these sources which often tie automated methods in knots.”
Within Hivemind, human tasks can be distributed to various pools of ‘contributors’, including crowdsourcing platforms like Mechanical Turk, an outsourced workforce, a company’s internal employees, or a panel of experts. Hivemind aggregates the responses to ensure the integrity of the final data. The key thing is that the human work is done in an organised and structured way.
“Hivemind allows you to use automated methods, including machine learning, together with human intelligence in systematic chains and workflows of tasks to create high quality data sets quickly,” Mitchell said.
Case Study
Forward-looking insights from human + artificial intelligence
- How serious are faults causing batches of cars to be recalled?
- What is the impact of board composition or M&A activity on shareholder return?
- Can we use prediction markets to create an objective consensus from experts on future climate change?
“In each of these projects, and many more in which the platform was used over the last four years, Hivemind enabled Winton to build bespoke datasets pertinent to the question at hand, facilitating research which would not otherwise have been possible,” Mitchell said.
The team also illustrated a case in the M&A space by which they collected the details of every merger in the S&P 500 going back to 1960. “This is probably 30 years further back than what is commercially available from most vendors,” Grunditz said.
The process provided a rich dataset by discovering an additional 5000 M&A deals from combinations of news archives (mainly WSJ and NYT), filings and other sources.
This clean, structured data could then be used to explore questions like:
- Does a company’s propensity for acquisitiveness over organic growth matter over the long term?
- Are there certain deal characteristics that make them more or less likely to complete?
- Is there a way of identifying a “reckless” or value destructive acquisition prior to the event by looking at historical patterns and outcomes over 1000s of deals?
Use cases for Hivemind extend far beyond the finance space. One example the team highlighted involved a fashion brand looking to gain insight into trends from images on social media and blogs. It can also be used to help resolve questions around the credibility of data sets. In one recent case, Hivemind was employed to scrub structured data sets and bring in humans to make judgement calls about the accuracy of a data feed, given some contextual, unstructured information. For instance, if data suggests a company lost 90% of its value in one day, a human can very quickly make a judgement about whether that is believable or not by looking at an appropriate news feed.
Similarly, Hivemind can aggregate human opinion and expertise to create forward-looking predictions, helping insurance companies, for instance, to predict when and where the next hurricane is going to hit the Eastern Seaboard.
Finally, a key use of Hivemind is to make the machine learning methods themselves better. By using a human workforce to tag, annotate or categorise information from unstructured sources accurately, you create the structured data vital to training new and improved machine learning algorithms.
Why we need man and machine together
In each of these cases, decision-makers needing to create data sets from complicated unstructured, or even unrecorded, sources, which all can be achieved with Hivemind’s combined man and machine approach.
“We believe humans and computers complement each other powerfully when it comes to dealing with unstructured data,” Mitchell said. “We use automated methods to do a lot of the heavy lifting and to help focus the human effort on those aspects of the process where we need human intuition or expertise to do this job accurately.”
Enterprise AI transformation
Combined with regular signals and forward-looking indicators from across an industry, gleaned through a number of external data sources via Outside Insight, insights discovered by analysing unique unstructured datasets and asking the right questions can give business leaders a significant competitive advantage in understanding the flow of their landscape where others are still simply guessing. The more data that can be analysed to test a particular hypothesis, the greater the certainty with which it can be proved or disproved.
For actionable insights from around the web and the latest live case studies. Join the weekly Outside Insight briefing for free. Get all the key insights direct to your inbox when you subscribe today.
