Love him or hate him, Jim Cramer creates a perfect narrative that describes the most significant problem big data has today and no one is talking about it.
If you browse through Cramer’s social media feeds, you’ll inevitably run into comments about how a stock he once endorsed, is now being recommended to get out of. Many people make claims about a “pump and dump” scheme or how he’s a flip-flopper.
These are all examples of Big Data’s greatest problem.
What Is Big Data
So, let’s first review what exactly “big data” is. And we're going to go really basic. Big data is comprised of three main components; the three “V’s.”
Volume. This is the “big” in big data
Velocity. The speed at which new data is available
Variety. Various types of data points from social, to demographic to financial and even IOT (internet of things).
Most of us understand ‘volume’ and the industry has done a good job in addressing that component. And we’re really good at storing + accessing data.
We’re not so great with velocity (4.5M videos viewed on YouTube, 188M emails sent, 3.8M Google searches…all within 1 minute, every day as of 2019) and even worse with variety. I have no doubt we’ll create tools to help us solve these last two components, but we’re not there yet. But that's still not the real "problem" with Big Data.
We’re hiring data scientists, learning Hadoop and creating committees and sub-committees to debate the merits of big data. Yet, very few people have identified the greatest threat to uncovering Big Data’s potential.
Us.
Inflexibility
Have you ever heard of the phrase “cognitive dissonance?” One definition:
In the field of psychology, cognitive dissonance is the mental discomfort experienced by a person who holds two or more contradictory beliefs, ideas, or values. This discomfort is triggered by a situation in which a person's belief clashes with new evidence perceived by the person.
The biggest problem with big data is our inability to remain objective, to be flexible in our closely held beliefs despite the presence of new data.
The Jim Cramer story illustrates this perfectly. Two years ago, a company and its stock may have been doing great! So, Cramer recommends it. But two years is a lifetime in some industries and as new data comes in, recommendations change. But we’re so inflexible that we blame the Jim Cramer’s of the world for being “wrong” even though he was making his recommendations two years ago.
The recommendations weren’t wrong at the time. But, facts changed. Data changed. The world changed. We, as people, hate change.
And that’s the greatest problem we need to solve in order to fully utilize Big Data. We’ll never accept facts and data until we adopt a more flexible mindset. The best data-scientists in the world can’t overcome our ego and pride.
Take a look at any social media feed and you’ll likely see my favorite (Dear Wix, we need a sarcasm font) phrase: “Do your research.” This phrase is kind of like the modern-day equivalent of “you’re rubber, I’m glue.” I digress...
While most of us are pretty terrible at real research beyond a Google search…the more pressing issue is that even when presented with new information, we dismiss it outright. We give it no consideration because it doesn’t align with our beliefs.
Worse, the type of research we tend to conduct is done in support of confirmation bias, meaning, we’re only looking for data to “prove” we’re right. We ignore data in opposition. That's not research.
The threat to Big Data is opinion. Opinions are personal. When Big Data leads us to new, and before-unknown-findings that are opposed to closely held opinions, we will do whatever it takes to dismiss the findings.
We’re investing billions of dollars, immeasurable amounts of time into leveraging Big Data. But will we ever be flexible enough to fully realize Big Data’s potential?
Comments