Telecommunications is an industry that touches all of us, whether through our broadband networks, our mobile devices, or access to TV and entertainment. Telco companies allow us to speak to each other, to share our thoughts, and to do business with anyone in the world.

It has always been an information-rich industry, but shifts in technology have led to an explosion of a variety of new and emerging sets of structured, unstructured, and semi-structured data sources – including everything from clickstream data, location data, mobile application logs, social media streams and sensor data.

Every operator knows that optimization of the customer experience is a revenue opportunity. Offering a personalized product or service to a customer at the right time means increased relevancy and a chance to improve engagement, revenue and brand loyalty. However, the data available to do so, while helpful, is highly siloed across the organization, varied in formats and schemas. It’s also expensive to store, process and manage, and therefore difficult to derive value from.

Telcos, while willing to devote resource to their data solutions, are seeing little of the return that they need to justify that investment. This begs the question: when the opportunities are so huge, why aren’t telco companies taking advantage? It’s high time to debunk the five biggest myths that are holding the telco industry back from realizing the potential of their data:

Myth 1: My data is too diverse to analyses

My data resides across hundreds of systems and platforms. With so much complex structured, unstructured, and semi-structured data existing in silos across my organization, there’s no way I can consolidate and centralize it all.

REALITY: “Big Data” is actually very diverse data coming from various sources, and need not always be highly structured and ready to use. For telco, the variety of data is particularly relevant; as they work across many different business streams and have access to not only huge volumes of data, but also diverse datasets. And this is where industry leading Big Data platforms such as Apache Hadoop excel. With a growing set of open source components that constitute the Apache Hadoop ecosystem, Telcos can now easily ingest, store, process, and analyze large volumes of data, regardless of its source, format or size.

Myth 2: Open source means open security

Any mention of “the cloud” or “open source” brings security to mind. Because Hadoop is open source, it’s perceived that security protocols governing data will naturally fall by the wayside.

REALITY: The Apache Hadoop ecosystem has come a long way since 2006. Powered by Apache Sentry, Record Service and other leading tools, Big Data platforms provide comprehensive, enterprise-grade security and governance for organizations to effectively ensure data security and compliance. These technologies deliver applications and frameworks for deploying, managing, and integrating the necessary data security controls demanded by today’s strictest regulatory environments and compliance policies.

Myth 3: Hadoop doesn’t provide adequate returns on investment

According to a Gartner survey, only 26% of respondents claimed to be either deploying, piloting or experimenting with Hadoop. It’s seen as too costly and is “overkill” for the problems businesses face.

REALITY: This is probably the biggest myth of all. Depending on the scope and scale of operations, enterprises have been able to drive significant cost savings in the range of 35- 90%, as compared to traditional data management mechanisms, by moving to Hadoop. Telcos, in particular, are using Hadoop as the data management platform to drive some of the most compelling use cases. An example of using Hadoop to derive value is to drive down revenue leakage and fraud.

Based on industry estimates, telcos lose approximately 2.8% of their revenues to fraud annually – costing the industry approximately US $38 billion every year. Telcos could in fact be adding that amount to their bottom line without selling any additional products or services. To compete and out-innovate cyberfraud criminals, leading Telcos are utilizing Apache Hadoop and machine learning algorithms to analyze large amounts of data to uncover fraud and revenue threats in real-time.

Myth 4: Hadoop is too difficult for my team to learn

Hadoop is very difficult with a need for complex skill sets. Only a few skilled data scientists and analysts that work for elite tech companies can derive real value from implementing Hadoop.

REALITY: Technological advancements have removed the perceived complexity. Every organization should be able to realize the value of their data and learn the applicable skills to make the most of it. Telco companies are able to save time and money on headhunting data scientists by training up their existing teams. Today there are a variety of workshops and certification programs on offer, including inexpensive self-paced Hadoop Basics tutorials, so teams can quickly get up to speed.

Myth 5: Legacy systems are too ingrained and complex to work with yet another data solution

Legacy systems still determine much of how organizations manage, analyze and gain value from their data. Organizations are understandably reluctant to invest in open source tools as they don’t want to add more complexity to an unwieldy stack that has to work across multiple business units and respond to multiple platforms.

REALITY: Apache Hadoop works seamlessly with legacy systems to organize data across analytics and management platforms. This isn’t about throwing out legacy systems, but rather consolidating them and pulling structured and unstructured data from unexpected sources. Hadoop-based solutions can help telcos to process and analyze both structured and unstructured data going back several years, complementing, rather than jarring with legacy systems. There may be value in the legacy systems that aren’t being mined but organizations don’t have the data platform in place to fully understand what it can achieve.