Four Megatrends in Big Data in 2019 and Beyond

Table of Contents

2018 was the second consecutive year when Gartner published an obituary of Big Data. If this sounds like bad news, don’t panic! No one, including Gartner, thinks Big Data is dead. Au contraire, Big Data has grown so ubiquitous it became “just data”, or so argue the authors of the obituaries.

Data certainly was all over the place in 2018 so let’s see how the market has changed and what to expect in 2019 and beyond. This article covers 4 trends that will shape the data technology in the next few years.

Trend 1: From Big Data to “Just Data”

Looking back at the technology landscape of 2016–2018, the buzz surrounding Big Data has been on the decline:

At the same time, thousands of companies have been embracing data. Forbes and Dresner report that enterprise adoption of Big Data reached 58% in 2018, a significant increase from 2015’s modest, but respectable, 17%.

Big Data adoption statistics
Source: Dresner (via Forbes)

Data has reached the late majority of adopters in the 2017–2018 period, a trend that persists in 2019. The public no longer views data as a new exciting frontier — rather, a technology that companies must adopt to stay competitive. Big Data is now less of a buzzword, yet it’s now an integral component in the ongoing digital transformation of enterprises.

Trend 2: Machine Learning is the New Black

Now that Big Data is officially a boring technology, machine learning became another buzzword hitting the spotlight. In August 2018, deep learning reached the peak of Gartner’s Hype Cycle for Emerging Technologies. Machine learning–enabled data management also entered the pre-peak section of Gartner’s Hype Cycle for Data Management.

Gartner Hype Cycle for Data Management
Source: Hype Cycle for Data Management, 2018

This perfectly illustrates the general craze about machine learning — and explains why large data companies are also AI/ML companies.

As Dieter Bohn from the Verge pointed out, nowadays, Google can’t string five words together without mentioning AI. And it’s not just the search giant, either. Google Cloud, Microsoft Azure, and AWS are some of the biggest brands in machine learning as a service (MLaaS). Coincidentally, these three companies are also among the leaders in modern Big Data infrastructures:

 Google CloudAWSMicrosoft Azure
Cloud EDW (Data Warehouse)Google BigQueryAmazon RedshiftSQL Server Data Warehousing
Data StorageGoogle Cloud StorageAmazon S3Azure Blob Storage
Hadoop in the CloudCloud DataprocAmazon Elastic Map ReduceMicrosoft Azure HDInsight
Stream analyticsCloud DataFlowAmazon KinesisAzure Stream Analytics
Cluster servicesBorg, Kubernetes, OmegaElastic Container Service (Amazon ECS)Azure Kubernetes Service (AKS)
Analytics / BI platformsGoogle Cloud BI solutionAmazon QuickSightMicrosoft Power BI
ML as a ServiceCloud AIAmazon MLAzure Machine Learning service

Here’s another indicator of the skyrocketing prominence of AI and ML in the data scene. What has been Matt Turck’s Big Data Landscape in 2017 became the Big Data and AI Landscape in 2018:

Big Data and AI Landscape in 2018
Source: Big Data and AI Landscape in 2018 (Click to view the full version).

Trend 3: Everyone, to the Cloud!

The domination of AWS, Google Cloud, and Microsoft Azure in the data market marks another important trend. An increasing number of the modern-day adopters of Big Data show little interest in developing and provisioning on-premise Hadoop or Spark infrastructures. Instead, they chose to rely on the simplicity of fully-managed, serverless solutions on the cloud. And they have quite a few reasons to do so:

  • Low entrance threshold. With cloud, there’s no need to order and wait for server equipment, rent server rooms, license software, or set up the infrastructure. As a result, the adoption of Big Data on the cloud feels like less of a commitment.
  • Simple scalability. Services like Amazon S3 offer readymade infrastructures that easily scale past tens of trillions of objects. Scaling an on-premise infrastructure, on the other hand, requires new equipment, new architecture/code, and new effort.
  • Cost: 1TB of cloud object storage can cost as little as $20 per month, which compares favorably to what companies pay for in-house HDFS infrastructures.
  • The flexibility of FaaS. Handling distributed queries on an ad-hoc basis is the default use case for Spark. BigQuery offers exactly the same functionality, and you only pay for what you use, when you use it.
  • Language barrier. The advent of data science in the enterprise segment raises the need to write and iterate ML models in Python and R. On-premise Hadoop and Spark setups, on the other hand, natively favor Java and Scala.

Practical reasoning aside, the growing popularity of cloud platforms is perfectly in line with the psychology of the Product Adoption Curve.

The product adoption curve
Source: CrazyEgg

If you think about it, it’s been 13 years since Hadoop emerged and Big Data became a thing. People adopting this technology today are the late majority, and the late majority wants pre-assembled, value-added, and reasonably-priced solutions. That’s exactly how Amazon, Google, and Microsoft market AWS, Google Cloud, and Azure.

Trend 4: Zero Tolerance for Data Mismanagement

In 2019, data governance strategies might need a stronger focus on data security. There had been quite a few prerequisites for this last year:

  • In February 2018, the breach of the MyFitnessPal app exposed the data of 150 million users.
  • March 2018 saw the news of Cambridge Analytica’s misappropriation of the personal information of 87 million Facebook users. Later investigations revealed broader issues with data mismanagement at Facebook.
  • In July, the breach of the Polar fitness app compromised the location data of the US military and security personnel.
  • In August and September, a hacker group named Magecart had been hijacking the payments data of 50 million online shoppers via the Newegg platform.
  • In November, a security vulnerability of ElasticSearch exposed the data of 82 million users (57 million people and 26 million companies).
  • In December, the names, email addresses, and passwords of 100 million Quora users leaked to a third party.

These are just of few high-profile examples that made headlines last year. The good news is that the privacy issues of 2018 faced public and political resonance. Facebook’s Mark Zuckerberg had to testify before the US Congress, and the EU adopted the GDPR, a regulation aimed to protect user privacy. In early February of 2019, Cisco and Apple urged the US government to adopt similar regulations.

The side effect of these events is greater public scrutiny of how companies handle and protect user data. Naturally, this implies an opportunity for startups ready to offer solutions for data governance and data security. There will be other opportunities for startups and enterprises in the data space as well.

Opportunities in the data space in 2019

As adoption of data continues, startups will need to differentiate themselves to conquer a niche in the still booming market. According to the investor Matt Turk, the niches that offer space for growth in 2019 are:

  • Data governance and security
  • Solutions for real-time streaming and processing of Big Data
  • Data fabrics
  • Data visualization
  • GPU databases
  • AI (including AI devops solutions)
  • Solutions that enable and streamline the deployment of data science and machine learning solutions for enterprises.

Speaking of enterprises, InsideBigData names several opportunities for efficiency gains in 2019 and beyond:

  • According to InsideBigData, up to 80% of all collected data remains cold/unused. Finding solutions for utilizing this data would be game-changing.
  • The maximum cluster utilization across the tech industry is 70%% while the average is 45%. Both figures indicate a huge potential for optimization.
  • Firefighting issues take up to 70% of time spent by professionals involved in Big Data. The factors leading to this statistic involve skills shortage, systems/tools complexity and inadequacy.

Whether you are a data startup, or an enterprise looking to optimize efficiency, finding the right talent is paramount. The good news is that we specialize in helping companies reach talented software developers. Our engineers have worked on numerous products building and streamlining Big Data solutions or helping enterprises integrate with AWS, Google Cloud, and Azure.

If you are looking for someone who’s right for the job, contact us and ask our guys about our expertise with Big Data.


Established in 2010, AgileEngine is a privately held company based in the Washington DC area. We rank among the fastest-growing US companies on the Inc 5000 list and are recognized as a top-3 software developer in DC on Clutch.

Established in 2010, AgileEngine is a privately held company based in the Washington DC area. We rank among the fastest-growing US companies on the Inc 5000 list and are recognized as a top-3 software developer in DC on Clutch.

Related articles