Skip to content

The Challenge

For years, HIDI had been utilizing a Hadoop platform to collect, extract, transform, and load the data their customers required. This type of technology environment presented a few specific challenges.

First, running IaaS (Infrastructure as a Service) in Azure was extremely expensive, costing more than $70,000/month. Additionally, IaaS required technical expertise that HIDI had to manage (ie: patching, scaling, upgrades, security, etc.). It also required them to keep Hadoop resources on staff to help them manage the technology and process.

Second, the platform frequently faced performance challenges. Specifically, it was not user-friendly, it was cumbersome to operate, and actions, queries, or tasks were often slow to load.

The Goal

To achieve a more modernized and easy-to-use experience, HIDI needed to transform from a Hadoop IaaS model to a more modern data platform utilizing PaaS (Platform as a Service) services in Azure, such as ADLS Gen2 and Azure Synapse Analytics.

The Solution

CEI created a modernized data platform in Azure, using Synapse Analytics and ADLS Gen 2. This solution switched HIDI from IaaS in Azure to PaaS, relying on Microsoft to do the heavy lifting of managing the infrastructure environment, and allowing HIDI to focus more on the experience and performance of their platform instead of on the underlying infrastructure.

Synapse Pipelines were developed to build data analytics and reporting products and host them in Dedicated SQL Pools. Additionally, Serverless SQL was configured for data lake exploration and reporting.

CI/CD pipelines in Azure DevOps were developed to provide automated, repeatable, and controlled releases to lifecycle environments.

The Stack

Azure Synapse Analytics | Azure Data Lake

The Results

By implementing CEI’s PaaS solution, HIDI was able to experience significant improvements to their platform, including:

Extreme Cost Savings

Reduced their monthly expenses from $70,000 to $30,000 per month.

Realization of the benefits of platform services

Autoscaling for performance, reduced management of the environment, and industry-accepted and expected automation.

Significant data processing performance enhancements

Reduced processing times and provided customers with more customizable, malleable data.

Improved ease of operations

Allowed for data engineers to monitor and update the system through a predefined process.

Our Services

Explore our end-to-end capabilities to drive digital transformation across your organization.