Data platform migration and modernization on AWS significantly reduces passenger wait time for United Airlines

Solution

The data and analytics platform migration and modernization journey involved three steps:

• Rearchitecting the platform
• Using frameworks and accelerators
• Onboarding key use cases

The Impetus team helped in automating common repeatable patterns and helped enable templates and reusable components for design and architecting workloads to an AWS lakehouse architecture. United also used a cloud transformation accelerator called LeapLogic to convert a significant portion of data transformation code to AWS. This product of Impetus Technologies helped gain significant momentum and acceleration in building data pipelines on AWS. LeapLogic’s automation capabilities helped save 22% of the time and 70% of the effort compared to coding it manually. The team also created ingestion, monitoring, and validation frameworks to ingest data feeds from various upstream systems to Amazon S3. Implemented robust mechanisms to detect and remediate vulnerabilities and security risks if any by using Wiz and Veracode platforms.

Building the next gen unified data and analytics platform also involved building new and modern data processing logic on AWS Glue, Apache Spark on Amazon EMR, Amazon S3, and Amazon Redshift. To accomplish this, United used the all-in-one data pipeline platform Gathr. The AWS lakehouse architecture was built using AWS offerings like AWS Glue, Amazon Kinesis, Amazon Athena, Amazon Redshift Spectrum, Amazon EMR, and Amazon SageMaker.

United’s centralized data platform’s capabilities are as follows:

• Support for 200+ batch and 30+ real-time multi-TB data feeds
• Unified data catalog and governance to authorize, manage, and audit access to data
• Automated data quality checks (null, regex, data type, etc.)
• Single-click deployment for data pipelines and platforms using AWS CloudFormation templates and AWS CodeDeploy
• Serverless data pipelines leveraging AWS Lambda, Amazon Managed Airflow, AWS Glue, Amazon EMR, and Amazon CloudWatch
• Leverage IAM services to provided robust cloud security by enabling fine-grained access control and management of user identities, roles, and permissions to AWS resources
• Enabled permission boundary, MFA, Privilege Access Management, and CloudTrail logs to restrict and monitor IAM activities
• End-to-end DevOps for data platform and use cases
• Intelligent data profiling and data quality checks
• Enabled SSL/TLS at all storage layers including S3, Redshift, RDS etc along with support for TLS to secure the data at rest and at transit

• Support for 200+ batch and 30+ real-time multi-TB data feeds
• Unified data catalog and governance to authorize, manage, and audit access to data
• Automated data quality checks (null, regex, data type, etc.)
• Single-click deployment for data pipelines and platforms using AWS CloudFormation templates and AWS CodeDeploy
• Serverless data pipelines leveraging AWS Lambda, Amazon Managed Airflow, AWS Glue, Amazon EMR, and Amazon CloudWatch
• Leverage IAM services to provided robust cloud security by enabling fine-grained access control and management of user identities, roles, and permissions to AWS resources
• Enabled permission boundary, MFA, Privilege Access Management, and CloudTrail logs to restrict and monitor IAM activities
• End-to-end DevOps for data platform and use cases
• Intelligent data profiling and data quality checks
• Enabled SSL/TLS at all storage layers including S3, Redshift, RDS etc along with support for TLS to secure the data at rest and at transit

Impact

The AWS-based data platform enabled easy onboarding of use cases and ensured data accuracy and quality for downstream applications. It helped the airline improve the passenger experience by:

• Personalizing bundle offers to include customer preferred amenities (e.g., wi-fi, entertainment options, etc.) using ML models
• Reducing wait time at the airport gates by predicting passengers’ adherence to COVID travel requirements
• Auto-approving travel readiness by verifying the validity of COVID documents like test reports, vaccinations, government forms, etc.
• 500+ data quality checks are being monitored through the Observability platform

The flexible and scalable platform enabled unified governance and consumption and equipped the airline’s business teams to process data faster for real-time decision making and improved cost efficiency.