Migrating On-Prem Payment Processing Platform to AWS
Introduction
Efficiently processing payments is crucial in the rapidly evolving fintech. Recently, had the opportunity to work on a project where migrated an on-premises payment processing infrastructure to AWS. This transformation enhanced the system’s performance and integrated advanced data processing and automation capabilities, ensuring scalability, reliability, and cost efficiency. Here’s a deep dive into how accomplished this migration was and the benefits it brought to our client.
Project Overview
The project centered around a loan processing application for a fintech, where the primary objective was to migrate the on-prem payment processing infrastructure to AWS. This migration involved leveraging advanced cloud-based data warehouse solutions, sophisticated data modeling techniques, and comprehensive ETL tools to ensure seamless data processing and analytics.
Employed a robust and scalable architecture that included a wide array of AWS services to achieve this. The entire infrastructure was deployed across multiple environments using Infrastructure as Code (IaC) with Terraform, which ensured consistency, version control, and flexibility in managing the environments. This approach also facilitated disaster recovery and the ability to easily scale operations as the business grows.
Key Objectives
- Reduce Processing Time: The move to AWS significantly reduced processing time compared to the on-prem environment, providing a more efficient and responsive system.
- Automated Notifications: Set up a robust notification system using Amazon SES (Simple Email Service) and SNS (Simple Notification Service) to keep teams informed of critical events.
- Data Pipeline Monitoring: With Datadog and CloudWatch, ensured that the entire data pipeline was monitored effectively, with alerts set up for any anomalies.
- Security Compliance: Ensuring security compliance was paramount. Implemented Role-Based Access Control (RBAC), IAM, Service Control Policies (SCP), and encryption (AES-256 KMS) to meet PCI requirements.
- Data Warehousing: Utilized AWS Redshift and Athena for data warehousing and querying, providing the client with valuable insights through an easy-to-use dashboard powered by Amazon QuickSight.
Key Components of the Tech Stack
Automation And Processing:
AWS Lambda: Used extensively to automate various processes, such as converting NACHA files to CSV, handling file transfers, triggering crawlers, and sending email notifications.
AWS Glue: Played a central role in our ETL pipeline, automating the process of crawling, cataloging, and transforming data. This ensured that data was always ready for querying in Redshift or Athena.
AWS Step Functions: Orchestrated the complex workflows, ensuring that each step in the payment processing pipeline was executed in sequence and according to business logic.
AWS EventBridge: Used for event-driven architecture, triggering Lambda functions based on specific events, such as file uploads to S3.
Monitoring And Alerts:
Amazon CloudWatch: Provided real-time monitoring of the infrastructure and applications, enabling us to set up alerts for any unusual activity or potential issues.
AWS CloudTrail: Ensured that every API call was logged in the environment, providing a comprehensive audit trail for security and compliance purposes.
SNS and SES: Integrated for sending notifications and alerts to the team, keeping them informed of the system’s status and any critical events.
Security And Compliance:
IAM (Identity and Access Management): Managed access to AWS resources, ensuring that only authorized personnel could interact with the system.
AWS Secrets Manager: Securely stored and managed sensitive information such as database credentials and API keys.
AWS Security Hub and SCP (Service Control Policies): Implemented to ensure that the entire infrastructure adhered to industry best practices and compliance requirements, including PCI DSS.
Data Storage And Query:
Amazon S3: Acted as the central storage for all incoming and processed files, providing a scalable and durable solution for data storage.
Amazon Redshift: Used for data warehousing, enabling complex queries and analytics on the processed payment data.
Amazon Athena: Allowed us to perform ad-hoc queries on data stored in S3 without the need to move it into a database, enhancing our ability to quickly derive insights.
