During the summer of 2024, I interned at Robinhood Markets on the Capacitative Engineering and Analytics (CEA) team. This team focuses on ensuring efficient utilization of cloud resources to maximize cost-effectiveness. My work revolved around AWS products, including ElastiCache, DynamoDB, and OpenSearch. I contributed by deriving engineering insights from data, creating visualizations, and developing an algorithm to optimize cost savings using AWS reservations.

Data Engineering

To kick off my project at Robinhood, I delved into the company’s internal Data Lakes and AWS Pricing Models. A major challenge was accurately attributing costs to specific applications and teams so they could better understand their service upkeep expenses. Addressing this required building a robust data pipeline using Apache Spark and Hadoop, which aggregated and updated data in a SQL storage, orchestrated through Airflow.

Data Visualization

In the second phase of my Robinhood project, I utilized Databricks and Looker to design clear and accessible visualizations for the CEA team and other engineering managers. This phase focused less on programming and more on refining graphing techniques and aligning with stakeholder requirements. Through iterative feedback, I successfully delivered a dashboarding system that offered valuable insights into cloud spending for key AWS product areas.

Cost Savings Optimization

As a stretch goal near the conclusion of my Robinhood internship, I applied my data science expertise to develop a distributed process for analyzing past utilization data. The goal was to identify optimal reservation amounts for various dimensions, such as services, sizes, teams, and applications. By combining User-Defined Functions with gradient descent models, I effectively delivered a solution that highlighted the most cost-effective strategies, reducing both waste and missed savings opportunities.

Impact

In the end, my internship project at Robinhood was a significant success. I exceeded expectations, delivering more than initially requested. My work provided visibility into over $7.5 million in annual cloud spending and generated savings recommendations totaling approximately $800,000 annually.