Situation: During a major modernization project at Capital One, my team and I were tasked with building a real-time data system from our legacy mainframe, a crucial step for transforming the company’s infrastructure. The challenge was that the teams involved lacked advanced data engineering experience and were hesitant to adopt new ideas. Their initial solution for initializing the data was flawed and could cause long delays—up to 30 days—before the system would be ready to serve real-time data.
Task: To initialize this new system, the team had developed a method that involved using nightly data exports and then applying updates throughout the day to keep the system current. However, this approach had a significant flaw: the nightly exports weren’t exact. They often missed important updates, and it could take up to 30 days for the system to fully “catch up” with accurate data. Worse, if any bugs occurred, the entire process would need to restart, causing long delays in delivering real value to the business.
Action: I recognized the inefficiencies and risks in this approach and proposed a solution that would save months of work for multiple teams. Instead of relying on the nightly export, I designed a system that could continuously build a more accurate snapshot of the data. By leveraging the event history we already had, I created a pipeline that would “patch” the gaps in the export before we even began the data initialization. This way, if we ever had to restart, we wouldn’t have to wait 30 days for the system to stabilize; we would have an up-to-date view available almost immediately.
Result: This new solution saved eight teams months of work, dramatically improved the reliability of the system, and accelerated our entire modernization process. The impact of this success was so significant that I was offered a stretch role, leading more than 180 engineers under a different VP, recognizing the increased scope and strategic value of my work.
I’m particularly proud of this achievement because it showcases my ability to simplify complex problems, drive innovation, and scale impactful solutions across the organization.