Faster Performance
Improve resiliency and reduced process time (6 hours to 40 min).
Unification and standardisation
Streamlined the data warehouse work and cut the manual daily batch processioning time by 90%.
Scalability
Built capacity to move towards a scalable process, both vertically as well as horizontally, allowing to meet client needs.
Challenge
Seeking speed and scalability
The system used by a large U.S. investment firm to warehouse important data about trades, the value of its holdings, market movements, and more was being updated once a day in a batch process that took as long as three hours. Because this data warehouse provides inputs for a range of downstream internal reports each day, including important risk analyses, the slow batch processing was a pain point.
What’s more, the firm needed a data warehouse system that could scale for growth — more assets, new businesses, and new ways of using data. The time needed for processing was increasing along with the volume of data, and this meant that the entire system could soon become untenable. It was growing urgent to improve the performance of the data warehouse and the extraction, transformation, and loading (ETL) software.
The firm also wanted to make more data in the warehouse system available in real time to enable new data uses. This is part of the IT roadmap that GFT is helping the firm to realise, and addressing the batch processing times would be an important step.
Engagement
Streamlining data processes
Using the firm’s existing technology more efficiently, GFT streamlined the data warehouse work and cut the daily batch processing time to about 15 minutes — a more-than-90% reduction.
To do this, GFT engineers improved logic patterns and created better controls. They followed standard practices for the ETL software, which had been neglected over the years since the system initially was put in place. Each daily data run generates more than 44,000 files, and by streamlining the process it was possible to reduce the consumption time.
Optimising the data processes helps the firm to integrate cloud-based servers as it scales. Previously it had been limited to vertical scaling, adding more resources to existing on-site machines. With the improved processes in place, the data warehouse system has more flexibility to respond to demand in the future.
Benefit
More timely reports
As a result of the improvement to the data warehouse, which serves as a data synchronisation hub, a whole menu of internal reports can be ready earlier each day. The faster batch processing gives the firm’s traders, managers, analysts, and executives more timely data to help them maintain a full and clear understanding of the status of their investments.
The work on this issue was grounded in GFT’s deep understanding of the investment firm’s technology, processes, and data needs. GFT engineers had begun a proof of concept on solutions for the batch processing even before the client had raised any concerns about the data warehouse performance issues. The client didn’t have to acquire any unnecessary new technology or systems. Instead, GFT used what the company already had.
The firm now can add new business and expand data requirements without concerns about the flexibility or scalability of the data warehouse.
“GFT didn’t initially have a requirement from the client saying they needed us to reduce the batch processing time. But we saw the need, and we started work proactively to improve this system. In the end, these efforts brought real value to operations within this investment firm.”
Download - just one click away
-
Success story
Data warehouse optimisation drives investment firm’s growth
