Summary
Brighter Super, a Queensland-based superannuation fund, faced several challenges with their new cloud data platform, including high costs, slow processing times, redundant code, and difficulties in onboarding new data sources. These inefficiencies hindered performance and escalated costs, impacting the organisation’s ability to support its growing operations and enhance customer experiences. In collaboration with Arinco, Brighter Super undertook a structured approach to optimise its data platform on Azure Databricks, implementing best practices and improving the overall infrastructure. As a result, they achieved a 90% reduction in costs, a 50% performance boost, and streamlined their codebase, making it easier to maintain and scale.
About Brighter Super
Brighter Super is a Queensland-based superannuation fund that has been providing trusted and reliable investment, advice, and insurance services for nearly 60 years. As a 100% member-owned fund, Brighter Super is committed to putting their members first in everything they do. With over 280,000 members and more than $34 billion in funds under management it has played a crucial role in securing the financial futures of Queensland local government employees and their spouses since 1965. In 2017, the fund transitioned to public offer model, welcoming members from private sectors as well as the broader community.
Setting up for future scalability
As Brighter Super expanded their operations, they sought to enhance their data platform built on Azure Databricks to improve customer experience and drive operational efficiency. The goal was to create a more streamlined, scalable, and cost-effective data environment that could support the organisation’s long-term growth and strategic objectives.
Platform challenges
Brighter Super’s new cloud data platform encountered several challenges that hindered performance and increased costs:
- Handling Large Data Volumes: Managing 350 million+ records required optimisation to improve processing times.
- High Costs: Inefficiencies in data processing led to excessive costs.
- Prolonged Job Execution: Some jobs were taking longer than expected indicating room for refinement.
- Code Simplification: Redundant code contributed to maintenance overhead and inefficiencies.
- Cumbersome new data source onboarding: Missing framework to onboard new data sources at scale onto the platform.
- Lakehouse Best Practices: Implementing additional optimisations would help algin better with best practices.
Arinco's approach
To address these issues, Arinco undertook a structured and data-driven approach:
- Assessment and Understanding: Conducted an in-depth analysis of the existing data platform to understand workloads, inefficiencies, and cost drivers.
- Solution Design and Benchmarking: Developed two alternative solutions to compare performance and cost efficiency against the current setup.
- Implementation of Best Practices: Introduced structured methodologies to optimise the lakehouse environment, ensuring scalability and maintainability.
- Upskilling and training: Provided training and support to the Brighter Super team to effectively use and maintain the new codebase, ensuring long-term sustainability.
- Lina Dickens, Head of Data Analytics and Peformance, Brighter Super
Key outcomes
Through Arinco’s intervention, Brighter Super achieved significant improvements in its data engineering processes:
- 90% Cost Reduction: Dramatic cost savings were realised through optimised data processing strategies.
- 50% Performance Boost: Data processing times were reduced thereby improving efficiency.
- 8,000+ Lines of Code Eliminated: Streamlined and optimised codebase to support easier maintainability.
- Scalable and Reusable Framework: Enabled easier ingestion of data from new sources, reducing future development effort.
- Improved Lakehouse Structure: Aligned with medallion architecture principles which enhanced data quality and reliability.
Conclusion
By leveraging Arinco’s expertise, Brighter Super transformed its data platform into a high-performing, cost-effective, and scalable system. The implemented solutions not only addressed immediate performance and cost concerns but also positioned Brighter Super for long-term data success. This project exemplifies how strategic data engineering optimisation can drive efficiency, scalability, and enhance member experiences in the financial services industry.
