The business driven development method was used to enable business analysts to use the Big Query SQL and JSON dependency definitions to generate data transformations thereby accelerating the time to deliver changes to the process.

The challenge

  • Provide an intuitive analytical framework and visualisation tools to allow the bank to effectively manage Liquidity Risk
  • Ingest trade data from across the divisions of the bank - investment banking, retail banking, commercial banking and private banking
  • The technical requirement was to replace part of the Big Data solution which had been deprecated, with a solution which was easy for the users to maintain and change
  • Whilst the previous non-GCP legacy solution did meet the business’ functional requirements, the GCP solution had to produce the same results over a parallel run of several months as the bank’s legacy solution
  • One of the key current areas of challenge was the use of Customer Supplied Encryption Keys (CMEK) capabilities within the platform such as automatic key rotation require that the encryption keys are owned by Google

The engagement

  • The project was delivered within a Google environment defined and managed by the client with the bank’s Information Security Policies mandated that Customer Supplied Encryption Keys (CMEK) were incorporated into the platform
  • Implementing the solution involved working closely with the bank’s infrastructure team on the following components:
    • IAM
    • Data Encryption & use of CMEK
  • Cloud Storage used as a staging area for all data entering the solution - custom metadata supplied which defines processing context and data lineage
  • Cloud Functions, once authorised for use will be used to trigger a more meaningful notification of data arrival to the orchestration service
  • A python-based orchestration service was built for another set of use cases at the bank - the python code developed to replace the legacy SAS code will initially be running on Compute Engine instances but will be migrated to Kubernetes Engine once the use of CMEK is enabled on GKE

The benefit

  • The data visualisation solution will either leverage existing bank technology investments or use low cost of ownership solutions from the Google Cloud Platform
  • Three Datalab components were developed for another use case, which could potentially be used to Monitor and Manage the ETL workflows - these components were:
    • Visualisation of ETL Workflow progress in the form of an interactive graph (DAG)
    • Datalab pivot table generator for data review
    • Datalab widget for executing ETL workflows
  • Customer Managed Encryption Keys are mandatory for the organisation in question in order for them to have their data stored and processed off premise in the Cloud
  • The selection of Dataflow/BigQuery as an ETL/Data Processing platform has resulted in:
    • Match or exceed existing SLAs
    • Delivery using open flexible technologies
  • Accurately calculated the daily LCR in line with both the bank’s and the regulators requirements