MLOps on Google Cloud for Explainability and Bias Detection
About the Customer & Challenges Faced:
An American multinational consumer credit reporting agency and is one of the three largest agencies. They collect and aggregate information on over 800 million individual consumers and more than 88 million businesses worldwide to create insights that help organizations make more informed decisions. The mortgage is one of the industries our client serves. They provide mortgage lenders with a 360-degree view of a borrower’s credit, capacity and collateral. Risk models eventually become less predictive or relevant due to evolving market conditions.
They wanted to leverage artificial intelligence for explainability and bias detection within their mortgage underwriting decisions. The situation demanded the need for an integrated platform for data scientists equipped with accelerators and tools to experiment, discover, share, and deliver insights.
Solution and Approach:
Based on the above problem statements, we collaborated with the client to build an advanced analytics solution for explainability and bias detection using the xpresso.ai platform, integrated with Google Cloud Service.
- Ingestdata from BigQuery
- Historical data of approx. 30,000 participants divided into train and test groups (80:20)
- Parameters associated with each record: Age, Work class, Education, Marital Status, Occupation, Relationship, Race, Sex/Gender, Work Hours, Native Country, Smoking, BMI, Active, Defaulter Store data in xpresso data versioning repository
- Store data in xpresso data versioning repository
- Perform EDA, visualization and transformation
- Push transformed data in xpresso data versioning repository
- Create and train a Sklearn model and validate it against the test data
- Integrate with Starfair tools for automatic bias detection and manual correction
- Mitigate bias and rerun the pipeline
- Store the transformed data in xpresso data versioning repository
- Deploy a Data Proc job
- The entire setup of xpresso resides on the Google platform and is closely integrated with the native stacks of Google provided tools & technologies such as Kubeflow, Kubernetes, Dataproc and Big Query
Leveraging xpresso.ai:
- By using xpresso.ai, one can leverage high-end data connectivity, efficient data versioning, perform exploratory data analysis and generate inferences using an intuitive process and through an industry-standardized manner.
- The unique, containerized platform-centric approach offered by xpresso.ai can be used to employ required infrastructure, deploy rapidly to multiple high-availability environments while aligning with best-in-class DevSecOps practices.
- xpresso.ai also brings in-depth QA-QC testing and logging frameworks, synchronous and asynchronous monitoring, and performance tracking ability.
- xpresso.ai also has SSO (single-sign-on) for various in-built tools and subsystems that make the platform access seamless throughout.
- In a nutshell, all the above features in a single plate under the same hood make xpresso.ai an unbeatable AI Ops framework.
