The Audit Log Featured

The Audit Log Q4'24 – Laying the Foundations for the Future of Compliance

Q4 2024 Update

AJ Asver

Oct 17, 2024 • 13 min read

The Audit Log is a newsletter from Parcha where we share what we have been working on and what has been top of mind for our team over the last quarter.

At Parcha, we're building AI that helps banks and fintech onboard more customers faster with stronger compliance. In this inaugural edition of The Audit Log, we'll cover how we're laying the foundations for AI-accelerated compliance reviews with three primary use cases: due diligence, document verification, and AML screening combined with our AI model risk framework, which helps us achieve 99% accuracy in production.

In this edition, we also share our thoughts on "glass box" vs. "black box" models, how to approach working with an AI vendor, and our favorite articles from across banking, fintech, compliance, and AI.

We hope you enjoy The Audit Log. Let us know if you have any feedback!

Not yet a subscriber?

Three Core Use Cases for AI in Compliance

Over the last year, we've talked to dozens of fintechs and banks and identified three major pain points in compliance reviews: time-consuming due diligence, back-and-forth document collection and remediation, and high false positives in AML screening. Together, these three points create customer friction, operational inefficiency, and bottlenecks to growth. Over the last quarter, we've made huge progress on all three of these areas with our first due diligence product focused on businesses, a new real-time document verification API that integrates directly into onboarding and intelligent AML screening for adverse media, PEP, and sanctions.

Business Due Diligence in Minutes

0:00

/0:54

With just a website address, Parcha’s AI can scan the entirety of a business’s online presence as part of Know Your Business (KYB) onboarding. AI Compliance Reports leverage various AI models, including multi-modal ones, to build auditable SMB risk profiles. In doing so, Parcha’s team spent hundreds of engineering hours building AI pipelines to determine the relevance of data ingested across every media type, including sanctions and watchlists. This is necessary when onboarding small businesses with common names, global footprints, and complex ownership structures. Parcha does it all in minutes — not hours or days.

Try it for yourself. Enter any business website by clicking the button below to generate a free compliance report.

Generate a free compliance report

Real-time Document Verification

Are you tired of the back-and-forth of trying to validate business documents like registrations, licenses, and tax confirmations with customers? Our new real-time document verification process integrates with your onboarding flow via API to automatically verify documents in seconds and remediate issues directly with customers.

Intelligent Anti-Money Laundering Screening

In 2023, every major jurisdiction grew its sanctions list by more than 10%. 2024 already looks like it will bring double-digit growth in sanctions designations, resulting in more false alarms for compliance teams. As fintechs and banks expand to serve a global customer base, we're seeing a massive rise in sanctions, Politically Exposed Person (PEP), and adverse media alerts. Existing providers like Refinitiv, ComplyAdvantage, Thomson Reuters, and LexisNexis ensure you never miss a high-risk customer, but at the cost of reviewing thousands of false positives. Our intelligent Anti-Money Laundering (AML) screening approach integrates with your existing provider to carry out alerts reviews, reducing false positives by up to 90% before they reach your compliance team.

Learn more below:

Deep Dive: How our AI models achieve 99% accuracy in production

At Parcha, we understand that the effectiveness of our AI-powered compliance solutions hinges on the reliability and accuracy of our AI models. We define an AI model at Parcha to include the underlying LLM as well as the unique prompts we have developed for each specific use case. As we use AI for dozens of different use cases from name matching to document extraction, we have lots of models to validate! That's why we've developed a robust and repeatable framework to ensure our AI models delivers consistent, trustworthy results. We call it the Parcha Model Validation Framework, and it consists of three key pillars:

Rigorous Validation
Continuous Monitoring
Proactive Improvement

Let's dive into each of these pillars and explore how they work together to deliver excellence in AI-powered compliance.

Pillar 1: Rigorous Validation

Before any of our AI models are deployed, they undergo a comprehensive validation process. This includes:

Backtesting: We use historical client data and carefully crafted synthetic data to simulate real-world scenarios. This allows us to evaluate how our models would have performed in past situations.
Adversarial Testing: We intentionally challenge our models with edge cases and unexpected inputs using real and synthetic data. This stress-testing helps us identify and address potential vulnerabilities.
Domain-Specific Evaluation: We tailor our testing to each use case we serve, ensuring our models perform well in the contexts that matter most to our clients. In order to achieve this we generate synthetic data and golden data sets for each compliance check.

This rigorous validation process helps us catch and address potential issues before they can impact our clients' operations.

Pillar 2: Continuous Monitoring

Once a model is deployed, our work is far from over. We keep a constant eye on performance through:

Real-Time Performance Tracking: We monitor key metrics like precision and recall, ensuring our models maintain high accuracy.
False Positive Management: We've set an ambitious goal of minimizing false positives. We keep our false positive rate below 10% through careful tuning and vigilant monitoring.
User Feedback Integration: We actively seek and incorporate feedback from our clients, allowing us to quickly identify and address any issues that arise in real-world use.

This continuous monitoring allows us to maintain 99% accuracy in production and quickly address any deviations from expected results.

Pillar 3: Proactive Improvement

The world of compliance is ever-changing, and our models must evolve to keep pace. We ensure our models stay current through:

Dynamic In-context Learning: This innovative approach to prompt engineering allows us to continuously optimize our models based on new data and real-world performance using a combination of in-context learning and retrieval augmented generation.
State-of-the-Art Model Integration: We're always testing and integrating the latest advancements in AI to ensure our models remain at the cutting edge.
Regular Audits: We conduct both internal and third-party audits to review performance, regulatory compliance, and alignment with business goals.

This proactive approach to improvement ensures our models not only keep up with changes in the compliance landscape but stay ahead of the curve.

Model Validation Framework in Action: AI Name Matching

These three pillars don't operate in isolation – they work together in a continuous cycle of validation, monitoring, and improvement. For example, our continuous monitoring (Pillar 2) might identify a potential issue, which then feeds into our proactive improvement process (Pillar 3). The resulting updates would then go through our rigorous validation process (Pillar 1) before being deployed. This comprehensive, cyclical approach allows us to deliver AI-powered compliance solutions that our clients can trust are 99% accurate in production. With the Parcha Model Validation Framework, we're not just reacting to changes in the compliance landscape – we're anticipating them, ensuring our clients always have the most accurate, reliable, and up-to-date tools at their disposal.

Name matching model

Our AI Name Matching is one of our most frequently used models at Parcha, relevant to any product that evaluates individuals. Name matching at scale is challenging due to cultural variations, transliteration, phonetic similarities, and the list goes on. For example, in Eastern Europe, the name Sasha is often used as a short version of the formal name Alexander. To evaluate our AI Name Matching, we created a synthetic dataset of the most common names across a variety of different countries and cultures. We came up with a list of all of the different ways two names could be a valid match and developed a prompt that would carry out the logic for each type of match. To measure our performance, we evaluated the accuracy of name matches as our primary metric, with others like precision, recall, F1, and F3 as guardrails. This was the work we did as part of Pillar 1 of our framework.

Assessing pilot results

As part of our Pillar 2, while backtesting at the beginning of a recent pilot, we noticed that incorrect name matches were happening more frequently than anticipated. This prompted us to take a deeper look into our evaluation methods and the metrics we collect. As part of this exercise, we decided to break down our accuracy metrics by cultural segments, grouping names by geographical regions. In doing so, we discovered that while the overall metric was quite high, some categories were clearly underperforming, such as Western European names and East Asian names.

Improved Approach using RAG

Our AI engineering team then evaluated several approaches and tuned the model using retrieval augmented generation (RAG), loading few-shot examples into the prompt and allowing the model to learn dynamically in context. This brought the underperforming categories back up to the same level of accuracy as the rest. This experience highlighted the importance of granular analysis and cultural sensitivity in our AI models, ensuring high accuracy across diverse name origins.

Cultural Name Group Analysis - Before & After

Here’s how our name-matching model performed before and after adding dynamic in-context learning.

Name Group	Initial Accuracy	Final Accuracy	Improvement
African	92%	100%	8%
East Asian	75%	93%	12.35%
Eastern European	93%	100%	7%
Latin American	100%	100%	0%
Middle Eastern	100%	100%	0%
South Asian	100%	100%	0%
Southeast Asian	89%	100%	11%
Western	97%	100%	3%
Western European	82%	97%	12.46%
Overall	92%	99%	6.07%

The results speak for themselves. Using our model validation framework, we quickly identified an underperforming model and improved accuracy to 99% in production.

Why does model governance matter?

We've seen AI startups claim to have industry-leading accuracy but share very little about how this is measured systematically. As a fintech or bank, it's critical to understand how an AI vendor you're working with develops, monitors, and improves models. Your regulators and auditors will also care about this. Our framework has been developed in partnership with our customers to meet the requirements of publicly traded companies with the highest risk management and governance criteria.

Want to learn more about our model validation approach? Contact Dmytro Kovalchuk or Jonathan Johannemann on our AI Team.

💼 Customer case study

Parcha’s customers are saving (literally) hundreds of hours each month while mitigating risk in their industries. Check out the case study below to learn more...

🎙️ New Podcast: Compliance Accelerated

We’re experimenting with a new podcast, Compliance Accelerated. It’s a weekly podcast with bite-sized discussions on the future of finance and compliance. The reason is that the podcast is generated using AI.

Here’s the first two episodes:

🧠 Top of mind

Here's what is top of mind for Parcha in the fintech, banking, and AI space.

From us

The difference between a "glass box" vs. "black box" AI solution and why it matters for AML [Read Post]
Caution: Don't believe the AI Hype - How to thoughtfully integrate AI into your compliance/risk workflows [Read Post]
The AI Pricing Revolution – Why usage-based pricing is going to dominate AI software [Read Post]

On Fintech, Banking, and Compliance

On AI

📣 Updates

🗽 Parcha joins FIS accelerator

In September we joined the FIS Fintech Accelerator. We were one of only nine fintech startups accepted into the accelerator and are excited to partner with the FIS team to bring Parcha to more of their customers! Here’s a photo from FIS’s swanky new office in Midtown Manhatten—what a great view!

Read more about the accelerator from FIS here.

Money2020 Future of Fintech Dinner

The Parcha team will be at Money2020 at the end of this month and we're hosting a private dinner with Initialized Capital for fintech and bank leaders on Monday 28th, at 6pm at the Venetian. We will be discussing the future of compliance in the age of AI over great food and drinks. If you would like to join us please RSVP below:

RSVP to Money2020 Future of Fintech Dinner

Even if you're not free on Monday night we would still love to connect. Please reach out to AJ Asver on the conference app, email aj@parcha.ai or grab time here.

Want more?

Keep up with everything we’re doing by subscribing below!

Or... with our founder for a live demo or just to talk shop

Speak with Parcha

And don't forget to follow us on LinkedIn

About Parcha

Parcha enables banks and fintechs to onboard more customers faster, with stronger compliance using AI to accelerate KYC/B reviews, document verification and AML screening.

Learn more here 👉 https://parcha.com

You received this email because you expressed an interest in Parcha. If you don’t want to receive more emails from us, feel free to unsubscribe below!

Sign up for Parcha's Updates