Telling the Story: How an Effective Human Intelligence Team Performs Manual Fraud Review

When we talk about the benefits of employing a human intelligence team (several of which we discussed in a previous post), we tend to emphasize that manual fraud review is a highly specialized skill that should be performed by experienced fraud analysts. But why does manual fraud review work require so much experience? Why is it that a fraud analyst from a human intelligence team can make more accurate decisions on edge cases than a machine learning model trained for fraud prevention?

It all comes down to human intuition.

Edge Cases

Ecommerce businesses that employ machine learning (ML) in their fraud prevention strategies generally have the ML model as the first line of defense. In this setup, all transactions (purchases, sign-ups, rewards redemptions, etc.) flow through the model. Using transaction data (more on this below) the model makes a decision about the transaction’s riskiness compared to a designated risk threshold. If the transaction’s risk is high, the model will block the transaction. If the transaction’s risk is low, the model will approve the transaction and allow it to be processed.

If the model determines that the customer’s transaction has a medium risk, this transaction becomes what we call an edge case and is ideally sent to a human intelligence team for manual fraud review.

The Manual Fraud Review Process

A manual fraud review is simple on the surface: a fraud analyst reviews the transaction data and decides if the transaction should be blocked or approved. But the reality is much more nuanced. Key to understanding the complexities of manual fraud review is recognizing that every transaction is potentially made by a legitimate customer. A fraud analyst must put themselves in the customer’s shoes to accurately make a fraud/not fraud decision. During manual fraud review, a fraud analyst must look beyond the data and understand the customer’s experience as they made the transaction. This is something an ML model cannot do.

The Consequences of Manual Fraud Review

If a transaction is fraudulent, the fraud analyst will block it. This will prevent the transaction from being processed. If the transaction is a purchase order for a physical product, the product will not be shipped to the customer. If the transaction is a sign-up for a subscription, the account will likely be blocked and the subscription canceled.

These are important steps to take to prevent fraud loss for your business. But on a customer experience level, the stakes are high. After all, it would not be a good customer experience if a legitimate customer was accused of fraudulent activity and had their order cancelled. Good customer experience is the heart of brand loyalty, with one customer experience management company reporting that more than 75% of consumers say consistent customer experience improves their likelihood of doing repeat business with a brand.

For a human intelligence team performing manual fraud review, customer experience is always a priority as they analyze transaction data and make review decisions.

Transaction Data

Depending on the business and transaction type, each transaction could have dozens or even hundreds of data attributes associated to it. An experienced fraud analyst from a human intelligence team will have a good understanding of how to prioritize these attributes and which are most important to each manual fraud review.

High priority attributes can include:

• Account information (name, phone, email address)
• Account history (logins, purchases, chargebacks)
• Payment information (credit card details or payment email)
• Shipping or delivery information (location, location, location)
• Link Analysis (who else is using the same attributes)
• Web search results (social media links, phone, and address verification)

Not all transactions will have all of the information above, but this is typically where a fraud analyst will begin to “tell the story” of the transaction.

Telling the Story

A human fraud analyst can connect dots that an ML model cannot. In the manual fraud review space, we usually call this “telling the story,” where an analyst will review data from a transaction and build up a narrative of how and why the transaction was made. It involves the fraud analyst asking themselves questions like:

These questions involve role-playing and speculation, but an experienced fraud analyst doesn’t need to act them out. The questions are built into how the analyst reviews the data.

Putting it All Together

A fraud analyst may begin a manual fraud review by looking at transaction type. A purchase order will have a different review flow than a subscription signup or an account change such as updating a payment instrument.

Account Information

If account information exists for the transaction, a fraud analyst may start their manual fraud review there. Many platforms require customers to create accounts, though others allow purchases to be made without one. Account age can be an important indicator of fraud, as many bad actors prefer to create new accounts for each fraudulent transaction. On the other hand, older accounts that are suddenly displaying inconsistencies may be an indication of compromise or account takeover.

Phone and Email

Phone and email can also be important pieces of the puzzle. A reverse lookup can determine if the phone belongs to the account holder (and more importantly, to the owner of the payment instrument, if those two aren’t the same person). Email addresses can also be used to locate the customer’s social media or other web presence, leading to information (such as location, occupation, and travel habits) that can help build the story of how and why the transaction was created.

Even the composition of the email address can be important, as bad actors frequently use free email providers (Outlook, Gmail, etc.), and have generic usernames. However, a bad actor may also use an email with a legitimate business domain (especially if they’re planning on doing some phishing). In this case, email verification can be an important factor; many platforms require customers to respond (usually by clicking on a link) to an email sent to their sign-up address to verify that they have access to that address. This step is aimed at preventing bad actors from signing up with someone else’s address to which they do not have access.

Verification Data

Payment information can also provide verification data. If the transaction is made with a credit card, the fraud analyst can make use of the Address Verification Service (AVS). AVS is a credit card authentication service provided by major credit card processors for merchant use. When processing a transaction originating from one of the countries where AVS is supported (including US, Canada and UK), fraud analysts can use the service to determine if the billing address provided on the transaction matches what the card company has on file.

A decline from AVS may indicate a bad actor who has wrong information because it was stolen, but it could also indicate a legitimate customer who made a typo in their billing address or who has recently moved to a new location. There are also many fraudulent transactions where the billing address matches what’s available in AVS, because valid billing address is usually included when bad actors purchase black market credit card numbers.

Location

Location is an equally important component to the manual fraud review. A fraud analyst will compare the location of the billing address to the shipping location and to the IP address from which the transaction was made. If the payment instrument is a credit card, the fraud analyst can also identify the card’s country of origin using the Bank Identification Number (BIN). Again, this is a situation where a bad actor may make mistakes.

They may make the transaction from their actual location (usually another country or a different part of the same country) rather than from a location near the billing address. They may use a credit card that was issued in a country different from the account information they provided.

But they may get everything right and there will be no discrepancies. Legitimate transactions may also have discrepancies, as customers travel or send gifts to their friends and family in other locations.

Account History

Account history should also be taken into consideration. Are there previous purchases, and if so, are they consistent with what is being purchased now? This can be a difficult distinction to make because, barring auto-renew purchases, people rarely buy the same thing every time. But if all previous purchases were in a certain price range and the latest purchase is drastically different, that could be a sign of account takeover.

A history of chargebacks can also be enlightening, because it may indicate that a compromised payment instrument has been in use by a bad actor for a while. But chargebacks aren’t a surefire indicator of fraud, as they can also be made after legitimate purchases if, for example, a customer experiences transaction confusion (where they don’t recognize a charge they actually made).

Link Analysis

Finally, a fraud analyst may also perform a link analysis review on the transaction. Many fraud review tools are designed to flag data attributes that can link two or more customer accounts together. This could be through phone number, IP address, shipping address, payment instrument, etc. If these connections exist, a fraud analyst must ask why. It could be that two family members, both with accounts, are sharing the same device and IP address. It could be that two coworkers are sharing a company credit card. Or it could be that a bad actor has set up multiple accounts using the same stolen payment information. It’s the fraud analyst’s job to tell the story and uncover the most likely scenario.

As you can see, fraud can look like many things. It may be an old or a new account. It may be an email with a free domain or a custom domain. A bad actor may make mistakes with their location information, but they may not. If there are chargebacks, they may indicate fraud, but they may not.

There is no one answer for what fraud looks like. And that’s a huge part of the reason why it’s so important to have a human intelligence team reviewing your edge cases. ML models can detect predictable fraud with straightforward patterns, but only a human fraud analyst can build all this data into a story and make an accurate decision. And accuracy in this case can make the difference between a good customer experience and a bad one, and between fraud prevented and fraud loss for your business.