Unlabeled Data Can Teach an Algorithm More than you Might Expect

-Murex AI Research explores how a few known examples could help map large datasets with less manual work and predefined rules-

Despite decades of investment in data platforms, one problem remains stubbornly difficult across financial institutions: connecting related information that lives in different systems.

Trades, cash flows, positions, counterparties and reference data are often represented in different ways across business processes. Making these datasets "speak the same language" requires significant mapping, reconciliation and maintenance effort. What if an algorithm could discover many of those connections automatically, using only a small number of known examples?

This question sits at the heart of one of the research topics being explored by Murex AI Research: data alignment. The aim is to find ways of connecting related data with much less manual mapping and fewer predefined rules.

Imagine having 1 million payment flows and want to map each of them to one of 5,000 loans. Because there are far more flows than loans, manually reviewing every relationship would be impractical. One possible approach is to ask an algorithm to first identify groups of similar flows before attempting to map them to the corresponding loans. This grouping process is known as clustering.

Clustering usually works without labels. The algorithm is not told in advance which loan each flow belongs to. Instead, it looks for similarities in the data and groups flows that appear to be related together.

However, these groups are rarely perfect. One group may contain flows linked to several loans, while the flows belonging to one loan may be spread across several groups. We therefore cannot simply check one flow, give its group the name of a loan and assume that every other flow in the group belongs to the same loan.

This led us to investigate a simple question: Can a small number of known examples help us understand the relationship between these groups and the underlying loans, allowing us to map the remaining flows much more accurately?

A flow whose correct loan is known is a labeled example. Such examples are valuable, but obtaining them can be expensive because someone—often a business expert—must check the correct answer. The objective is therefore not to remove labels completely, but to understand how few of them we need.

This is one of the first questions explored by Léo Saci, who has started a three-year PhD with Murex and Université Paris Dauphine–PSL. Léo’s Towards Data Science article “You Don’t Need Many Labels to Learn” presents initial findings ahead of a forthcoming research paper.

How did we test the idea?

To test the approach, Léo used a well-known reference dataset called EMNIST Letters. A reference dataset makes it possible to compare different methods while knowing the correct answer.

The dataset contains 145,600 pictures of handwritten letters. There are 26 possible answers: A, B, C and so on.

Léo used a clustering algorithm called GMVAE. The algorithm first examined all the pictures without using their labels. It looked for similarities and organized them into 100 groups.

As there are only 26 letters, the groups did not correspond directly to the alphabet. Different ways of writing an A could appear in several groups, while one group could contain several similar-looking letters.

For each picture, the clustering algorithm also calculated how likely it was to belong to each of the 100 groups. A picture could therefore be strongly linked to one group while also having similarities with several others.

The challenge was then to determine whether a very small number of labeled pictures could help interpret these 100 groups and correctly map them to the 26 letters.

A first method: hard decoding

The most straightforward approach is to assign each picture to the group it most likely belongs to and then use the few labeled pictures to determine which letter that group represents. This method, known as hard decoding, works well but ignores valuable information.

If an image could plausibly belong to several groups, only the most likely one is retained while the rest is discarded. It may give a picture a 45 percent probability of belonging to group 12 and a 40 percent probability of belonging to group 37. Hard decoding keeps group 12 and throws away the information from group 37. This matters because a group can contain several letters and the same letter can appear in several groups.

Our approach: soft decoding

Our approach, called soft decoding, keeps the full picture.

Rather than assigning an image to a single group, it considers the probability that the image belongs to every group. Together, these probabilities create a unique fingerprint.

Using a small set of labeled examples, the model learns the characteristic fingerprint of each letter. When a new picture arrives, soft decoding compares its fingerprint with those learned patterns and identifies the closest match. Instead of relying on a single clue, it uses all the available evidence.

This allows the algorithm to make better use of information that would otherwise be discarded, particularly when groups overlap or when the clustering process is uncertain.

Why does this matter? Because labeled data is expensive. Every label typically requires validation by a subject matter expert. If comparable results can be achieved with fewer labels, organizations can make better use of large datasets without requiring the same level of manual effort.

What did we find?

The initial results are encouraging. Soft decoding reached around 80 percent accuracy using only 0.2 percent labeled data, or approximately 291 labeled pictures.

By comparison, a standard supervised-learning approach using XGBoost, and trained only on labeled examples, required around 7 percent labeled data to reach a similar level of accuracy. In this experiment, that represented roughly 35 times more labels.

In experiments where labels are scarce, soft decoding also improved accuracy by around 18 percent compared with hard decoding using the same labeling budget.

Why does this matter?

Organizations often have far more unlabeled data than labeled data. Every label typically requires validation by a subject matter expert. The results suggest that large volumes of unlabeled data contain far more useful information than we often assume.

By keeping all the probabilities instead of forcing an early decision, soft decoding makes better use of this structure. On EMNIST Letters, it reached the same level of accuracy as a standard method trained only on labeled examples while using far fewer labels. It also performed significantly better than hard decoding when labels were very scarce.

While these findings are only an initial step, they point toward a broader opportunity: reducing the amount of manual effort needed to connect and understand complex datasets. For data-intensive industries such as capital markets, where labeled data is often difficult and expensive to obtain, that possibility is particularly compelling.

This is one of the questions Murex AI Research continues to investigate as part of its exploration of the how artificial intelligence will redefine capital markets and help the industry prepare for what comes next.

In July, we announced the launch of Murex AI Research. As part of its commitment to the research community, Murex AI Research will continue to share its work through research papers, technical articles and accessible summaries. Follow us on LinkedIn, where we’ll promote this work.