Fuzzy matching will help with the typos, but from experience we crafted nicknames by hand.

If your jurisdiction(s) have accessible company records then you could match on those names to determine which rows are official names. This solves half your problem, as you then just need to match the remaining rows to an accepted official name.

You could also modify Levenshtein distance so that dropping characters is free in an attempt to match full names with shorter names, but this will be computationally expensive.

In the past, I did fuzzy matching plus a manual selection for smaller lists like a few thousand strings. But for millions of rows, this is just impossible. So we are wondering whether AI-based approaches can help.

I would be wary - AI approaches tend to give you plausible answers, not true answers. Also it may be worth updating your post to make it clear that you're looking for AI solutions to your problem, rather than looking for data cleaning advice for a dataset that you are going to feed into a machine learning system (which is what I inferred)

There is definitely a tradeoff between accuracy and efficiency. We are not sure which approach would be better, so want to keep the discussion broad.

TF-IDF on letters. Not a lot of companies will have capital J and capital P in the name.

https://github.com/dedupeio/dedupe

https://github.com/Living-with-machines/DeezyMatch

This is a classic MDM (master data mgmt) problem while creating a crm system. If it’s US only u can start with D&B hierarchies esp if the companies are public. For many small businesses or starts c&b records don’t exist. There are tools like informatica / tibco that play in here.

Yes, most companies in our list are in the US, but the majority of them are privately held companies. Does tools like informatica / tibco charge tons of money?

The same problem exists with music artists. One solution is not to repeat the work that has already been done by others.

1. Scrape https://www.chartsurfer.de/archiv/artist-a.html for a list of all artist names.

2. Scrape https://www.discogs.com/de/artist/28795-Prince for a list of alias names for each artist.

3. Do just simple Levensthein for spelling errors and prompt the user if in doubt.

It's worth noting that many of these are different companies, or subdivisions of JPM. Depending on the application, that matters a lot.

Gonna be honest, that sounds like a horrible idea

So no. You can’t. This problem is solvable by having the correct data entered into your system.

You are lucky since they all in the US so mostly English I assume…

But the reality is that because you have a mix of different businesses under the same name (like you have listed here) the only person who can untangle this is the owner of the company which is named…

There are some cool ML tips you can find to reduce the complexity of this problem - but you will not ever solve it at the output stage -

This must be solved at input.

U need to find a method tobturn these names into a feature vector, such that in feature space similar names ate clustered together naturally. Start with standard string similarities to get the feature vector, if that does not result in sufficiently unambigious cluster formations proceed by lemmatization methods and if it still is not sufficient try out some prelearned mod ls to generate the feature encoding

Where can we get "alias names" for the universe of US companies?

<offtopic> Where do I get a large-ish list of company names? Also, product names. </>