Yes, it is "all that different". You're comparing conceptual similarity, not data similarity. Data (and usually) appear similar without being conceptually similar, or appear different while actually being conceptually similar. A dolphin, a lungfish, a coelacanth, a tuna, a shark, and a hagfish may all appear visually to be the same sort of "thing" (fish), but they're all actually radically different things. What you actually want to be comparing is the core conceptual aspects of things, not their outward (raw data) appearances.

Oh, and I forgot to include the "why". TL/DR: it just works better.

To form a latent, the network fundamentally has to learn correlations, in order to be able to accurately compress and then decompress the information in its input space. Due to the limited size of the pinch, you simply can't store every possible data-space representation of a given input. The encoding down to these vectors must persist despite the sort of "real-world" variations that occur in the input space.

Within the latent space (latents are vectors), you can think of individual subconcepts as being directions with in the latent space. In the "king" / "queen" case, for example, adding "man" times some scalar shifts you forward and backwards along degrees of masculinity. In the case of a spectrum, these vectors represent various fundamental molecular properties of the material being scanned. By computing the cosine distance vs. known entries in the vector db and picking the most similar entries, you are in effect minimizing the distance between the molecular properties of the samples.

By contrast, you certainly CAN minimize distances in data space! The problem is that data space lacks correlations. Two things being physically similar in data space doesn't actually mean that much if the things that are different are extremely definitionally-critical aspects. By contrast, definitionally-critical aspects inherently become a key part of vector encoding. Trying to work in text space, for example, a distance algorithm might declare that "king" and "kingdom" are textually similar, but "queen" does not appear at all similar to it. But it's fundamentally similar in vector space.

For YHVH's sake, does nobody in this thread know what a vector DB is?

Translating that to signatures for chemicals is basic spectroscopy which has worked for decades.

The emerging standard for spectroscopy is to translate the spectrum into a latent vector and match against vector keys. For example, Spec2Vec.

This is not what you may at first be thinking of when you see vector, e.g. "the spectrum itself is represented as a vector". These vectors are latents, not spectra.

The most common way to create a latent is via an autoencoder. You task a neural network with recreating its input and put a "pinch" in the middle of the network. This forces the network, to maximize its predictive accuracy, to create a dense conceptual representation of the concepts at hand at the pinch: a latent. The left hand side of the network becomes a latent encoder (detecting concepts in the input space and encoding them into the latent space), while the right hand side becomes a latent decoder (translating the latent space back to data space).

(Note that Spec2Vec is a bit different - it's based on Word2Vec and treats the peaks and neutral losses as "words" and the entire spectrum as a "document", seeking to learn the correlations between peaks - but this is getting out into the weeds)

One of the properties of latents (in most latent spaces) is that they let you interpolate concepts (for the classic example, the latent for "king", minus the latent for "man", plus the latent for "woman", resembles the latent for "queen"), and they let you measure the conceptual distance between concepts (cosine similarity)

Once you have a trained encoder and decoder, a vector database can handle vector queries. The ideal situation would be that all rows would be keyed with vectors and the cosine similiarity to your query key is calculated vs. the table keys, but since that's not practical to query every row, Approximate Nearest Neighbor algorithms are used to approximate that. Hits matching a desired minimum level of similarity are returned. Note that the decoder is no longer needed, you only care about encoding the latents.

Search engines widely use vector databases, BTW, though they usually also combine them with text matching ("hybrid search") - the latter is fast and cheap, so it doesn't hurt to add it.

It's not "fuzzy matching", it's a vector database. You're looking up with conceptual keys into conceptual-indexed tables. The keys are latents, not fuzzy string-matching rules.

Um, no. AI was founded as an academic discipline in 1956, at the Dartsmouth Workshop. The term has been in widespread use for seven decades.

Also, AI != ML. ML is an AI technique. Most AI today is ML, but it wasn't always that way. Symbolic AI, Rules-Based AI, etc used to dominate before NNs advanced enough to push them aside. Expert Systems are a classic example of non-ML AI.

Digging more into RxScanner:

As far as I can tell, 100% of the information out there about its claimed efficacy comes from the manufacturer itself, with zero independent studies. Innovations for Poverty Action (IPA) Nigeria+ USAID + Bloom Public Health initiated a RCT that was supposed to be evaluating it against lab tests with results to be published in April 2025, but as far as I can tell, the results were never published.

From a technological perspective, if it is what it says at all, one would expect that it could be pretty good at confirming something as genuine and in pristine condition but pretty bad at telling if something is counterfeit. E.g. a generic of the same drug, or a slight formulation change (such as the binder or coatings), or some meaningless degradation during storage, will give a different result. Sounds like they're trying to bypass this by using AI to tell what differences are meaningful or not. But without third party validation.... *shrug*

Also, the manufacturer initially marketed itself as a hardware and SaaS company, but has now been broadening out, making e.g. a B2B marketplace where the sell drugs to pharmacies (RxDelivered) and a financing and point-of-sale software service for those pharmacies (RxPay). So as far as supply chain auditing goes, i's a red flag when the "independent" quality tester also operates the marketplace selling the goods (something that they try to spin as a "trusted network"). Because they obviously benefit from any biases to detect their goods as legit but flag competitors as fake.

As it stands, I don't think one can say it's an outright grift, but on the other hand, I wouldn't put too much trust in this company as it stands.

This is not fuzzy string matching. This is a vector database. Please learn the difference.

Vector databases return conceptual matches, not fuzzy data matches (such as fuzzy string matches).

Also: "The results of drug sample scans are recorded and stored on the blockchain", which isn't something that affects the choice of AI models, but does make me immediately suspicious that this whole thing is a grift / scam.

What are you talking about? You generally run inference on the GPU. Sometimes in RAM. Almost never on swap. It is certainly possible to run a model without being able to load the whole thing into memory, but it's comically slow.

When I was completing my hort sci degree a few years back we had a lecturer talking about how they were using AI to control tomato production in test greenhouses in the Netherlands (e.g. making decisions about when to vent, what to set the thermostat to, how much to water, what eC, how much light, when to prune, when to harvest, etc etc), and how it was trouncing human operators in maximizing yield * quality / cost.

The only complaint they were getting was the fact that they didn't factor in worker comfort into their model, so it was e.g. scheduling labour during times it also had the heat and humidity cranked way up ;) Disadvantage to a small bespoke model vs. a generalist.

I have a ":w! saves" mug ;)

I have to ask: did you literally never use a computer lab at all in the DOS era?

Not "logging into DOS" - logging into your account. I literally said "mimicked the DOS prompt, including common commands", e.g., you're at the DOS prompt. When you want to login, you ran LOGIN.EXE, which "mounted" your network account. I believe it was Novell NetWare-based.

Once the target enters the correct password, PamStealer displays a message stating that the file is damaged and can't be installed. This is designed to be a decoy to prevent the target from suspecting anything is amiss.

Same sort of technique I used back in secondary school, lol ;) We had a programming class (in Basic on DOS), and it was painfully trivial, so I'd always complete the assignments in like 5 minutes and then spend the rest of class messing around. So one thing I wrote was a program that mimicked the DOS prompt, including common commands, and when someone ran the login command and typed in their username and password, it would say that the password was incorrect so they'd think they had typed it wrong (while it was actually saving their username and password, then logging out of my account), so that when they tried again, it worked. I would launch on a bunch of computers in the lab after class when I could get away with it..

Among the passwords collected were the teacher's administrator username and password. So when it came time to write my final project for the course, among the various demo-style scenes in it was a stereogram generator. The hidden image in the stereogram was her username and password. ;)

(Thankfully she had a good attitude about it... seemed like she wanted to get mad at me but also found it funny. In retrospect, that could have gone very badly had she gotten angry...)

Yeah, this is what I always worry about when I see studies like this. I know they always try to control for confounders, but it's really hard to do right. If you mess up, you get another "Regular wine drinking improves your health!" craze (wine consumption is correlated with wealth and better access to healthcare, and also, people with serious health problems often have to give up drinking)