James McKinney and Volodymyr Tarnay of the Open Contracting Partnership have revealed ‘A delicate introduction to making use of AI in procurement’. It’s a very accessible and useful primer on a few of the most salient points to be thought of when exploring the potential of utilizing AI to extract insights from procurement huge knowledge.
The OCP introduction to AI in procurement supplies useful pointers in relation to activity identification, methodology, enter, and mannequin choice. I might add that an preliminary exploration of the likelihood to deploy AI additionally (and maybe before everything) requires cautious consideration of the extent of precision and the sort (and measurement) of errors that may be tolerated within the particular activity, and methods to check and measure it.
One of many essential and maybe extra obscure points coated by the introduction is how AI seeks to seize ‘that means’ with the intention to extract insights from huge knowledge. That is additionally a controversial subject that retains arising in procurement knowledge evaluation contexts, and one which triggered some heated debate on the Public Procurement Knowledge Superpowers Convention final week—the place, in my opinion, firms promoting procurement perception providers had been peddling hyped claims (see session on ‘Transparency in public procurement – Knowledge readability’).
On this put up, I enterprise some ideas on that means, AI, and public procurement huge knowledge. As at all times, I’m very inquisitive about suggestions and alternatives for additional dialogue.
That means
In fact, the idea of that means is complicated and open to philosophical, linguistic, and different interpretations. Right here I take a comparatively pedestrian and pragmatic method and, following the Cambridge dictionary, think about two methods during which ‘that means’ is known in plain English: ‘the that means of one thing is what it expresses or represents’, and that means as ‘significance or worth’.
To place it merely, I’ll argue that AI can not seize that means correct. It could possibly carry complicated evaluation of ‘content material in context’, however we must always not equate that with that means. This might be necessary afterward.
AI, that means, embeddings, and ‘content material in context’
The OCP introduction helpfully addresses this subject in relation to an instance of ‘sentence similarity’, the place the researchers are searching for phrases which might be alike in tender notices and predefined inexperienced standards, and due to this fact wish to use AI to match sentences and assign them a similarity rating. Intuitively, ‘that means’ could be necessary to the comparability.
The OCP introduction explains that:
Computer systems don’t perceive human language. They should function on numbers. We will symbolize textual content and different data as numerical values with vector embeddings. A vector is a listing of numbers that, within the context of AI, helps us specific the that means of data and its relationship to different data.
Textual content could be transformed into vectors utilizing a mannequin. [A sentence transformer model] converts a sentence right into a vector of 384 numbers. For instance, the sentence “don’t panic and at all times carry a towel” turns into the numbers 0.425…, 0.385…, 0.072…, and so forth.
These numbers symbolize the that means of the sentence.
Let’s examine this sentence to a different: “preserve calm and always remember your towel” which has the vector (0.434…, 0.264…, 0.123…, …).
One solution to decide their similarity rating is to make use of cosine similarity to calculate the gap between the vectors of the 2 sentences. Put merely, the nearer the vectors are, the extra alike the sentences are. The results of this calculation will at all times be a quantity from -1 (the sentences have reverse meanings) to 1 (identical that means). You would additionally calculate this utilizing different trigonometric measures resembling Euclidean distance.
For our two sentences above, performing this mathematical operation returns a similarity rating of 0.869.
Now let’s think about the sentence “do you want cheese?” which has the vector (-0.167…, -0.557…, 0.066…, …). It returns a similarity rating of 0.199. Hooray! The pc is right!
However, this methodology shouldn’t be fool-proof. Let’s strive one other: “do panic and by no means deliver a towel” (0.589…, 0.255…, 0.0884…, …). The similarity rating is 0.857. The rating is excessive, as a result of the phrases are related… however the logic is reverse!
I believe there are two necessary observations in relation to using that means right here (highlighted above).
First, that means can hardly be captured the place sentences with reverse logic are thought of very related. It’s because the strategy described above (vector embedding) doesn’t seize that means. It captures content material (phrases) in context (round different phrases).
Second, it’s not attainable to totally specific in numbers what textual content expresses or represents, or its significance or worth. What the vectors seize is the illustration or expression of such that means, the illustration of its worth and significance by way of using these particular phrases within the specific order during which they’re expresssed. The string of numbers is thus a second-degree illustration of the that means meant by the phrases; it’s a numerical illustration of the phrase illustration, not a numerical illustration of the that means.
Unavoidably, there may be loads scope for loss, alteration and even inversion of that means when it goes by way of a number of imperfect processes of illustration. Because of this the extra open textured the expression in phrases and the much less contextualised in its presentation, the harder it’s to realize good outcomes.
It is very important keep in mind that the present methods based mostly on this or related strategies, resembling these based mostly on giant language fashions, clearly fail on essential points resembling their factuality—which in the end requires checking whether or not one thing with a given that means is true or false.
It is a burgeoning space of technnical analysis however plainly even essentially the most correct fashions are inclined to hover round 70% accuracy, save in extremely contextual non-ambiguous contexts (see eg D Quelle and A Bovet, ‘The perils and guarantees of fact-checking with giant language fashions’ (2024) 7 Entrance. Artif. Intell., Sec. Pure Language Processing). Whereas that is a formidable function of those instruments, it may hardly be acceptable to extrapolate that these instruments could be deployed for duties that require precision and factuality.
Procurement huge knowledge and ‘content material and context’
In some senses, the appliance of AI to extract insights from procurement huge knowledge is properly suited to the truth that, by and enormous, current procurement knowledge could be very exactly contextualised and more and more considerations structured content material—that’s, that many of the procurement knowledge that’s (more and more) out there is captured in structured notices and tends to have a narrowly outlined and extremely contextual function.
From that perspective, there may be potential to search for implementations of superior comparisons of ‘content material in context’. However it will most probably have a tough boundary the place ‘that means’ must be interpreted or analysed, as AI can not carry out that activity. At most, it may assist collect the data, but it surely can not analyse it as a result of it can not ‘perceive’ it.
Coverage implications
For my part, the above exhibits that the potential of utilizing AI to extract insights from procurement huge knowledge must be approched with warning. For duties the place a ‘broad brush’ method will do, these could be useful instruments. They may also help mitigate the informational deficit procurement coverage and apply are inclined to encounter. As put within the convention final week, these instruments may also help get a way of broad traits or instructions, and might thus inform coverage and decision-making solely in that regard and to that extent. Conversely, AI can’t be utilized in contexts the place precision is necessary and the place errors would have an effect on necessary rights or pursuits.
That is necessary, for instance, in relation to the fascination that AI ‘enterprise insights’ appears to be triggering amongst public consumers. One of many points that saved arising considerations why contracting authorities can not profit from the identical advances which might be touted as being provided to (non-public) tenderers. The case at hand was that of figuring out ‘enterprise alternatives’.
A lot of firms are utilizing AI to assist searches for contract notices to focus on probably fascinating tenders to their purchasers. They provide providers resembling ‘tender summaries’, whereby the AI creates a one-line abstract on the idea of a contract discover or a young description, and this abstract could be mechanically translated (eg into English). In addition they provide search providers based mostly on ‘capturing that means’ from an organization’s web site and matching it to probably fascinating tender alternatives.
All these providers, nonetheless, are at backside a complicated comparability of content material in context, not of that means. And these are deployed to go from extra to much less data (summaries), which may scale back issues with factuality and precision besides in excessive circumstances, and in a setting the place getting it fallacious has solely a marginal value (ie the corporate will put aside the non-interesting tender and transfer on). That is additionally an space the place expectations could be managed and the place outcomes properly under 100% accuracy could be fascinating and have worth.
The other doesn’t apply from the angle of the general public purchaser. For instance, a abstract of a young is unlikely to have a lot worth as, with all chance, the abstract will merely affirm that the tender matches the marketed object of the contract (which has no worth, in another way from a abstract suggesting a young matches the enterprise actions of an financial operator). Furthermore, factuality is extraordinarily necessary and solely 100% accuracy will do in a context the place decision-making is topic to good administration ensures.
Subsequently, we should be very cautious about how we consider using AI to extract insights from procurement (huge) knowledge and, because the OCP introduction highlights, one of the necessary issues is to obviously outline the duty for which AI could be used. For my part, there are way more restricted duties than one may dream up if we let our collective creativeness run excessive on hype.