Amazon Web Services (AWS), the cloud computing division of Amazon, is making significant strides in addressing the issue of hallucinations in AI models. At the AWS re:Invent 2024 conference in Las Vegas, a new service called Automated Reasoning Checks was announced. This service validates a model's responses by cross-referencing customer-supplied information for accuracy. AWS claims it is the "first" and "only" safeguard for hallucinations, but this claim may be somewhat exaggerated.

Similarities and Differences with Other Providers

Automated Reasoning Checks is remarkably similar to the Correction feature Microsoft rolled out this summer. It also flags AI-generated text that might be factually wrong. Google offers a similar tool in its Vertex AI platform, allowing customers to "ground" models using data from third-party providers or their own datasets.In AWS's Bedrock model hosting service, specifically the Guardrails tool, Automated Reasoning Checks attempts to determine how a model arrived at an answer and discern its correctness. Customers upload information to establish a ground truth, and the tool creates rules that can be refined and applied to the model. As the model generates responses, Automated Reasoning Checks verifies them and provides the correct answer in case of a probable hallucination.AWS states that PwC is already using Automated Reasoning Checks to design AI assistants for its clients. Swami Sivasubramanian, VP of AI and data at AWS, believes that such tooling is attracting customers to Bedrock. The customer base of Bedrock grew by 4.7 times in the last year to reach tens of thousands of customers.However, as one expert pointed out this summer, trying to eliminate hallucinations from generative AI is like trying to eliminate hydrogen from water. AI models hallucinate because they don't actually "know" anything; they are statistical systems that identify patterns and predict answers based on previously seen examples. AWS claims that Automated Reasoning Checks uses "logically accurate" and "verifiable reasoning" to arrive at its conclusions, but it has not provided any data to prove the tool's reliability.

Model Distillation: Transferring Capabilities

In other Bedrock news, AWS announced Model Distillation today. This tool allows the transfer of the capabilities of a large model (such as Llama 405B) to a smaller model (like Llama 8B), which is cheaper and faster to run. It is a response to Microsoft's Distillation in Azure AI Foundry and provides a way to experiment with different models without incurring high costs.After the customer provides sample prompts, Amazon Bedrock will generate responses and fine-tune the smaller model. It can even create more sample data if needed to complete the distillation process. However, there are some caveats. Model Distillation currently only works with Bedrock-hosted models from Anthropic and Meta. Customers must select a large and small model from the same "family," and the distilled models will lose some accuracy, according to AWS (less than 2%).

Multi-Agent Collaboration: Streamlining Projects

Also available in preview is "multi-agent collaboration," a new Bedrock feature. This allows customers to assign AI to subtasks in a larger project. As part of Bedrock Agents and contributing to the AI agent craze, multi-agent collaboration provides tools to create and tune AI for tasks such as reviewing financial records and assessing global trends.Customers can designate a "supervisor agent" to break up and route tasks automatically. The supervisor can give specific agents access to the necessary information and determine which actions can be processed in parallel and which need details from other tasks before an agent can move forward. Once all the specialized AI complete their inputs, the supervisor agent can pull the information together and synthesize the results.This feature sounds promising, but like all new technologies, we will need to see how well it performs in real-world deployments.