Announcing a framework to balance productivity and risk for enterprise deployments of GenAI models and releasing two fine-tuned models—Mistral QS-Sentry and Llama 3 QS-Sentry—with optimized productivity-risk profiles
Gartner Data & Analytics Summit 2024, LONDON, May 13, 2024 /PRNewswire/ — Forum Systems, a leader in LLM and API technologies, today announced the public release of two language models fine-tuned to optimize their productivity-risk profile from the Gartner® Data & Analytics Summit 2024. The research is discussed at length in two recent articles, Framework for LLM Selection by Balancing Model Risk with Workforce Productivity and Improving Productivity-Risk Profile through LLM Fine-tuning. This groundbreaking work presents a framework for balancing productivity and risk in GenAI deployments, an urgent question among business leaders today.
Results of base vs. fine-tuned Mistral and Llama 3 models
“LLMs security has its tradeoffs. More restrictive models will be safer but may hamper productivity. If enterprises aren’t measuring the productivity-risk balance of their models, they are in the dark about whether they’ve achieved an optimal tradeoff,” remarked Mamoon Yunus, CEO of Forum Systems. He continued, “classic machine learning metrics like precision and recall can serve as proxies for productivity and risk. Fine-tuning on extensive manual multi-vote labeled data, our LLMs show superior performance compared to base models.”
The fine-tuned models—Mistral QS-Sentry and Llama 3 QS-Sentry—are based on Mistral-7B-Instruct-v0.2 and Meta-Llama-3-8B-Instruct.
In the first article, Forum Systems developed a framework for balancing risk and productivity and assessed the productivity-risk profile of Mistral and Llama 3 before fine-tuning. It found that, when asked to classify prompts as either safe or unsafe, Mistral was more precise and thus aligned with higher productivity, while Llama 3 was more restrictive and thus aligned with lower risk.
The second article analyzed the models after they were fine-tuned on a hand-curated dataset of about 20,000 prompts. The study showed that the productivity-risk profile of both models can see meaningful improvements after fine-tuning. Forum Systems has released both fine-tuned models on Hugging Face to contribute to the broader community of researchers and those working in AI governance and AI alignment, believing its framework for analyzing the trade-offs between productivity will also prove valuable to business leaders deploying safe and effective GenAI offerings.
As Gartner analyst Arun Chandrasekaran recently stated, “Generative AI (GenAI) has the potential to transform businesses across industries. Most business and technology leaders believe that the benefits of GenAI far outweigh its risks.” He recommends, “Put responsible AI at the heart of your generative efforts. Promote harmonious interaction among humans and machines with design thinking and by incorporating human feedback into GenAI applications.” Forum Systems agrees with Chandrasekaran’s analysis and recommendation. Its work in optimizing productivity-risk profiles of small models demonstrates that enterprise-class responsible AI deployments are within reach through fine-tuning.
Source: Gartner, 10 Best Practices for Scaling Generative AI Across the Enterprise, Arun Chandrasekaran, Leinar Ramos, Alberto Pietrobon, 10 January 2024.
GARTNER is a registered trademark and service mark of Gartner, Inc. and/or its affiliates in the U.S. and internationally and is used herein with permission. All rights reserved.
About the Gartner Data & Analytics Summit 2024 Gartner analysts will provide additional analysis on data and analytics trends at the Gartner Data & Analytics Summits, taking place March 11-13 in Orlando, FL., March 26-27 in São Paulo, April 24-25 in Mumbai, May 13-15 in London May 21-23 in Tokyo, and July 29-30 in Sydney. Follow news and updates from the conferences on X using #GartnerDA.
About Forum SystemsForum Systems is leading the Enterprise GenAI revolution. Forum’s patent-pending QuantumSim™, together with Forum Sentry, mitigate the unpredictable nature of LLMs through integration with corporate APIs, ensuring LLM output is truthful and accurate. QuantumSim™ is secure by design and ships with industry-leading guardrails, moderation, data obfuscation, and observability for LLM-enabled offerings. Used by some of the largest global companies for building intelligent business workflows, Forum’s suite of products provides unique solutions that allow enterprises to reinvent themselves with GenAI.
SOURCE Forum Systems
