Reinforcement Fine-Tuning (RFT) was the spotlight of the day for the Open AI “12 Days of OpenAI” event. Representing a new approach to AI customisation in the OpenAI Playground, this method shifts focus from replicating patterns to reasoning in domain-specific ways.  

Now, enterprises and researchers have the tools to create AI that operates with expert-level precision. Made available to alpha users, RFT is set for public rollout in early 2025, promising to redefine AI and its role across industries.

What You Need to Know

RFT is not just about improving models — it’s about making them smarter. OpenAI revealed details during the event, explaining how RFT changes the fine-tuning paradigm:

• Reasoning-Driven Training: RFT trains models to reason by rewarding correct reasoning pathways. Unlike traditional fine-tuning that aligns models with data patterns, this reinforcement-based approach allows AI to solve complex medical, law, and financial problems.

• Custom Graders: The cornerstone of RFT is the grader schema, a customisable mechanism for evaluating a model’s outputs. Users can design graders tailored to their needs or rely on OpenAI’s auto-generated schemas. These graders measure the reasoning quality behind outputs, ensuring alignment with expert thinking.

• Domain-Specific Expertise: Enterprises can fine-tune AI into experts in their field with minimal data. A domain-specific dataset, such as case law summaries or medical diagnostic records, is enough to create a highly accurate model.

• Three Fine-Tuning Methods: RFT comes alongside two other options—supervised fine-tuning (mimics data patterns) and direct reference optimisation (aligns outputs with reference answers)—offering flexibility based on use cases.

• Alpha Rollout and Beyond: The RFT feature is currently in alpha, accessible to a select group of users, with applications open for enterprises and researchers. OpenAI aims to refine the process based on their feedback before the 2025 release.

Why It Matters

RFT addresses a key gap in AI today: reasoning in specialised domains. Here’s why this development is transformative:

• Custom Reasoning in Any Domain: OpenAI highlighted how RFT enables models to reason like professionals in specific fields. For example, an AI fine-tuned with RFT can:

• Diagnose symptoms of rare diseases using limited medical data.

• Draft customised contracts with professional lawyer-level precision.

• Analyse emergent financial risks like a seasoned economist.

• Efficient Fine-Tuning: With RFT, models can be trained using hundreds of examples instead of the thousands or millions traditionally required. This reduces costs while enhancing the model’s accuracy and usability.

Demo Spotlight:
During the announcement, Justin Reese, a computational biologist from Berkeley Lab, demonstrated RFT’s power in a real-world scenario. He used RFT to fine-tune a model for identifying genetic causes of rare diseases. Using just a few hundred curated examples:

• The RFT-tuned model achieved 31% accuracy on its first attempt, compared to 25% by the base GPT-4 model.

• The grader rewarded outputs that followed logical, step-by-step reasoning to pinpoint genetic variations.

• The AI was also able to generalise to new examples more effectively, a key advantage of reinforcement-based training.

The Competitive Landscape

While OpenAI has taken a significant step with RFT, several other competitors are exploring similar reinforcement-based fine-tuning efforts. These include:

• DeepMind AlphaCode: Apply reinforcement learning to enhance reasoning in logic-heavy tasks, exploring diverse problem-solving paths.

• Anthropic Claude: Use feedback-driven loops to improve task-specific reasoning and align models with user intent.

• Meta’s Code Llama: Leverage reinforcement techniques like proximal policy optimisation (PPO) to refine complex problem-solving abilities.

While these programs align closely with RFT’s focus on advanced reasoning and explainability, none explicitly use grader-driven reasoning—the unique process employed by OpenAI. OpenAI’s RFT sets itself apart by using human feedback to actively guide how models assess and solve problems, enhancing transparency and accountability in high-stakes scenarios.

The Bigger Potential: Who Benefits?

RFT is designed to create AI systems that are experts, not generalists, that aim to benefit industry sectors such as:

• Healthcare: AI fine-tuned with RFT can analyse diagnostic imaging, identify rare genetic conditions, and recommend personalised treatment plans.

• Legal: OpenAI’s existing partnerships, such as with Thomson Reuters, can be enhanced with RFT, enabling AI-powered legal assistants to navigate dense contracts and laws.

• Engineering and Finance: RFT allows AI to tackle structural design analysis, market forecasting, and optimisation problems with reasoning that mirrors human experts.

RFT’s ability to deliver expert-level results with minimal training data is a game changer for smaller businesses or industries with niche requirements.

What’s the Secret Sauce?

The core of RFT lies in the interplay between graders and reinforcement learning:

• Graders as Teachers: Graders evaluate model outputs based on user criteria or auto-generated schemas. For example, a grader might assess whether the model explained a medical diagnosis logically and correctly.

• Reward and Penalty System: The model is rewarded for following correct reasoning pathways and penalised for producing faulty logic or incorrect answers. Over time, this shapes the model into a domain-specific expert.

• Customizability: Enterprises can tailor graders to match their specific needs, whether for diagnosing diseases, drafting contracts, or optimising workflows.

This system ensures the AI doesn’t just “guess” answers—it learns to think through them, making it ideal for domains where reasoning is critical.

What Comes Next?

OpenAI’s roadmap for RFT includes:

• Feedback-Driven Refinement: Insights from alpha testers will shape the final product, ensuring it meets the needs of diverse industries.

• Public Rollout in 2025: A broader audience will gain access in Q1 2025, including researchers and enterprises.

• Integration with ChatGPT Pro: High-compute RFT models will be included in the $200/month subscription, making advanced fine-tuning accessible to businesses of all sizes.

Additionally, OpenAI teased its voice-cloning feature, allowing users to replicate their voices by reading a short text. While not directly linked to RFT, this feature represents another step in AI customisation and is expected to launch with safeguards like age restrictions.

The Final Word

With Reinforcement Fine-Tuning, OpenAI is reshaping our thinking about AI’s capabilities. Models no longer just provide answers—they learn to reason like experts. Whether diagnosing diseases, drafting legal documents, or analysing market trends, RFT positions AI as a critical tool for solving complex, high-stakes problems.

Stay Curious with Us!”
Catch up on Day 1 of 12 with our blog on OpenAI’s O1 model, and follow along as we dive deeper into the world of AI each day.

‍