DeepSeek R-1 Model Overview and how it Ranks against OpenAI’s O1

DeepSeek is a Chinese AI company “devoted to making AGI a reality” and open-sourcing all its models. They began in 2023, but have actually been making waves over the past month or two, and specifically this previous week with the release of their two newest thinking designs: DeepSeek-R1-Zero and the more innovative DeepSeek-R1, likewise understood as DeepSeek Reasoner.

They have actually released not just the designs but also the code and assessment triggers for public use, in addition to an in-depth paper describing their approach.

Aside from developing 2 extremely performant designs that are on par with OpenAI’s o1 model, the paper has a great deal of important details around reinforcement knowing, chain of idea thinking, prompt engineering with thinking designs, and more.

We’ll begin by focusing on the training procedure of DeepSeek-R1-Zero, which uniquely relied entirely on reinforcement knowing, instead of traditional supervised knowing. We’ll then carry on to DeepSeek-R1, how it’s reasoning works, and some prompt engineering finest practices for thinking designs.

Hey everyone, Dan here, co-founder of PromptHub. Today, we’re diving into DeepSeek’s most current design release and comparing it with OpenAI’s thinking designs, particularly the A1 and A1 Mini models. We’ll explore their training process, thinking abilities, and some crucial insights into prompt engineering for reasoning designs.

DeepSeek is a Chinese-based AI company dedicated to open-source development. Their recent release, the R1 thinking model, is groundbreaking due to its open-source nature and innovative training techniques. This consists of open access to the designs, prompts, and research documents.

Released on January 20th, R1 accomplished impressive efficiency on various criteria, matching OpenAI’s A1 models. Notably, they also released a precursor model, R10, which serves as the structure for R1.

Training Process: R10 to R1

R10: This model was trained specifically utilizing reinforcement learning without monitored fine-tuning, making it the very first open-source model to achieve high performance through this technique. Training included:

– Rewarding proper responses in deterministic tasks (e.g., math issues).
– Encouraging structured reasoning outputs using design templates with “” and “” tags

Through countless versions, R10 developed longer reasoning chains, self-verification, and even reflective habits. For instance, during training, the model demonstrated “aha” minutes and self-correction behaviors, which are rare in conventional LLMs.

R1: Building on R10, R1 included a number of improvements:

– Curated datasets with long Chain of Thought examples.
– Incorporation of R10-generated reasoning chains.
– Human choice alignment for sleek reactions.
– Distillation into smaller models (LLaMA 3.1 and 3.3 at different sizes).

Performance Benchmarks

DeepSeek’s R1 design carries out on par with OpenAI’s A1 designs across lots of reasoning criteria:

Reasoning and Math Tasks: R1 rivals or surpasses A1 models in accuracy and depth of thinking.
Coding Tasks: A1 models typically perform much better in LiveCode Bench and CodeForces jobs.
Simple QA: R1 typically outpaces A1 in structured QA jobs (e.g., 47% precision vs. 30%).

One noteworthy finding is that longer thinking chains normally enhance efficiency. This aligns with insights from Microsoft’s Med-Prompt framework and OpenAI’s observations on test-time compute and reasoning depth.

Challenges and Observations

Despite its strengths, R1 has some limitations:

– Mixing English and Chinese responses due to a lack of supervised fine-tuning.
– Less refined reactions compared to chat designs like OpenAI’s GPT.

These issues were addressed during R1’s improvement process, including monitored fine-tuning and human feedback.

Prompt Engineering Insights

An interesting takeaway from DeepSeek’s research is how few-shot prompting abject R1’s performance compared to zero-shot or concise tailored prompts. This aligns with findings from the Med-Prompt paper and OpenAI’s suggestions to restrict context in thinking designs. Overcomplicating the input can overwhelm the design and minimize accuracy.

DeepSeek’s R1 is a significant action forward for open-source thinking models, demonstrating capabilities that measure up to OpenAI’s A1. It’s an exciting time to experiment with these models and their chat interface, which is complimentary to use.

If you have questions or wish to find out more, have a look at the resources connected below. See you next time!

Training DeepSeek-R1-Zero: A reinforcement learning-only approach

DeepSeek-R1-Zero stands apart from a lot of other cutting edge designs because it was trained using only support knowing (RL), no supervised fine-tuning (SFT). This challenges the present conventional technique and opens new chances to train reasoning designs with less human intervention and effort.

DeepSeek-R1-Zero is the first open-source design to verify that innovative reasoning capabilities can be established purely through RL.

Without pre-labeled datasets, the design finds out through trial and error, improving its behavior, parameters, and weights based solely on feedback from the services it creates.

DeepSeek-R1-Zero is the base design for DeepSeek-R1.

The RL procedure for DeepSeek-R1-Zero

The training process for DeepSeek-R1-Zero involved providing the model with numerous reasoning tasks, varying from math problems to abstract logic obstacles. The model generated outputs and was assessed based on its performance.

DeepSeek-R1-Zero received feedback through a reward system that helped guide its knowing procedure:

Accuracy rewards: Evaluates whether the output is correct. Used for when there are deterministic outcomes (mathematics issues).

Format rewards: Encouraged the design to structure its thinking within and tags.

Training prompt design template

To train DeepSeek-R1-Zero to produce structured chain of thought sequences, the scientists utilized the following prompt training design template, changing prompt with the thinking question. You can access it in PromptHub here.

This design template triggered the model to clearly describe its thought procedure within tags before providing the final response in tags.

The power of RL in thinking

With this training procedure DeepSeek-R1-Zero began to produce sophisticated thinking chains.

Through countless training actions, DeepSeek-R1-Zero developed to resolve increasingly complex problems. It learned to:

– Generate long thinking chains that allowed deeper and more structured analytical

– Perform self-verification to cross-check its own answers (more on this later).

– Correct its own errors, showcasing emerging self-reflective habits.

DeepSeek R1-Zero performance

While DeepSeek-R1-Zero is primarily a precursor to DeepSeek-R1, it still accomplished high performance on numerous criteria. Let’s dive into some of the experiments ran.

Accuracy improvements during training

– Pass@1 precision began at 15.6% and by the end of the training it enhanced to 71.0%, similar to OpenAI’s o1-0912 design.

– The red strong line represents performance with majority ballot (comparable to ensembling and self-consistency techniques), which increased precision even more to 86.7%, exceeding o1-0912.

Next we’ll take a look at a table comparing DeepSeek-R1-Zero’s efficiency throughout multiple thinking datasets against OpenAI’s thinking models.

AIME 2024: 71.0% Pass@1, slightly listed below o1-0912 however above o1-mini. 86.7% cons@64, beating both o1 and o1-mini.

MATH-500: Achieved 95.9%, beating both o1-0912 and o1-mini.

GPQA Diamond: Outperformed o1-mini with a rating of 73.3%.

– Performed much worse on coding jobs (CodeForces and LiveCode Bench).

Next we’ll take a look at how the reaction length increased throughout the RL training procedure.

This chart reveals the length of reactions from the model as the training process progresses. Each “action” represents one cycle of the model’s learning process, where feedback is provided based upon the output’s efficiency, examined using the timely template gone over earlier.

For each question (representing one step), 16 actions were sampled, and the average accuracy was determined to make sure stable assessment.

As training progresses, the design produces longer reasoning chains, allowing it to solve significantly complex reasoning jobs by leveraging more test-time calculate.

While longer chains don’t constantly ensure better results, they typically associate with enhanced performance-a trend also observed in the MEDPROMPT paper (check out more about it here) and in the initial o1 paper from OpenAI.

Aha minute and self-verification

One of the coolest aspects of DeepSeek-R1-Zero’s advancement (which also applies to the flagship R-1 model) is just how excellent the model ended up being at reasoning. There were advanced reasoning behaviors that were not clearly configured however emerged through its support finding out procedure.

Over thousands of training steps, the design started to self-correct, review problematic logic, and confirm its own solutions-all within its chain of thought

An example of this kept in mind in the paper, referred to as a the “Aha moment” is below in red text.

In this instance, the model literally stated, “That’s an aha moment.” Through DeepSeek’s chat feature (their variation of ChatGPT) this type of reasoning normally emerges with phrases like “Wait a minute” or “Wait, however … ,”

Limitations and challenges in DeepSeek-R1-Zero

While DeepSeek-R1-Zero had the ability to perform at a high level, there were some drawbacks with the model.

Language blending and coherence problems: The design occasionally produced responses that mixed languages (Chinese and English).

Reinforcement knowing trade-offs: The lack of supervised fine-tuning (SFT) meant that the design lacked the refinement required for completely polished, human-aligned outputs.

DeepSeek-R1 was developed to attend to these issues!

What is DeepSeek R1

DeepSeek-R1 is an open-source reasoning design from the Chinese AI lab DeepSeek. It constructs on DeepSeek-R1-Zero, which was trained totally with support learning. Unlike its predecessor, DeepSeek-R1 integrates monitored fine-tuning, making it more fine-tuned. Notably, it outperforms OpenAI’s o1 model on several benchmarks-more on that later on.

What are the primary distinctions in between DeepSeek-R1 and DeepSeek-R1-Zero?

DeepSeek-R1 develops on the foundation of DeepSeek-R1-Zero, which works as the base model. The two differ in their training methods and general efficiency.

1. Training approach

DeepSeek-R1-Zero: Trained entirely with reinforcement knowing (RL) and no monitored fine-tuning (SFT).

DeepSeek-R1: Uses a multi-stage training pipeline that consists of supervised fine-tuning (SFT) first, followed by the very same support learning procedure that DeepSeek-R1-Zero wet through. SFT helps improve coherence and readability.

2. Readability & Coherence

DeepSeek-R1-Zero: Dealt with language blending (English and Chinese) and readability concerns. Its thinking was strong, however its outputs were less polished.

DeepSeek-R1: Addressed these problems with cold-start fine-tuning, making reactions clearer and more structured.

3. Performance

DeepSeek-R1-Zero: Still an extremely strong thinking model, often beating OpenAI’s o1, however fell the language mixing concerns lowered use considerably.

DeepSeek-R1: Outperforms R1-Zero and OpenAI’s o1 on the majority of thinking standards, and the reactions are much more polished.

In short, DeepSeek-R1-Zero was a proof of principle, while DeepSeek-R1 is the fully enhanced variation.

How DeepSeek-R1 was trained

To tackle the readability and coherence problems of R1-Zero, the researchers incorporated a cold-start fine-tuning stage and a multi-stage training pipeline when building DeepSeek-R1:

Cold-Start Fine-Tuning:

– Researchers prepared a premium dataset of long chains of idea examples for initial monitored fine-tuning (SFT). This data was gathered utilizing:- Few-shot triggering with comprehensive CoT examples.

– Post-processed outputs from DeepSeek-R1-Zero, improved by human annotators.

Reinforcement Learning:

DeepSeek-R1 underwent the exact same RL process as DeepSeek-R1-Zero to improve its thinking capabilities even more.

Human Preference Alignment:

– A secondary RL phase enhanced the design’s helpfulness and harmlessness, guaranteeing much better positioning with user needs.

Distillation to Smaller Models:

– DeepSeek-R1’s thinking abilities were distilled into smaller, effective models like Qwen and Llama-3.1 -8 B, and Llama-3.3 -70 B-Instruct.

DeepSeek R-1 criteria efficiency

The researchers checked DeepSeek R-1 across a variety of standards and versus leading designs: o1, GPT-4o, and Claude 3.5 Sonnet, o1-mini.

The benchmarks were broken down into numerous classifications, shown listed below in the table: English, Code, Math, and Chinese.

Setup

The following criteria were applied throughout all models:

Maximum generation length: 32,768 tokens.

Sampling setup:- Temperature: 0.6.

– Top-p worth: 0.95.

– DeepSeek R1 exceeded o1, Claude 3.5 Sonnet and other models in the majority of reasoning benchmarks.

o1 was the best-performing model in 4 out of the 5 coding-related criteria.

– DeepSeek performed well on imaginative and long-context job job, like AlpacaEval 2.0 and ArenaHard, exceeding all other designs.

Prompt Engineering with reasoning designs

My preferred part of the post was the scientists’ observation about DeepSeek-R1’s level of sensitivity to prompts:

This is another datapoint that lines up with insights from our Prompt Engineering with Reasoning Models Guide, which referrals Microsoft’s research on their MedPrompt framework. In their research study with OpenAI’s o1-preview design, they discovered that frustrating thinking models with few-shot context deteriorated performance-a sharp contrast to non-reasoning models.

The key takeaway? Zero-shot triggering with clear and succinct directions appear to be best when utilizing thinking designs.