Checkpoint at step 2000

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

1_Pooling/config.json

@@ -0,0 +1,10 @@
+{
+    "word_embedding_dimension": 1024,
+    "pooling_mode_cls_token": false,
+    "pooling_mode_mean_tokens": false,
+    "pooling_mode_max_tokens": false,
+    "pooling_mode_mean_sqrt_len_tokens": false,
+    "pooling_mode_weightedmean_tokens": false,
+    "pooling_mode_lasttoken": true,
+    "include_prompt": true
+}

README.md

@@ -0,0 +1,204 @@
+---
+language:
+- en
+license: apache-2.0
+library_name: sentence-transformers
+tags:
+- sentence-transformers
+- sentence-similarity
+- feature-extraction
+- code-retrieval
+- embeddings
+base_model: Qwen/Qwen3-Embedding-0.6B
+datasets:
+- aysinghal/code-retrieval-training-dataset
+pipeline_tag: sentence-similarity
+---
+
+# ide-code-retrieval-qwen3-0.6b-ebs128
+
+A [SentenceTransformer](https://www.sbert.net/) model fine-tuned from
+[Qwen/Qwen3-Embedding-0.6B](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B) for **IDE code retrieval** --
+mapping natural-language commit queries to relevant source code documents via
+dense vector similarity.
+
+> **Note:** This is an intermediate checkpoint at step 2,000 / 8,000
+> (25.0% through 3 epochs). Training loss is still decreasing,
+> so a later checkpoint may perform better.
+
+## Model Description
+
+This model encodes both short natural-language queries (commit messages, search
+queries) and longer code documents into a shared embedding space. Retrieval is
+performed by computing cosine similarity between the query embedding and
+candidate code embeddings.
+
+- **Base model:** [Qwen/Qwen3-Embedding-0.6B](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B) (0.6B parameters)
+- **Max sequence length:** 1024 tokens
+- **Output dimensionality:** 1024 (normalized)
+- **Similarity function:** Cosine similarity
+
+## Training Details
+
+### Dataset
+
+- **Source:** [aysinghal/code-retrieval-training-dataset](https://huggingface.co/datasets/aysinghal/code-retrieval-training-dataset)
+- **Total pairs:** 2,465,694
+- **Train split:** 2,342,409 pairs (95%)
+- **Eval split:** 123,285 pairs (5%)
+- **Text strategy:** truncate (max 4096 chars)
+- **Negatives:** Explicit hard negatives from the dataset
+- **Pre-tokenized:** Yes (token IDs stored on disk for zero-overhead data loading)
+
+### Loss Function
+
+[MultipleNegativesRankingLoss](https://www.sbert.net/docs/package_reference/sentence_transformer/losses.html#multiplenegativesrankingloss)
+(InfoNCE) with explicit hard negatives. Each training example consists of an
+anchor (query), a positive (relevant code), and a hard negative (similar but
+irrelevant code). In-batch negatives provide additional contrast.
+
+### Hyperparameters
+
+| Parameter | Value |
+|:---|:---|
+| Base model | `Qwen/Qwen3-Embedding-0.6B` |
+| Learning rate | 2e-05 |
+| LR schedule | Linear with warmup |
+| Warmup ratio | 0.1 |
+| Epochs | 3 |
+| Effective batch size | 128 |
+| Per-GPU batch size | 64 |
+| Gradient accumulation | 1 |
+| Max sequence length | 1024 tokens |
+| Precision | BFloat16 |
+| Gradient checkpointing | True |
+| torch.compile | Enabled (max-autotune) |
+| Seed | 42 |
+| Eval strategy | Every 1600 steps |
+| Early stopping patience | 3 |
+
+### Hardware
+
+- **GPUs:** 2x NVIDIA L40S
+- **Total training steps:** 8,000 (3 epochs)
+
+### Training Progress (at checkpoint step 2,000)
+
+- **Training loss:** 2.8207 (step 50) → 0.7387 (step 2000)
+- **Best eval loss:** 0.2037 (step 1,600)
+- **Progress:** 2,000 / 8,000 steps (25.0%)
+
+#### Evaluation Results
+
+| Step | Epoch | Eval Loss |
+|---:|---:|---:|
+| 0 | 0.00 | 1.4170 |
+| 1,600 | 0.09 | 0.2037 |
+
+<details>
+<summary>Full training loss history (click to expand)</summary>
+
+| Step | Epoch | Loss | Learning Rate |
+|---:|---:|---:|---:|
+| 50 | 0.0027 | 2.8207 | 1.23e-06 |
+| 100 | 0.0055 | 2.6561 | 2.48e-06 |
+| 150 | 0.0082 | 2.1570 | 3.73e-06 |
+| 200 | 0.0109 | 1.8427 | 4.98e-06 |
+| 250 | 0.0137 | 1.6992 | 6.23e-06 |
+| 300 | 0.0164 | 1.5763 | 7.48e-06 |
+| 350 | 0.0191 | 1.5178 | 8.73e-06 |
+| 400 | 0.0219 | 1.4620 | 9.98e-06 |
+| 450 | 0.0246 | 1.3918 | 1.12e-05 |
+| 500 | 0.0273 | 1.3362 | 1.25e-05 |
+| 550 | 0.0301 | 1.2610 | 1.37e-05 |
+| 600 | 0.0328 | 1.2662 | 1.50e-05 |
+| 650 | 0.0355 | 1.1824 | 1.62e-05 |
+| 700 | 0.0383 | 1.1612 | 1.75e-05 |
+| 750 | 0.0410 | 1.1657 | 1.87e-05 |
+| 800 | 0.0437 | 1.1154 | 2.00e-05 |
+| 850 | 0.0464 | 1.0884 | 1.99e-05 |
+| 900 | 0.0492 | 1.0507 | 1.97e-05 |
+| 950 | 0.0519 | 1.0255 | 1.96e-05 |
+| 1,000 | 0.0546 | 0.9912 | 1.94e-05 |
+| 1,050 | 0.0574 | 0.9509 | 1.93e-05 |
+| 1,100 | 0.0601 | 0.9108 | 1.92e-05 |
+| 1,150 | 0.0628 | 0.9017 | 1.90e-05 |
+| 1,200 | 0.0656 | 0.8805 | 1.89e-05 |
+| 1,250 | 0.0683 | 0.8652 | 1.88e-05 |
+| 1,300 | 0.0710 | 0.8438 | 1.86e-05 |
+| 1,350 | 0.0738 | 0.8249 | 1.85e-05 |
+| 1,400 | 0.0765 | 0.8118 | 1.83e-05 |
+| 1,450 | 0.0792 | 0.8413 | 1.82e-05 |
+| 1,500 | 0.0820 | 0.8085 | 1.81e-05 |
+| 1,550 | 0.0847 | 0.7821 | 1.79e-05 |
+| 1,600 | 0.0874 | 0.8028 | 1.78e-05 |
+| 1,650 | 0.0902 | 0.7820 | 1.76e-05 |
+| 1,700 | 0.0929 | 0.7595 | 1.75e-05 |
+| 1,750 | 0.0956 | 0.7295 | 1.74e-05 |
+| 1,800 | 0.0984 | 0.7334 | 1.72e-05 |
+| 1,850 | 0.1011 | 0.7484 | 1.71e-05 |
+| 1,900 | 0.1038 | 0.7308 | 1.69e-05 |
+| 1,950 | 0.1066 | 0.7228 | 1.68e-05 |
+| 2,000 | 0.1093 | 0.7387 | 1.67e-05 |
+
+</details>
+
+## Usage
+
+### Loading the Model
+
+```python
+from sentence_transformers import SentenceTransformer
+
+model = SentenceTransformer("aysinghal/ide-code-retrieval-qwen3-0.6b-ebs128")
+```
+
+### Computing Embeddings
+
+```python
+queries = [
+    "fix null pointer exception in user authentication",
+    "add retry logic to API client",
+]
+code_docs = [
+    "def authenticate(user):\n    if user is None:\n        raise ValueError...",
+    "class APIClient:\n    def request(self, url, retries=3):\n        ...",
+]
+
+query_embeddings = model.encode(queries)
+code_embeddings = model.encode(code_docs)
+
+# Compute cosine similarities
+from sentence_transformers.util import cos_sim
+similarities = cos_sim(query_embeddings, code_embeddings)
+print(similarities)
+```
+
+## Intended Use
+
+- **Primary use case:** Retrieving relevant code files/functions given a
+  natural-language query (commit message, bug description, feature request)
+- **Search pipeline:** Encode a corpus of code documents offline, then at query
+  time encode the query and find nearest neighbors via cosine similarity
+
+## Limitations
+
+- This is an **early checkpoint** (25.0% through training). The
+  loss curve is still decreasing, so later checkpoints will likely perform
+  better.
+- Trained on a specific code retrieval dataset; may not generalize to all
+  programming languages or query styles without further fine-tuning.
+- Max context is 1024 tokens -- very long
+  files are truncated.
+
+## Citation
+
+If you use this model, please cite the base model:
+
+```bibtex
+@article{qwen3embedding,
+  title={Qwen3-Embedding},
+  author={Qwen Team},
+  year={2025}
+}
+```

chat_template.jinja

@@ -0,0 +1,85 @@
+{%- if tools %}
+    {{- '<|im_start|>system\n' }}
+    {%- if messages[0].role == 'system' %}
+        {{- messages[0].content + '\n\n' }}
+    {%- endif %}
+    {{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
+{%- else %}
+    {%- if messages[0].role == 'system' %}
+        {{- '<|im_start|>system\n' + messages[0].content + '<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
+{%- for message in messages[::-1] %}
+    {%- set index = (messages|length - 1) - loop.index0 %}
+    {%- if ns.multi_step_tool and message.role == "user" and not(message.content.startswith('<tool_response>') and message.content.endswith('</tool_response>')) %}
+        {%- set ns.multi_step_tool = false %}
+        {%- set ns.last_query_index = index %}
+    {%- endif %}
+{%- endfor %}
+{%- for message in messages %}
+    {%- if (message.role == "user") or (message.role == "system" and not loop.first) %}
+        {{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>' + '\n' }}
+    {%- elif message.role == "assistant" %}
+        {%- set content = message.content %}
+        {%- set reasoning_content = '' %}
+        {%- if message.reasoning_content is defined and message.reasoning_content is not none %}
+            {%- set reasoning_content = message.reasoning_content %}
+        {%- else %}
+            {%- if '</think>' in message.content %}
+                {%- set content = message.content.split('</think>')[-1].lstrip('\n') %}
+                {%- set reasoning_content = message.content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
+            {%- endif %}
+        {%- endif %}
+        {%- if loop.index0 > ns.last_query_index %}
+            {%- if loop.last or (not loop.last and reasoning_content) %}
+                {{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content.strip('\n') + '\n</think>\n\n' + content.lstrip('\n') }}
+            {%- else %}
+                {{- '<|im_start|>' + message.role + '\n' + content }}
+            {%- endif %}
+        {%- else %}
+            {{- '<|im_start|>' + message.role + '\n' + content }}
+        {%- endif %}
+        {%- if message.tool_calls %}
+            {%- for tool_call in message.tool_calls %}
+                {%- if (loop.first and content) or (not loop.first) %}
+                    {{- '\n' }}
+                {%- endif %}
+                {%- if tool_call.function %}
+                    {%- set tool_call = tool_call.function %}
+                {%- endif %}
+                {{- '<tool_call>\n{"name": "' }}
+                {{- tool_call.name }}
+                {{- '", "arguments": ' }}
+                {%- if tool_call.arguments is string %}
+                    {{- tool_call.arguments }}
+                {%- else %}
+                    {{- tool_call.arguments | tojson }}
+                {%- endif %}
+                {{- '}\n</tool_call>' }}
+            {%- endfor %}
+        {%- endif %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {{- message.content }}
+        {{- '\n</tool_response>' }}
+        {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+    {%- if enable_thinking is defined and enable_thinking is false %}
+        {{- '<think>\n\n</think>\n\n' }}
+    {%- endif %}
+{%- endif %}

config.json

@@ -0,0 +1,63 @@
+{
+  "architectures": [
+    "Qwen3Model"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 151643,
+  "dtype": "bfloat16",
+  "eos_token_id": 151643,
+  "head_dim": 128,
+  "hidden_act": "silu",
+  "hidden_size": 1024,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "layer_types": [
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention"
+  ],
+  "max_position_embeddings": 32768,
+  "max_window_layers": 28,
+  "model_type": "qwen3",
+  "num_attention_heads": 16,
+  "num_hidden_layers": 28,
+  "num_key_value_heads": 8,
+  "pad_token_id": null,
+  "rms_norm_eps": 1e-06,
+  "rope_parameters": {
+    "rope_theta": 1000000,
+    "rope_type": "default"
+  },
+  "sliding_window": null,
+  "tie_word_embeddings": true,
+  "transformers_version": "5.3.0",
+  "use_cache": true,
+  "use_sliding_window": false,
+  "vocab_size": 151669
+}

config_sentence_transformers.json

@@ -0,0 +1,14 @@
+{
+  "prompts": {
+    "query": "Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery:",
+    "document": ""
+  },
+  "default_prompt_name": null,
+  "similarity_fn_name": "cosine",
+  "model_type": "SentenceTransformer",
+  "__version__": {
+    "sentence_transformers": "5.2.3",
+    "transformers": "5.3.0",
+    "pytorch": "2.10.0+cu128"
+  }
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1f136b8d148ee0452689b1ff967deb0a12b43a73bb34a8c5f64d59b5c870040f
+size 1191586416

modules.json

@@ -0,0 +1,20 @@
+[
+  {
+    "idx": 0,
+    "name": "0",
+    "path": "",
+    "type": "sentence_transformers.models.Transformer"
+  },
+  {
+    "idx": 1,
+    "name": "1",
+    "path": "1_Pooling",
+    "type": "sentence_transformers.models.Pooling"
+  },
+  {
+    "idx": 2,
+    "name": "2",
+    "path": "2_Normalize",
+    "type": "sentence_transformers.models.Normalize"
+  }
+]

sentence_bert_config.json

@@ -0,0 +1,4 @@
+{
+    "max_seq_length": 1024,
+    "do_lower_case": false
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:642b05b6b6732f9ef1189d89d58c713112ac377bc857b9633423ced970a111ae
+size 11423968

tokenizer_config.json

@@ -0,0 +1,14 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "is_local": false,
+  "model_max_length": 131072,
+  "pad_token": "<|endoftext|>",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}

trainer_state.json

@@ -0,0 +1,339 @@
+{
+  "best_global_step": 1600,
+  "best_metric": 0.20368987321853638,
+  "best_model_checkpoint": "./output/run_20260415_015154_truncate_hard/checkpoint-1600",
+  "epoch": 0.1092896174863388,
+  "eval_steps": 1600,
+  "global_step": 2000,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0,
+      "eval_loss": 1.4169917106628418,
+      "eval_runtime": 2162.9853,
+      "eval_samples_per_second": 56.998,
+      "eval_steps_per_second": 3.563,
+      "step": 0
+    },
+    {
+      "epoch": 0.00273224043715847,
+      "grad_norm": 38.5,
+      "learning_rate": 1.2250000000000001e-06,
+      "loss": 2.8207391357421874,
+      "step": 50
+    },
+    {
+      "epoch": 0.00546448087431694,
+      "grad_norm": 32.0,
+      "learning_rate": 2.475e-06,
+      "loss": 2.656055908203125,
+      "step": 100
+    },
+    {
+      "epoch": 0.00819672131147541,
+      "grad_norm": 18.75,
+      "learning_rate": 3.7250000000000003e-06,
+      "loss": 2.157032318115234,
+      "step": 150
+    },
+    {
+      "epoch": 0.01092896174863388,
+      "grad_norm": 18.125,
+      "learning_rate": 4.975000000000001e-06,
+      "loss": 1.8427268981933593,
+      "step": 200
+    },
+    {
+      "epoch": 0.01366120218579235,
+      "grad_norm": 18.625,
+      "learning_rate": 6.225000000000001e-06,
+      "loss": 1.6992074584960937,
+      "step": 250
+    },
+    {
+      "epoch": 0.01639344262295082,
+      "grad_norm": 20.5,
+      "learning_rate": 7.475000000000001e-06,
+      "loss": 1.5763182067871093,
+      "step": 300
+    },
+    {
+      "epoch": 0.01912568306010929,
+      "grad_norm": 19.625,
+      "learning_rate": 8.725000000000002e-06,
+      "loss": 1.5177923583984374,
+      "step": 350
+    },
+    {
+      "epoch": 0.02185792349726776,
+      "grad_norm": 22.125,
+      "learning_rate": 9.975000000000002e-06,
+      "loss": 1.461956787109375,
+      "step": 400
+    },
+    {
+      "epoch": 0.02459016393442623,
+      "grad_norm": 26.125,
+      "learning_rate": 1.1225000000000002e-05,
+      "loss": 1.3917791748046875,
+      "step": 450
+    },
+    {
+      "epoch": 0.0273224043715847,
+      "grad_norm": 20.5,
+      "learning_rate": 1.2475000000000002e-05,
+      "loss": 1.3361837768554687,
+      "step": 500
+    },
+    {
+      "epoch": 0.030054644808743168,
+      "grad_norm": 20.25,
+      "learning_rate": 1.3725000000000002e-05,
+      "loss": 1.2610369110107422,
+      "step": 550
+    },
+    {
+      "epoch": 0.03278688524590164,
+      "grad_norm": 22.125,
+      "learning_rate": 1.4975000000000001e-05,
+      "loss": 1.2662028503417968,
+      "step": 600
+    },
+    {
+      "epoch": 0.03551912568306011,
+      "grad_norm": 20.125,
+      "learning_rate": 1.6225e-05,
+      "loss": 1.1824015808105468,
+      "step": 650
+    },
+    {
+      "epoch": 0.03825136612021858,
+      "grad_norm": 21.0,
+      "learning_rate": 1.7475e-05,
+      "loss": 1.1612301635742188,
+      "step": 700
+    },
+    {
+      "epoch": 0.040983606557377046,
+      "grad_norm": 29.25,
+      "learning_rate": 1.8725e-05,
+      "loss": 1.1656887817382813,
+      "step": 750
+    },
+    {
+      "epoch": 0.04371584699453552,
+      "grad_norm": 19.625,
+      "learning_rate": 1.9975e-05,
+      "loss": 1.1154251098632812,
+      "step": 800
+    },
+    {
+      "epoch": 0.04644808743169399,
+      "grad_norm": 19.375,
+      "learning_rate": 1.9863888888888892e-05,
+      "loss": 1.0884272003173827,
+      "step": 850
+    },
+    {
+      "epoch": 0.04918032786885246,
+      "grad_norm": 18.5,
+      "learning_rate": 1.9725000000000002e-05,
+      "loss": 1.0506919097900391,
+      "step": 900
+    },
+    {
+      "epoch": 0.05191256830601093,
+      "grad_norm": 19.0,
+      "learning_rate": 1.958611111111111e-05,
+      "loss": 1.0255317687988281,
+      "step": 950
+    },
+    {
+      "epoch": 0.0546448087431694,
+      "grad_norm": 17.75,
+      "learning_rate": 1.9447222222222224e-05,
+      "loss": 0.9911944580078125,
+      "step": 1000
+    },
+    {
+      "epoch": 0.05737704918032787,
+      "grad_norm": 17.875,
+      "learning_rate": 1.9308333333333336e-05,
+      "loss": 0.9508729553222657,
+      "step": 1050
+    },
+    {
+      "epoch": 0.060109289617486336,
+      "grad_norm": 21.75,
+      "learning_rate": 1.9169444444444445e-05,
+      "loss": 0.9108209228515625,
+      "step": 1100
+    },
+    {
+      "epoch": 0.06284153005464481,
+      "grad_norm": 18.625,
+      "learning_rate": 1.9030555555555558e-05,
+      "loss": 0.9017164611816406,
+      "step": 1150
+    },
+    {
+      "epoch": 0.06557377049180328,
+      "grad_norm": 18.375,
+      "learning_rate": 1.8891666666666667e-05,
+      "loss": 0.8805242919921875,
+      "step": 1200
+    },
+    {
+      "epoch": 0.06830601092896176,
+      "grad_norm": 17.75,
+      "learning_rate": 1.875277777777778e-05,
+      "loss": 0.8652438354492188,
+      "step": 1250
+    },
+    {
+      "epoch": 0.07103825136612021,
+      "grad_norm": 24.375,
+      "learning_rate": 1.8613888888888893e-05,
+      "loss": 0.8438261413574218,
+      "step": 1300
+    },
+    {
+      "epoch": 0.07377049180327869,
+      "grad_norm": 15.8125,
+      "learning_rate": 1.8475000000000002e-05,
+      "loss": 0.8248822021484375,
+      "step": 1350
+    },
+    {
+      "epoch": 0.07650273224043716,
+      "grad_norm": 20.0,
+      "learning_rate": 1.833611111111111e-05,
+      "loss": 0.8117513275146484,
+      "step": 1400
+    },
+    {
+      "epoch": 0.07923497267759563,
+      "grad_norm": 17.75,
+      "learning_rate": 1.8197222222222224e-05,
+      "loss": 0.8413444519042969,
+      "step": 1450
+    },
+    {
+      "epoch": 0.08196721311475409,
+      "grad_norm": 19.5,
+      "learning_rate": 1.8058333333333336e-05,
+      "loss": 0.8084825134277344,
+      "step": 1500
+    },
+    {
+      "epoch": 0.08469945355191257,
+      "grad_norm": 21.125,
+      "learning_rate": 1.7919444444444446e-05,
+      "loss": 0.7820854949951171,
+      "step": 1550
+    },
+    {
+      "epoch": 0.08743169398907104,
+      "grad_norm": 17.25,
+      "learning_rate": 1.7780555555555555e-05,
+      "loss": 0.8028193664550781,
+      "step": 1600
+    },
+    {
+      "epoch": 0.08743169398907104,
+      "eval_loss": 0.20368987321853638,
+      "eval_runtime": 1970.35,
+      "eval_samples_per_second": 62.57,
+      "eval_steps_per_second": 3.911,
+      "step": 1600
+    },
+    {
+      "epoch": 0.09016393442622951,
+      "grad_norm": 22.75,
+      "learning_rate": 1.7641666666666667e-05,
+      "loss": 0.7820030212402344,
+      "step": 1650
+    },
+    {
+      "epoch": 0.09289617486338798,
+      "grad_norm": 19.125,
+      "learning_rate": 1.750277777777778e-05,
+      "loss": 0.7595337677001953,
+      "step": 1700
+    },
+    {
+      "epoch": 0.09562841530054644,
+      "grad_norm": 16.625,
+      "learning_rate": 1.7363888888888893e-05,
+      "loss": 0.7295108795166015,
+      "step": 1750
+    },
+    {
+      "epoch": 0.09836065573770492,
+      "grad_norm": 21.75,
+      "learning_rate": 1.7225000000000002e-05,
+      "loss": 0.7333818054199219,
+      "step": 1800
+    },
+    {
+      "epoch": 0.10109289617486339,
+      "grad_norm": 20.875,
+      "learning_rate": 1.708611111111111e-05,
+      "loss": 0.748420181274414,
+      "step": 1850
+    },
+    {
+      "epoch": 0.10382513661202186,
+      "grad_norm": 21.5,
+      "learning_rate": 1.6947222222222224e-05,
+      "loss": 0.7308194732666016,
+      "step": 1900
+    },
+    {
+      "epoch": 0.10655737704918032,
+      "grad_norm": 21.375,
+      "learning_rate": 1.6808333333333336e-05,
+      "loss": 0.7228285217285156,
+      "step": 1950
+    },
+    {
+      "epoch": 0.1092896174863388,
+      "grad_norm": 21.125,
+      "learning_rate": 1.6669444444444446e-05,
+      "loss": 0.7387281036376954,
+      "step": 2000
+    }
+  ],
+  "logging_steps": 50,
+  "max_steps": 8000,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 1,
+  "save_steps": 400,
+  "stateful_callbacks": {
+    "EarlyStoppingCallback": {
+      "args": {
+        "early_stopping_patience": 3,
+        "early_stopping_threshold": 0.0
+      },
+      "attributes": {
+        "early_stopping_patience_counter": 0
+      }
+    },
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 0.0,
+  "train_batch_size": 64,
+  "trial_name": null,
+  "trial_params": null
+}