AbstractPhila PRO

I can say with a solid underlined statement: The model DID WORK.

The preliminary chat amoe arm trained atop responded. The conversation was shallow but the conversation wasn't simple noise or chaos, the responses were confidently wrong, and effectively related to the questions and answers.

The structure in the huggingface space DOES NOT HAVE this chat arm yet, but I plan to have this fully functional and operational by tomorrow evening. Each arm needs to be tailored to the model because the model is brand new, the deviations from the core to the pretrain are still high, so the structure needs to be aligned with the AMOE arm correctly.

This is a sample from the arm based on 21.9k steps, roughly 11 billion tokens trained I think.

The core was not trained with chat, so she requires an anchored mixture of experts leg or a lora trained atop. She's quite compliant, so if you wish to train a lora she'll listen.

For now she's a next byte prediction auto-complete model, a pretrained trunk capable of AMOE expansion directly.

Once the pretrain concludes I'll prepare 2 chat arms.

After an anealment of around 2b tokens of various behavioral attunements, concepts, conversational learning, and so on; the stage 1 variant will have 2 core chat arms for direct huggingface space conversation.

Pre anealment chat arm, and post anealment chat arm. This will allow a structured fusion between the two behaviors and the necessary outcomes, creating a unique and interpretable duality between pure pretraining with a chat module, and post pretraining scattered conceptual finetuning with a chat arm. Post will essentially understand more about conversation, while the pre will have less knowledge and more a crash course in utilization.

I'll name them accordingly when the time comes, but they will both be Beatrix variants.

After this process we will begin forming candidate arm extensions for behavior. Concepts like wikipedia recall, toolchain utilization, mathematics, coding basics, and more. Very small pieces of the information for recall with the KV cache.

How effective they will be is another story. The tests and utilizations of the outcomes will show which arms are to be integrated into the larger form on pretrain. In other words, which arms will intentionally have their finetuning directly tied to the core and become post-train guarantees that don't decouple.

Stay tuned my friends. She's just getting started.

I have a list of upcoming prototype arms.

• Deterministic chat - Can we speak to the pretraining directly with an AMOE arm?
• Retokenization arm - Can we retokenize and cluster the bytes into BPE?
• Arbitration arm - Can we teach a small arbitration arm to communicate with another system?
• Mathematics arm - Multiple mathematics formats within an arm cluster.

This model will run on CPU. I ran it on windows 10 with my 12 core, roughly 10 bytes/s give or take, a fair prediction ratio for cpu.

Alright the prelim went well. We're at about 3.4 billion tokens learned.

Model very stable as the prelim tests showed. The structure is not collapsing and the model is in fact learning useful pathways of information. So far it hasn't established legitimate ground-form informational segmentation pathology, the model requires many more tokens for this.

Model learns at around 100k tokens per second scaled to the RTX 6000 PRO card available on colab, so the bf16 training is scaling nicely to the benefits from the blackwell hardware.

Each is packed with an fp8 variation for inference as well, which is substantially faster for inference, but they aren't very smart yet. They ARE available. The results show the correctly aligned fp8 variants are roughly the same accuracy on inference, but training they essentially collapse the attention in less than a thousand steps.

There is a full pretrain lineup ready. We're looking at maybe 80 billion tokens or so, and with that the chat AMOE cluster will be ran after each major finetune. We'll be drawing the conversation out of the model each major train, and with each train we'll determine if the model even needs to be finetuned with chat to allow the model to behave.

AMOE arm training planning begins already, as each layer can have direct integrated AMOE arms into the core for couple/decouple purposes. The structure itself will be learning chat through finetuning arms with chat directly onto the trunk. Based on the responses, we will continue pretraining the trunk until the construction of the AMOE chat conforms to the behavior.

So far it's showing the model likely does not require the chat to be baked into the core in finetune state. Prelim shows the chat is likely corrupting the core of the data compendium by learning it atop the core system. The AMOE could very well handle the first-class behavior, but there's no guarantees for this just yet.

wikitext 103 and fineweb main + extended are the preliminary.

These two are targeting;

  [  done  ] warmup_wikitext    wikitext-103   0.300/0.30B
  [  done  ] fineweb_main       fineweb-edu    3.000/3.00B
  [planned ] fineweb_extended   fineweb-edu    0.000/12.00B

So roughly 30 hours until fineweb's larger structure is trained. After this our first chat arms will be trained, which ought to let us have a conversation with her. See how well she took to the information.

https://huggingface.co/spaces/AbstractPhil/alephllm-chat

Alright I've begun training the prototype v1. The structure is holding together, the system aligning, and the subsystem is converging.
It works.

The smaller version is aligning and working. The erank is higher than expected, the system more confined, and some interesting effects already starting to emerge.

Huggingface spaces will have an application to speak with her up within the hour. She isn't chat trained yet, but will be very soon.

Geometric LLM.

Dusting up the week and preparing the article. It'll be an interesting read. I'm having Claude attend to a more readable format for this article, and with that the article will have the more technical documentation attached as links rather than interweaved. This should make it a bit more readable, as Claude Fable and Opus 5 are very strangely verbose and it seems many people have trouble reading the articles.

I admit they have gotten odd since Fable and Opus 5 started writing them. I have trouble following and reading the logic for some of the points in FT4, so I'll make this next article cleaner and more readable. I'll attempt to remedy this by having Claude produce a more human-reader friendly pair of articles, which I will directly audit, rewrite, and proofread the public-faced article myself while allowing Claude free reign on the more technical article.

Friday time.

This upcoming week I'll be preparing smaller tests, bulk analysis, structure tweaks, and a series of smaller autoregression trainings that will offer the answers to some pressing questions. I'll be micro managing the next week using Colab instead of automated runpod, so it may be a little more scattered, but Claude Code will keep the docket so it'll line up.

Some of the answers will make sense to the observer, some will make little to no sense at a topical glance. Just know that I'm following many routes of possibility at the same time, so the answers may not make sense at first glance. Having a goal like a large-scale differential homogenous structure with multiple communicative sub-structures speaking a new language that none of their teachers knew, is a tricky system and requires subtle nuances that I simply have to explore, crash the machines, and figure out the reasoning behind why the machines crash. This will require many tests, many analysis, and many wrong hypothesis. I'll try to fire through as many as quickly as possible without sacrificing the scientific method, so the AlephLM's next week iteration will potentially be stronger than this week.