Are you interested in contributing to open source multilingual AI with Hugging Face and Argilla?

The MPEP initiative (https://github.com/huggingface/data-is-better-together/tree/main/prompt_translation) of the Data is Better Together project offers the opportunity to do just that by helping to create multilingual model checkpoints.

If you're interested in contributing to the Russian-language dataset, please get in touch as I am the Russian-language lead. If you're interested in contributing to another language, the MPEP link above has all the information you need to do so. 🤗

davanstrien samusenps victor taufiqdp osanseviero Priceva sedayilmazer dvilasuero clefourrier kristaller486 ZennyKenny alielfilali01

davanstrien samusenps osanseviero cstr dvilasuero clefourrier ZennyKenny alielfilali01 Tonic


➡ 𝗟𝗟𝗠-𝗮𝘀-𝗮-𝗷𝘂𝗱𝗴𝗲 𝗰𝗼𝗼𝗸𝗯𝗼𝗼𝗸:

➡ 𝗟𝗟𝗠-𝗮𝘀-𝗮-𝗷𝘂𝗱𝗴𝗲 𝗰𝗼𝗼𝗸𝗯𝗼𝗼𝗸:

➡ 𝗔𝗹𝗹 𝗰𝗼𝗼𝗸𝗯𝗼𝗼𝗸𝘀:

➡ 𝗔𝗹𝗹 𝗰𝗼𝗼𝗸𝗯𝗼𝗼𝗸𝘀:

𝗨𝘀𝗶𝗻𝗴 𝗟𝗟𝗠-𝗮𝘀-𝗮-𝗷𝘂𝗱𝗴𝗲 🧑⚖ 𝗳𝗼𝗿 𝗮𝗻 𝗮𝘂𝘁𝗼𝗺𝗮𝘁𝗲𝗱 𝗮𝗻𝗱 𝘃𝗲𝗿𝘀𝗮𝘁𝗶𝗹𝗲 𝗲𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗼𝗻

Evaluating LLM outputs is often hard, since many tasks require open-ended answers for which no deterministic metrics work: for instance, when asking a model to summarize a text, there could be hundreds of correct ways to do it. The most versatile way to grade these outputs is then human evaluation, but it is very time-consuming, thus costly.

🤔 Then 𝘄𝗵𝘆 𝗻𝗼𝘁 𝗮𝘀𝗸 𝗮𝗻𝗼𝘁𝗵𝗲𝗿 𝗟𝗟𝗠 𝘁𝗼 𝗱𝗼 𝘁𝗵𝗲 𝗲𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗼𝗻, by providing it relevant rating criteria? 👉 This is the idea behind LLM-as-a-judge.

⚙ To implement a LLM judge correctly, you need a few tricks.
✅ So 𝗜'𝘃𝗲 𝗷𝘂𝘀𝘁 𝗽𝘂𝗯𝗹𝗶𝘀𝗵𝗲𝗱 𝗮 𝗻𝗲𝘄 𝗻𝗼𝘁𝗲𝗯𝗼𝗼𝗸 𝘀𝗵𝗼𝘄𝗶𝗻𝗴 𝗵𝗼𝘄 𝘁𝗼 𝗶𝗺𝗽𝗹𝗲𝗺𝗲𝗻𝘁 𝗶𝘁 𝗽𝗿𝗼𝗽𝗲𝗿𝗹𝘆 𝗶𝗻 𝗼𝘂𝗿 𝗛𝘂𝗴𝗴𝗶𝗻𝗴 𝗙𝗮𝗰𝗲 𝗖𝗼𝗼𝗸𝗯𝗼𝗼𝗸! (you can run it instantly in Google Colab)
➡ 𝗟𝗟𝗠-𝗮𝘀-𝗮-𝗷𝘂𝗱𝗴𝗲 𝗰𝗼𝗼𝗸𝗯𝗼𝗼𝗸: https://huggingface.co/learn/cookbook/llm_judge

The Cookbook is a great collection of notebooks demonstrating recipes (thus the "cookbook") for common LLM usages. I recommend you to go take a look!
➡ 𝗔𝗹𝗹 𝗰𝗼𝗼𝗸𝗯𝗼𝗼𝗸𝘀: https://huggingface.co/learn/cookbook/index

Thank you @MariaK for your support!

MariaK lewtun loubnabnl andrewrreed AtAndDev ajibawa-2023 andysalerno samusenps Csplk osanseviero Priceva avinash02 clefourrier

MariaK lewtun loubnabnl andrewrreed AtAndDev samusenps Zyn123 osanseviero


BioT5: Enriching Cross-modal Integration in Biology with Chemical
  Knowledge and Natural Language Associations (2310.07276)

Extreme Compression of Large Language Models via Additive Quantization (2401.06118)

QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Models (2310.16795)

Diaries of Open Source. Part 6!

🏎xAI releases Grok-1, a 314B MoE
Blog: https://x.ai/blog/grok-os
GH repo: https://github.com/xai-org/grok-1
Model: https://hf.co/xai-org/grok-1

🕺MusicLang, a model for controllable music generation
Demo: https://hf.co/spaces/musiclang/musiclang-predict
GH repo: https://github.com/musiclang/musiclang_predict

🔬BioT5: a family of models for biology and chemical text tasks
Base model: https://hf.co/QizhiPei/biot5-base
Model for molecule captioning and design: https://hf.co/QizhiPei/biot5-base-mol2text and https://hf.co/QizhiPei/biot5-base-text2mol
GH Repo: https://github.com/QizhiPei/BioT5
Paper: https://hf.co/papers/2310.07276

🤏Check out the AQLM and QMoE official weights from ISTA-DAS lab
Org: https://hf.co/ISTA-DASLab
Papers: https://hf.co/papers/2401.06118 and https://hf.co/papers/2310.16795

🚀Community releases
Einstein-v4-7B, a Mistral fine-tune on high-quality data https://hf.co/Weyaxi/Einstein-v4-7B
IL-7B, a Misttral fine-tune merge for rheumatology https://hf.co/cmcmaster/il_7b
 Caselaw Access Project, a collaboration to digitalize 40 million US court decisions from 6.7 million cases from 360 years https://hf.co/datasets/TeraflopAI/Caselaw_Access_Project

🌍Data and models around the world
HPLT Monolingual, a dataset of 75 languages with over 40TB of data https://hf.co/datasets/HPLT/hplt_monolingual_v1_2
OpenLLM Turkish Benchmarks & Leaderboard https://hf.co/collections/malhajar/openllmturkishleadboard-datasets-65e5854490a87c0f2670ec18 and https://hf.co/spaces/malhajar/OpenLLMTurkishLeaderboard
Occiglot, a collaborative effort for European LLMs with an initial release of 7B models for French, German, Spanish, and Italian https://hf.co/collections/occiglot/occiglot-eu5-7b-v01-65dbed502a6348b052695e01
Guftagoo, a Hindi+Hinglish multi-turn conversational dataset https://hf.co/datasets/Tensoic/gooftagoo
AryaBhatta-Orca-Maths-Hindi dataset https://hf.co/datasets/GenVRadmin/Aryabhatta-Orca-Maths-Hindi

ajibawa-2023 YaTharThShaRma999 normanschizogh samusenps Euclid-Jie lunarflu Priceva Andron00e not-lain vapuck


Fast High-Resolution Image Synthesis with Latent Adversarial Diffusion
  Distillation (2403.12015)

Fast High-Resolution Image Synthesis with Latent Adversarial Diffusion Distillation

https://huggingface.co/papers/2403.12015

Diffusion models are the main driver of progress in image and video synthesis, but suffer from slow inference speed. Distillation methods, like the recently introduced adversarial diffusion distillation (ADD) aim to shift the model from many-shot to single-step inference, albeit at the cost of expensive and difficult optimization due to its reliance on a fixed pretrained DINOv2 discriminator. We introduce Latent Adversarial Diffusion Distillation (LADD), a novel distillation approach overcoming the limitations of ADD. In contrast to pixel-based ADD, LADD utilizes generative features from pretrained latent diffusion models. This approach simplifies training and enhances performance, enabling high-resolution multi-aspect ratio image synthesis. We apply LADD to Stable Diffusion 3 (8B) to obtain SD3-Turbo, a fast model that matches the performance of state-of-the-art text-to-image generators using only four unguided sampling steps. Moreover, we systematically investigate its scaling behavior and demonstrate LADD's effectiveness in various applications such as image editing and inpainting.

clem JayMokoena samusenps krzysztofpapciak


Today, I'm excited to launch two new models on the TTS Arena: MeloTTS and StyleTTS 2. Both are open sourced, permissively licensed, and highly efficient.

Curious to see how they compare with other leading models? Vote on the TTS Arena ⬇

https://huggingface.co/spaces/TTS-AGI/TTS-Arena

MeloTTS, released by MyShell AI, provides realistic and lifelike text to speech while remaining efficient and fast, even when running on CPU. It supports a variety of languages, including but not limited to English, French, Chinese, and Japanese.

StyleTTS 2 is another fully open sourced text to speech framework. It's permissively licensed, highly-efficient, and supports voice cloning and longform narration. It also provides natural and lifelike speech.

Both are available now to try on the TTS Arena - vote to find which one is better! The leaderboard will be revealed once we collect enough votes.

Kukedlc samusenps osanseviero theamazinceo DmitryRyumin AARon99 taufiqdp victor clem diwank jbilcke-hf Hyperionllama zironycho pierrci mirellyssl


so what was your favorite or most surprising announcement from GTC 2024?
https://www.youtube.com/live/Y2F8yisiS6E?si=shoZxQMHo_TTptCg


Uni-SMART: Universal Science Multimodal Analysis and Research
  Transformer (2403.10301)

Uni-SMART

Universal Science Multimodal Analysis and Research Transformer

https://huggingface.co/papers/2403.10301

In scientific research and its application, scientific literature analysis is crucial as it allows researchers to build on the work of others. However, the fast growth of scientific knowledge has led to a massive increase in scholarly articles, making in-depth literature analysis increasingly challenging and time-consuming. The emergence of Large Language Models (LLMs) has offered a new way to address this challenge. Known for their strong abilities in summarizing texts, LLMs are seen as a potential tool to improve the analysis of scientific literature. However, existing LLMs have their own limits. Scientific literature often includes a wide range of multimodal elements, such as molecular structure, tables, and charts, which are hard for text-focused LLMs to understand and analyze. This issue points to the urgent need for new solutions that can fully understand and analyze multimodal content in scientific literature. To answer this demand, we present Uni-SMART (Universal Science Multimodal Analysis and Research Transformer), an innovative model designed for in-depth understanding of multimodal scientific literature. Through rigorous quantitative evaluation across several domains, Uni-SMART demonstrates superior performance over leading text-focused LLMs. Furthermore, our exploration extends to practical applications, including patent infringement detection and nuanced analysis of charts. These applications not only highlight Uni-SMART's adaptability but also its potential to revolutionize how we interact with scientific literature.

samusenps muhtasham osanseviero hypnopump ajibawa-2023 clem AtAndDev

samusenps muhtasham hypnopump clem AtAndDev


Our 🐑 PECoRe 🐑 method to detect & attribute context usage in LM generations finally has an official Gradio demo! 🚀

https://huggingface.co/spaces/gsarti/pecore

Highlights:
🔍 Context attribution for several decoder-only and encoder-decoder models using convenient presets
🔍 Uses only LM internals to faithfully reflect context usage, no additional detector involved
🔍 Highly parametrizable, export Python & Shell code snippets to run on your machine using 🐛 Inseq CLI (https://github.com/inseq-team/inseq)

Want to use PECoRe for your LMs? Feedback and comments are welcome! 🤗

samusenps muhtasham clem louisbrulenaudet santiviquez


NECA: Neural Customizable Human Avatar (2403.10335)

🚀🕺🌟 New Research Alert - CVPR 2024! 🌟 💃🏻🚀
📄 Title: NECA: Neural Customizable Human Avatar 🌟🚀

📝 Description: The NECA paper presents a novel method for creating customizable human avatars from video, allowing detailed manipulation of pose, shadow, shape, lighting, and texture for realistic rendering and editing.

👥 Authors: Junjin Xiao, Qing Zhang, Zhan Xu, and Wei-Shi Zheng

📅 Conference: CVPR, Jun 17-21, 2024 | Seattle WA, USA 🇺🇸

🔗 Paper: https://huggingface.co/papers/2403.10335

📁 Repository: https://github.com/iSEE-Laboratory/NECA

📚 More Papers: more cutting-edge research presented at other conferences in the https://huggingface.co/spaces/DmitryRyumin/NewEraAI-Papers curated by @DmitryRyumin

🚀 Added to the Avatars Collection: https://huggingface.co/collections/DmitryRyumin/avatars-65df37cdf81fec13d4dbac36

🔍 Keywords: #NECA #AvatarCustomization #RealisticRendering #HumanRepresentation #CVPR2024 #DeepLearning #Animation #Innovation

DmitryRyumin osanseviero victor samusenps clem nvspavankalyanch mexicanamerican


🖴 The HPLT monolingual dataset has a new home!

After being in touch with HPLT folks, I've transfered the data to their org. That only makes sense. You can find it below.

https://huggingface.co/datasets/HPLT/hplt_monolingual_v1_2


How about engaging in a creative chat with your favorite video character?  💬

@chansung and I worked on a weekend project combining the benefits of Gemini 1.0 and powerful chat models like Zephyr to demo this.

We use Gemini 1.0 to produce the personality traits of any character found in an input video. We then prepare a system prompt with the discovered traits to start chatting with an LLM (Zephyr in this case).

Managing a video captioning model is a little out of our expertise, hence Gemini FTW here 😶🌫

👨💻 Code: https://github.com/deep-diver/Vid2Persona
🤗 Demo: https://hf.co/spaces/chansung/vid2persona

chansung osanseviero CharlieWells samusenps clem


🌟 New Research Alert - CVPR 2024! 🌟
📄 Title: "SVGDreamer: Text-Guided SVG Generation with Diffusion Model"
📝 TL;DR: Given a text prompt, SVGDreamer can generate editable and versatile high-fidelity vector graphics.
📝 Description: In this work, the author has introduced SVGDreamer, an innovative model for text-guided vector graphics synthesis. SVGDreamer incorporates two crucial technical designs: semantic-driven image vectorization (SIVE) and vectorized particle-based score distillation (VPSD), which empower our model to generate vector graphics with high editability, superior visual quality, and notable diversity.
👥 Authors: [Ximing Xing](https://ximinng.github.io/), Haitao Zhou, Chuang Wang, [Jing zhang](https://hellojing89.github.io/), [Dong Xu](https://www.cs.hku.hk/index.php/people/academic-staff/dongxu), and [Qian Yu](https://yuqian1023.github.io/)
📅 Conference: CVPR, Jun 17-21, 2024 | Seattle WA, USA 🇺🇸
🔍 Keywords: #SVGDreamer #Text-to-SVG #SVG #Diffusion #CVPR2024

Resources:
🔗 arXiv Paper: https://arxiv.org/abs/2312.16476
🌐 Github Page: https://ximinng.github.io/SVGDreamer-project/
📁 Code Repository: https://github.com/ximinng/SVGDreamer
📜 Blog: https://huggingface.co/blog/xingxm/svgdreamer

xingxm DmitryRyumin osanseviero samusenps clem


xAI releases the weights for Grok-1.  Apparently it's a 314B MoE with 25% of the weights active on a given token.

Blog: https://x.ai/blog/grok-os
Code: https://github.com/xai-org/grok
Model: https://huggingface.co/xai-org/grok-1
Weights: magnet:?xt=urn:btih:5f96d43576e3d386c9ba65b883210a393b68210e&tr=https%3A%2F%2Facademictorrents.com%2Fannounce.php&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A6969&tr=udp%3A%2F%2Ftracker.opentrackr.org%3A1337%2Fannounce

samusenps MrNobodyzac sarthakkapila jackyes AnaDP osanseviero clem LeroyDyer KaleDivergence

samusenps GO4code mathiasn1 jamone LexFree kramp clem KaleDivergence

samusenps malhajar Carlainsworth clem KaleDivergence


there were only 5 Major Releases last week !

😱 it's so over

samusenps osanseviero ABX-AI pcuenq thomwolf LucienL


What’s missing in today’s AI?
Adaptive AI Assistant(s) (AAA)

AAA - an ai assistant that gradually reflects an intelligently amplified version of you.

All I see in today's advance AI systems are tools that are heavily engineered to do all the work while we seat back and watch.

Shouldn't it be:

- tools that augment our information processing capabilities? (as once proposed by @karpathy)
- tools that can adapt to each person's needs?
- tools that reflect a more intelligent version of ourselves?

Because think about it, if we continue to build AI systems that we end up heavily relying on for even the most complex tasks, without actively involving us in the process, we risk losing out on the opportunity for personal growth and development.

Just saying.

Triple A is a more compelling path:)

And I'm writing a paper on this.

osanseviero clem dathinge Tom-Neverwinter


Animatable Gaussians: Learning Pose-dependent Gaussian Maps for
  High-fidelity Human Avatar Modeling (2311.16096)

🚀💃🏻🌟 New Research Alert - CVPR 2024! 🌟🕺 🚀
📄 Title: Animatable Gaussians: Learning Pose-dependent Gaussian Maps for High-fidelity Human Avatar Modeling 🌟🚀

📝 Description: Animatable Gaussians - a novel method for creating lifelike human avatars from RGB videos, utilizing 2D CNNs and 3D Gaussian splatting to capture pose-dependent garment details and dynamic appearances with high fidelity.

👥 Authors: Zhe Li, Zerong Zheng, Lizhen Wang, and Yebin Liu

📅 Conference: CVPR, Jun 17-21, 2024 | Seattle WA, USA 🇺🇸

🔗 Paper: https://huggingface.co/papers/2311.16096

🌐 Github Page: https://animatable-gaussians.github.io
📁 Repository: https://github.com/lizhe00/AnimatableGaussians

📺 Video: https://www.youtube.com/watch?v=kOmZxD0HxZI

📚 More Papers: more cutting-edge research presented at other conferences in the https://huggingface.co/spaces/DmitryRyumin/NewEraAI-Papers curated by @DmitryRyumin

🚀 Added to the Avatars Collection: https://huggingface.co/collections/DmitryRyumin/avatars-65df37cdf81fec13d4dbac36

🔍 Keywords: #AnimatableGaussians #HumanAvatars #3DGaussianSplatting #CVPR2024 #DeepLearning #Animation #Innovation

DmitryRyumin jawadur vladbogo ktibbs9413 samusenps osanseviero AlekseiPravdin merve thedigitized1 kingabzpro pcuenq clem 2dts


MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training (2403.09611)

MM1

Methods, Analysis & Insights from Multimodal LLM Pre-training

https://huggingface.co/papers/2403.09611

In this work, we discuss building performant Multimodal Large Language Models (MLLMs). In particular, we study the importance of various architecture components and data choices. Through careful and comprehensive ablations of the image encoder, the vision language connector, and various pre-training data choices, we identified several crucial design lessons. For example, we demonstrate that for large-scale multimodal pre-training using a careful mix of image-caption, interleaved image-text, and text-only data is crucial for achieving state-of-the-art (SOTA) few-shot results across multiple benchmarks, compared to other published pre-training results. Further, we show that the image encoder together with image resolution and the image token count has substantial impact, while the vision-language connector design is of comparatively negligible importance. By scaling up the presented recipe, we build MM1, a family of multimodal models up to 30B parameters, consisting of both dense models and mixture-of-experts (MoE) variants, that are SOTA in pre-training metrics and achieve competitive performance after supervised fine-tuning on a range of established multimodal benchmarks. Thanks to large-scale pre-training, MM1 enjoys appealing properties such as enhanced in-context learning, and multi-image reasoning, enabling few-shot chain-of-thought prompting.

zhoudasheng merve abidlabs samusenps AlekseiPravdin nezubn osanseviero pcuenq sukuya clem nebulae09


Platypose: Calibrated Zero-Shot Multi-Hypothesis 3D Human Motion
  Estimation (2403.06164)

Excited to share my latest preprint with @oliveiracaio and @fabee: "Platypose: Calibrated Zero-Shot Multi-Hypothesis 3D Human Motion Estimation"! 💃🕺

Platypose is our new approach to estimating 3D human motions from 2D observations. What makes it special? 🤩 Well, we're able to estimate multiple hypotheses for motion, which is pretty cool!

But here's the best part: our model is not only accurate but also well-calibrated. We've made sure that the predicted uncertainty matches the models confidence, so you can have a better understanding of Platypose's predictions.

Project Page: https://sinzlab.org/publications/2024-platypose.html
Paper: https://huggingface.co/papers/2403.06164
Model Weights: https://huggingface.co/sinzlab/platypose

The code is open-source, so please leave a star ⭐
https://github.com/sinzlab/platypose

dmnkbckr TheKonstantinWilleke oliveiracaio Himadri-ML samusenps osanseviero maxburg louisbrulenaudet ZennyKenny

oliveiracaio TheKonstantinWilleke Himadri-ML AlekseiPravdin maxburg clem Conradcon

ArneNix TheKonstantinWilleke oliveiracaio Himadri-ML osanseviero


KTO offers an easier way to preference train LLMs (only 👍👎 ratings are required). As part of #DataIsBetterTogether, I've written a tutorial on creating a preference dataset using Argilla and Spaces.

Using this approach, you can create a dataset that anyone with a Hugging Face account can contribute to 🤯

See an example of the kind of Space you can create following this tutorial here: https://huggingface.co/spaces/davanstrien/haiku-preferences

🆕 New tutorial covers:
💬 Generating responses with open models
👥 Collecting human feedback (do you like this model response? Yes/No)
🤖 Preparing a TRL-compatible dataset for training aligned models

Check it out here: https://github.com/huggingface/data-is-better-together/tree/main/kto-preference

merve transZ osanseviero AlekseiPravdin jaigouk JJhooww jayomb mmhamdy AtAndDev medmac01 pcuenq mathiasn1 michaelbenayoun dball ZennyKenny Cossale

AtAndDev dark-pen medmac01 michaelbenayoun clem


Unlocking the conversion of Web Screenshots into HTML Code with the
  WebSight Dataset (2403.09029)

With the new WebSight dataset, converting the screenshot of a web page to its corresponding HTML code is just one fine-tuning step away

We release a new version of our synthetic dataset:
-Real images within web pages 🖼
-Tailwind CSS 🎨
-2M examples 📈

Our initial release, v0.1, featured web designs in HTML + CSS, using simple colored rectangles as image placeholders.
It was a good start to help models grasp the basics of web page structure and coding associations.
Yet, it was missing the look of a real website.

Improving visual appeal, we've now embedded actual images in our web designs, ensuring they match the site's content for a more authentic look.

Switching to Tailwind CSS offers a more compact representation of the code.

We've also expanded our dataset to 2 million examples!

After fine-tuning our forthcoming foundation vision-language model on this dataset, we've observed some encouraging capabilities, such as converting sketches directly into functional HTML code.

We're excited to hear your thoughts and suggestions for future versions. What would you like to see next? Feel free to open a discussion on the hub!

Dataset: https://huggingface.co/datasets/HuggingFaceM4/WebSight
Technical report: https://huggingface.co/papers/2403.09029
Blog post: https://huggingface.co/blog/websight
Google Colab: https://colab.research.google.com/drive/1LdamGKR2oacrDk-kYwz_Wfc1-RBUdzcO?usp=sharing

Work done with @VictorSanh @Leyo

VictorSanh jeremy-london aidystark ajibawa-2023 seyf1elislam Tonic osanseviero clem Joseph717171

facenuma Sylvestre Tonic osanseviero clem Joseph717171

aidystark Tonic osanseviero WesPro Joseph717171


🎉Today, the 5000th Sentence Transformer model was uploaded to Hugging Face! Embedding models are extremely versatile, so it's no wonder that they're still being trained.

Here's a few resources to get you started with them:
- All Sentence Transformer models: https://huggingface.co/models?library=sentence-transformers&sort=trending
- Sentence Transformer documentation: https://sbert.net/
- Massive Text Embedding Benchmark (MTEB) Leaderboard: https://huggingface.co/spaces/mteb/leaderboard

The embedding space is extremely active right now, so if you're using an embedding model for your retrieval, semantic similarity, reranking, classification, clustering, etc., then be sure to keep an eye out on the trending Sentence Transformer models & new models on MTEB.

Also, I'm curious if you've ever used Sentence Transformers via a third party library, like a RAG framework or vector database. I'm quite interested in more integrations to bring everyone free, efficient & powerful embedding models!

slava-medvedev merve AtAndDev urchade osanseviero carlesoctav ajibawa-2023 pcuenq clem Waldoh hassamniaz7 alvarobartt aamirshakir nickprock

samusenps osanseviero pcuenq clem alvarobartt aamirshakir SeanLee97


When Greg Brockman demo-ed GPT4 by hand-sketching a joke website on a piece of paper and asking the system to convert that into an HTML webpage, it blew my mind.

Can you build your own Screenshot-to-HTML system with much fewer resources?

With this new resource, most likely yes! Current vision-language models can learn this task with the right data (and the right tricks).

We have iterated on WebSight-v0.1 and are releasing its v0.2.
WebSight is an open dataset of synthetically generated webpages with their corresponding rendered screenshot.

A few noticeable improvements:
- 💨From traditional CSS to Tailwind CSS. Tailwind is CSS directly embedded in the HTML attribute class and is much more compact
- 🚛2M pairs of synthetic HTML webpages with their associated rendered screenshot, along with the prompt generated by an LLM to create that webpage
- 🖼Much more visually appealing pages with the integration of real images

👀Blog: https://huggingface.co/blog/websight
💽Dataset: https://huggingface.co/datasets/HuggingFaceM4/WebSight
📜Technical report: https://huggingface.co/papers/2403.09029
🎮Want to create your own synthetic data pipelines? A starting point: https://colab.research.google.com/drive/1LdamGKR2oacrDk-kYwz_Wfc1-RBUdzcO?usp=sharing

Built with @HugoLaurencon & @Leyo

HugoLaurencon osanseviero Sylvestre merve clem vinwizard


tinyBenchmarks: evaluating LLMs with fewer examples (2402.14992)

Design2Code: How Far Are We From Automating Front-End Engineering? (2403.03163)

Diaries of Open Source. Part 5!

🤯Contextual KTO Mistral PairRM: this model combines iterative KTO, SnorkelAI DPO dataset, Allenai PairRM for ranking, Mistral for the base model, and is a very strong model with Claude 3 quality on AlpacaEval 2.0
Final model: https://hf.co/ContextualAI/Contextual_KTO_Mistral_PairRM
Dataset: https://hf.co/datasets/snorkelai/Snorkel-Mistral-PairRM-DPO-Dataset
Leaderboard: https://tatsu-lab.github.io/alpaca_eval/
Base model: https://hf.co/mistralai/Mistral-7B-Instruct-v0.2

🤏 tinyBenchmarks: Quick and cheap LLM evaluation!
Code: https://github.com/felipemaiapolo/tinyBenchmarks
Paper: https://hf.co/papers/2402.14992
Data: https://hf.co/datasets/tinyBenchmarks/tinyMMLU

🎨Transformers.js 2.16 includes StableLM,  speaker verification and diarization, and better chat templating. Try some fun demos!
- https://hf.co/spaces/Xenova/video-object-detection
- https://hf.co/spaces/Xenova/cross-encoder-web
- https://hf.co/spaces/Xenova/the-tokenizer-playground

🏴☠ Abascus Liberated-Qwen1.5-72B, a Qwen 72B-based model that strongly follows system prompts
Model: https://hf.co/abacusai/Liberated-Qwen1.5-72B

👀Design2Code: benchmark of webpage screenshots  to code
Data: https://hf.co/datasets/SALT-NLP/Design2Code
Project https://salt-nlp.github.io/Design2Code/
Paper https://hf.co/papers/2403.03163

🌎Data and models around the world
- One of the biggest Italian datasets https://hf.co/datasets/manalog/UsenetArchiveIT
- IndicLLMSuite: argest Pre-training and Instruction Fine-tuning dataset collection across 22 Indic languages https://hf.co/collections/ai4bharat/indicllmsuite-65ee7d225c337fcfa0991707
- Hebrew-Gemma-11B, the best base Hebrew model https://huggingface.co/yam-peleg/Hebrew-Gemma-11B
- Komodo-7B, a family of multiple Indonesian languages LLMs https://hf.co/Yellow-AI-NLP/komodo-7b-base

You can find the previous part at https://huggingface.co/posts/osanseviero/127895284909100

DmitryRyumin alielfilali01 mikonvergence merve samusenps ajibawa-2023 pcuenq SSamDav victor Stopwolf


Komodo: A Linguistic Expedition into Indonesia's Regional Languages (2403.09362)

Komodo-7B is here !! Today we are releasing the base version of Komodo-7B along with the  technical report.

Komodo-7B is a family of LLMs that consist of Komodo-7B-Base and Komodo-7B-Instruct.

Komodo-7B performers really good in multiple Indonesian languages including Indonesian, Acehnese, Balinese, Banjarese, Buginese, Dayak Ngaju, Javanese, Lampungnese, Madurese, Minangkabau, Sundanese, and Toba Batak.

Our model outperforms various existing large language models including some multilingual models.

Technical Report: https://arxiv.org/abs/2403.09362

Base Model HuggingFace:  https://huggingface.co/Yellow-AI-NLP/komodo-7b-base

Kudos to the team @louisowen6 , @akanyaani  & @biddwan https://huggingface.co/papers/2403.09362

louisowen6 Kernel biddwan samusenps osanseviero kramp hoangdev1104 merve victor taufiqdp loubnabnl


We released 🧨 Diffusers 0.27.0, and it's a versatile release 💫

Among other things, we shipped:

* Stable Cascade
* Playground v2.5 and EDM-style training
* EDM-formulated schedulers
* Trajectory Consistency Distillation for accelerated sampling
* A new guide on merging LoRAs
* A new image editing pipeline -- LEDITS++

Check out the release notes to catch everything that went into the release
https://github.com/huggingface/diffusers/releases/tag/v0.27.0

Thanks to everyone that contributed to the release 🤗

DmitryRyumin osanseviero kramp mvaloatto AdinaY fffiloni merve

samusenps osanseviero louisbrulenaudet AndroidAttack


Follow-Your-Click: Open-domain Regional Image Animation via Short
  Prompts (2403.08268)

"Follow-Your-Click: Open-domain Regional Image Animation via Short Prompts" is a new framework designed to animate specific regions within an image through user inputs.

Key points:
* Enables precise animation of selected image regions with just a user click and a concise motion description.
* Achieves promising results for generating localized animations.

Paper: https://huggingface.co/papers/2403.08268

Congrats to the authors for their work!

merterbak samusenps xsmq DmitryRyumin merve victor


Huggingface is carrying the AI open source ecosystem https://huyenchip.com/2024/03/14/ai-oss.html

osanseviero samusenps nikilpatel94 julien-c lewtun lvwerra merve loubnabnl AdinaY joaogante hanguyen146 RaushanTurganbay thomwolf andrewrreed xianbao nezubn ajibawa-2023 beberik pcuenq LucienL michaelbenayoun rafaelpierrehf Srulikbd

osanseviero Zyn123 lewtun merve AdinaY joaogante RaushanTurganbay thomwolf andrewrreed sayakpaul xianbao evdcush pcuenq LucienL michaelbenayoun rafaelpierrehf jeffboudier Bkarine

osanseviero kramp lewtun merve loubnabnl AdinaY RaushanTurganbay thomwolf andrewrreed LucienL michaelbenayoun rafaelpierrehf jeffboudier


Simple and Scalable Strategies to Continually Pre-train Large Language
  Models (2403.08763)

Simple and Scalable Strategies to Continually Pre-train Large Language Models

https://huggingface.co/papers/2403.08763

Large language models (LLMs) are routinely pre-trained on billions of tokens, only to start the process over again once new data becomes available. A much more efficient solution is to continually pre-train these models, saving significant compute compared to re-training. However, the distribution shift induced by new data typically results in degraded performance on previous data or poor adaptation to the new data. In this work, we show that a simple and scalable combination of learning rate (LR) re-warming, LR re-decaying, and replay of previous data is sufficient to match the performance of fully re-training from scratch on all available data, as measured by final loss and language model (LM) evaluation benchmarks. Specifically, we show this for a weak but realistic distribution shift between two commonly used LLM pre-training datasets (EnglishrightarrowEnglish) and a stronger distribution shift (EnglishrightarrowGerman) at the 405M parameter model scale with large dataset sizes (hundreds of billions of tokens). Selecting the weak but realistic shift for larger-scale experiments, we also find that our continual learning strategies match the re-training baseline for a 10B parameter LLM. Our results demonstrate that LLMs can be successfully updated via simple and scalable continual learning strategies, matching the re-training baseline using only a fraction of the compute. Finally, inspired by previous work, we propose alternatives to the cosine learning rate schedule that help circumvent forgetting induced by LR re-warming and that are not bound to a fixed token budget.

samusenps boapps osanseviero Kernel neerajjulka RachidAR dolemole anthonyivn rnella01 PcHome20242 TangoHumanity louisbrulenaudet


I've released several new Hugging Face Spaces.

My primary objective is to create consistent character facial animation using image-to-image techniques:

https://huggingface.co/spaces/Akjava/CreateConsistentCharacterFacialAnimationWithImg2Img

A short-term goal is create simple talk-head animation.

WebP-3-Frame-Talking-Animation
https://huggingface.co/spaces/Akjava/AIDiagramChatWithVoice-FaceCharacter

[Space]

- GPU tools
Flux1-schnell img2img
https://huggingface.co/spaces/Akjava/flux1-schnell-img2img

Flux1-schnell Inpaint with mask-file
https://huggingface.co/spaces/Akjava/flux1-schnell-img2img

 - Tiny CPU tools
WebP-3F-TH - create webp animation from 3 images
OpenCV-Inapint - classic inpaint
Whitebalance - simple white balance
Paste Image - just paste image with mask
WebP Resize Convert - resize and convert webp-animation


We are happy to introduce MedIT SUN 1B, a downscaled version of the MedIT SUN 2.5B Llama 3.2 variant.

Give it a try!
https://huggingface.co/meditsolutions/Llama-3.2-SUN-1B-chat


Do you guys want to see my training code for https://huggingface.co/nroggendorff/smallama ?


Imagine being able to talk directly to your API connection. "I have a field in the CRM named Customer_ID that needs to map to a field in the ERP named ERP_Customer_ID." Imagine being able to give your API connections both a brain and swarm of agents as a body to execute any task or function. This isn't science fiction, this is the revolutionary power of Liquid API. A product 10 years in the making!

https://youtu.be/cHI_k1Dkdr4


New Mann-E model just released:
https://huggingface.co/mann-e/mann-e_flux

I will be glad if you test it!


hi everyone,
i have trained a Qwen 14b model on a smaller dataset, but its now very tricky because i have got nowhere to use it via inference (the paid for inference on hf costs quite a lot), does anyone know of anywhere where i can deploy my model and use it via api for a reasonable cost, or ideally none. thanks

John6666 robertomachorro hakutaku victor


LLaMA-Berry: Pairwise Optimization for O1-like Olympiad-Level
  Mathematical Reasoning (2410.02884)

Accessing GPT-4 level Mathematical Olympiad Solutions via Monte Carlo
  Tree Self-refine with LLaMa-3 8B (2406.07394)

LLaMA-O1: Open Large Reasoning Model Frameworks For Training, Inference and Evaluation With PyTorch and HuggingFace
Large Reasoning Models powered by Monte Carlo Tree Search (MCTS), Self-Play Reinforcement Learning, PPO, AlphaGo Zero's dua policy paradigm and Large Language Models!
https://github.com/SimpleBerry/LLaMA-O1/

What will happen when you compound MCTS ❤ LLM ❤ Self-Play ❤RLHF?
Just a little bite of strawberry!🍓

Past related works:
https://huggingface.co/papers/2410.02884
https://huggingface.co/papers/2406.07394

iwancobain John6666 zsqzz jwu323 ALYTV Azzedde Svngoku timmylai nbroad csabakecskemeti Syzygianinfern0 sekkit flozi00 qftop victor ajibawa-2023 seyf1elislam KvrParaskevi dingo-actual createtheimaginable ai-everyday


Hi there HuggingFacers!🤗

Are you working with Streamlit on Spaces and struggling with authentication and user management?🧐

Well, you can check out my last community article (https://huggingface.co/blog/as-cle-bert/streamlit-supabase-auth-ui) on a new python package I've been working on, that connects Supabase to Streamlit UI, in order to create a seamless authentication for your seamless Streamlit apps!🚀

You can find a demo of it on Spaces: https://huggingface.co/spaces/as-cle-bert/streamlit-supabase-auth-ui

Have fun!🍕


OmniGen 1-Click Automatic Installers for Windows, RunPod and Massed Compute

OmniGen is a unified image generation model that can generate a wide range of images from multi-modal prompts. It is designed to be simple, flexible, and easy to use

Installers are here : https://www.patreon.com/posts/omnigen-1-click-115233922

Look attached images to understand what capabilities it has. It is simply amazing so many features.

What is OmniGen : https://github.com/VectorSpaceLab/OmniGen

Windows Requirements
Python 3.10.11, CUDA 12.4, Git, FFMPEG, cuDNN 9.x, C++ Tools

A tutorial that shows how to install all above : https://youtu.be/DrhUHnYfwC0

How To Install & Use
After installing requirements by following above tutorial, double-click Windows_Install.bat and install
After that use Windows_Start.bat to start the app

When offload_model is enabled (checked) on the Gradio interface, it uses 5.4 GB VRAM, 2x slower

When offload_model is not used (not checked) it uses 12.2 GB VRAM

When separate_cfg_infer is not checked, and offload_model is not checked, it uses 18.7 GB VRAM

To install on RunPod and Massed Compute please follow Massed_Compute_Instructions_READ.txt and Runpod_Instructions_READ.txt

Look at the examples on the Gradio interface closely to understand how to use

MonsterMMORPG CYGDEN salemseidmohamed Sethblocks clem


Smol models ftw! AMD released AMD OLMo 1B - beats OpenELM, tiny llama on MT Bench, Alpaca Eval - Apache 2.0 licensed 🔥

> Trained with 1.3 trillion (dolma 1.7) tokens on 16 nodes, each with 4 MI250 GPUs

> Three checkpoints:

- AMD OLMo 1B: Pre-trained model
- AMD OLMo 1B SFT: Supervised fine-tuned on Tulu V2, OpenHermes-2.5, WebInstructSub, and Code-Feedback datasets
- AMD OLMo 1B SFT DPO: Aligned with human preferences using Direct Preference Optimization (DPO) on UltraFeedback dataset

Key Insights:
> Pre-trained with less than half the tokens of OLMo-1B
> Post-training steps include two-phase SFT and DPO alignment
> Data for SFT:
- Phase 1: Tulu V2
- Phase 2: OpenHermes-2.5, WebInstructSub, and Code-Feedback

> Model checkpoints on the Hub & Integrated with Transformers ⚡

Congratulations & kudos to AMD on a brilliant smol model release! 🤗

https://huggingface.co/collections/amd/amd-olmo-6723e7d04a49116d8ec95070

AtAndDev John6666 CYGDEN louisbrulenaudet kimleang123 Dolfini Joseph717171 not-lain KvrParaskevi

AtAndDev CYGDEN Joseph717171 rizky-gumelar not-lain


VLOGGER: Multimodal Diffusion for Embodied Avatar Synthesis (2403.08764)

🚀🎭🌟 New Research Alert! 🌟🎭 🚀
📄 Title: VLOGGER: Multimodal Diffusion for Embodied Avatar Synthesis 🌟🚀

📝 Description: VLOGGER is a method for text- and audio-driven generation of talking human video from a single input image of a person, building on the success of recent generative diffusion models.

👥 Authors: @enriccorona, @Andreiz, @kolotouros, @thiemoall, and et al.

🔗 Paper: https://huggingface.co/papers/2403.08764

🌐 Github Page: https://enriccorona.github.io/vlogger/

📚 More Papers: more cutting-edge research presented at other conferences in the https://huggingface.co/spaces/DmitryRyumin/NewEraAI-Papers curated by @DmitryRyumin

🚀 Added to the Avatars Collection: https://huggingface.co/collections/DmitryRyumin/avatars-65df37cdf81fec13d4dbac36

🔍 Keywords: #VLOGGER #EmbodiedAvatarSynthesis #MultimodalDiffusion #GenerativeDiffusionModels #DeepLearning #Animation #Innovation

DmitryRyumin merve fffiloni whappolow lunarflu osanseviero Nicolas55 AdinaY mychen76 EddyGiusepe Ivan18 beepboopbeep zarazi VanshRana12

merve lunarflu samusenps God-of-ai clefourrier


A Modular End-to-End Multimodal Learning Method for Structured and
  Unstructured Data (2403.04866)

🚀 Hello HF Posts World!

I'm excited to share in my first HF post that we at Neuraptic AI have released MAGNUM, the first open-source AI model designed to natively support any structured and unstructured data modality.

MAGNUM can learn a holistic representation of your business logic from any source of digital information—be it images, documents, emails, databases, audio, signals, and more. This rich context empowers it to deliver significantly more accurate answers.

If you want to know more about it, feel free to ask or read the paper here 🤗
https://huggingface.co/papers/2403.04866

Have a nice week!

samusenps osanseviero Madhanraj merve lunarflu AdinaY danielus clem DataMorpheus


Hello HF!

Recently, I authored an article on setting up multiple GPUs in a desktop PC. I discovered a bug during the process and have documented my observations in this article:

https://morgangiraud.medium.com/multi-gpu-nvidia-p2p-capabilities-and-debugging-tips-fb7597b4e2b5

It's not directly related to ML, but maybe this will save someone some time! :)


Prompt Engineering: Playing A Game of Chance With LLMs.

It's obvious these days, that trying to get the best out of  LLMs  resembles playing a game of chance, where the choice of prompts acts as your moves in shaping the model's responses, as you recursively seek the best.

Each prompt you craft carries the potential to lead the LLM down different paths, influencing the quality and relevance of its outputs. By experimenting with various prompts and observing how the model responds, you can uncover new insights into the inner workings of these complex systems and push the boundaries of what they can achieve.

Not long ago, this craftsmanship has been termed "Prompt Engineering", it's a job now. To better understand the "Engineering" of it, let's go through the paper by Google's Brain Team that shed light on it: Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.

The paper starts off with a clear definition of Chain-of-Thought — a coherent series of intermediate  natural language reasoning steps that lead to the final answer for a problem.

The researchers explored how generating a series of intermediate reasoning steps significantly improves the ability of large language models to perform complex reasoning. They found that such reasoning abilities "emerge naturally" in sufficiently large language models via a simple method called chain of thought prompting, where a few chain of thought demonstrations are provided as exemplars in prompting.

Experiments on three large language models showed that chain of thought prompting improves performance on a range of arithmetic, commonsense, and symbolic reasoning tasks. For instance, prompting a 540B-parameter language model with just eight chain of thought exemplars achieves state of the art accuracy on the GSM8K benchmark of math word problems, surpassing even finetuned GPT-3 with a verifier.

ReadMore: https://x.com/jaykef_/status/1767173517345485232?s=46&t=V2mWOpm9AdMX0spmmr0yNQ


New moondream update out with significantly improved OCR performance (among other benchmarks)!
https://huggingface.co/vikhyatk/moondream2

merve fffiloni Csplk Sylvestre Tom-Neverwinter not-lain cnmoro avinash02 damerajee radames


Synth$^2$: Boosting Visual-Language Models with Synthetic Captions and
  Image Embeddings (2403.07750)

Synth^2 is a new approach that leverages large language models and text-to-image generators to create synthetic image-caption data for boosting visual-language model performance.

Key Points:
* Overcomes data limitations by generating high-quality synthetic image-caption pairs, reducing reliance on costly human annotations.
* Achieves competitive results on image captioning tasks using 40x less paired data than state-of-the-art methods.

Paper: https://huggingface.co/papers/2403.07750

Congrats to the authors for their work!


Wonderful open source Italian dataset from @manalog and @ruggsea:

https://huggingface.co/datasets/manalog/UsenetArchiveIT

The dataset contributes to the  https://huggingface.co/mii-community project, aimed at advancing the creation of Italian open-source Language Models (LLMs).🇮🇹 🤖  About 10-20 billion token, probably the best conversational open source dataset in the Italian language. 🇮🇹🇮🇹🇮🇹🇮🇹🇮🇹🇮🇹🇮🇹

victor giux78 ruggsea samusenps mrinaldi osanseviero gsarti jackyes AdinaY


A combined effort from the IBM + Pytorch teams achieved an incredible training performance with ZeRO/FSDP on par with 3D parallelism on H100s, while having just 800Gbps inter-node connection.

This is because they got an almost full overlap between comms and compute and have introduced a novel selective activation recomputation method which recalculates only large but inexpensive activations.

Check out their post here: https://pytorch.org/blog/maximizing-training/

sugatoray victor samusenps osanseviero stas hunoutl muhtasham


GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection (2403.03507)

Interesting paper: 𝐆𝐚𝐋𝐨𝐫𝐞: 𝐭𝐫𝐚𝐢𝐧 𝟕𝐁 𝐦𝐨𝐝𝐞𝐥𝐬 𝐨𝐧 𝐜𝐨𝐧𝐬𝐮𝐦𝐞𝐫-𝐠𝐫𝐚𝐝𝐞 𝐆𝐏𝐔𝐬 💪
It's now possible to 𝙛𝙪𝙡𝙡𝙮 𝙥𝙧𝙚-𝙩𝙧𝙖𝙞𝙣 a 7B model on a consumer-grade GPU of 24Gb RAM, without any performance loss!

The memory usage of training models has always been an acute issue. For instance full pre-training of a 7B model used to eat ~50Gb of RAM!

The common workarounds to reduce memory load are:
- separate models on multiple GPUs ("sharding")
- quantize models: encode weights on fewer bits

Another technique is to 𝙥𝙧𝙤𝙟𝙚𝙘𝙩 𝙩𝙝𝙚 𝙬𝙚𝙞𝙜𝙝𝙩 𝙢𝙖𝙩𝙧𝙞𝙭 𝙩𝙤 𝙡𝙤𝙬𝙚𝙧-𝙧𝙖𝙣𝙠 𝙨𝙥𝙖𝙘𝙚𝙨, (since sometimes the weights do not really vary on all dimensions): this can save a lot of space!
This low-rank projection can be done on adapters to preserve the original weights (go check out LoRA), but it still generally hurts the performance too much for pre-training.

➡ Enter the authors of 𝘎𝘢𝘓𝘰𝘳𝘦: 𝘔𝘦𝘮𝘰𝘳𝘺-𝘌𝘧𝘧𝘪𝘤𝘪𝘦𝘯𝘵 𝘓𝘓𝘔 𝘛𝘳𝘢𝘪𝘯𝘪𝘯𝘨 𝘣𝘺 𝘎𝘳𝘢𝘥𝘪𝘦𝘯𝘵 𝘓𝘰𝘸-𝘙𝘢𝘯𝘬 𝘗𝘳𝘰𝘫𝘦𝘤𝘵𝘪𝘰𝘯. They gather (and prove) interesting insights:
⛔ The weight matrix does not reliably converge to lower ranks during training.
✅ But the gradient matrix does!

Based on these insights, 𝘁𝗵𝗲𝘆 𝗯𝘂𝗶𝗹𝗱 𝗚𝗮𝗟𝗼𝗿𝗲, that projects the gradient to lower ranks.
🗺 𝗚𝗿𝗲𝗮𝘁 𝗶𝗱𝗲𝗮: to leave the optimization free to explore more space, they periodically re-build the low-rank projection throughout the training (a nice illustration is in the paper).

🤝 This method can even be combined with previous ones such as 8-bit Adam (quantizing the optimizer states to 8-bit).

➡ 𝐑𝐞𝐬𝐮𝐥𝐭𝐬:
📉 Of course, huge reduction in memory footprint allowing the training on consumer-grade GPU (cf figure).
💪 No reduction in performance: this scales well up to 7B parameters (and was independently confirmed since) ⇒ this is essential, it confirms that the method is viable!

Read the full paper here: https://huggingface.co/papers/2403.03507


Self-Play Fine-Tuning Converts Weak Language Models to Strong Language
  Models (2401.01335)

🔥 Community and Data Quality Are More For Alignment

A recipe to replicate SPIN (Self-Play Fine Tuning) with 30x less data:

🗣 50K samples vs 1.8K prompts curated by the 350+ amazing DIBT contributors.
⚗ Distillation of Mistral Large instead of OpenAI
🙌 Open data & code with ⚗distilabel

SPIN Paper:
https://huggingface.co/papers/2401.01335

SPIN DIBT Collection with datasets and models:
https://huggingface.co/collections/argilla/dibt-prompt-collective-spin-65ef59062518776024395fc3

Repo:
https://github.com/argilla-io/distilabel-spin-dibt

Joint work with the amazing DIBT community 👇
@aashish1904, @flozi00, @sayhan, @munish0838, @0-hero, @dvilasuero, @eren23, @davanstrien, @ahnz, @BlackKakapo, @kitano-o, @mmhamdy, @sdiazlor, @Stopwolf, @gabrielmbmb, @tculler91, @plaguss, @ignacioct, @Hugi-R, @davidberenstein1957, @Korla, @alvarobartt, @Hugs4Llamas, @Sumandora, @nataliaElv, @jfcalvo, @Averill, @steventrouble, @vasilis, @aeros93, @kayyshf, @thomasgauthier, @jeromebas, @Ameeeee, @ayoubelmhamdi, @TuringsSolutions, @efels, @Haleyok, @abrazador, @emessy, @Nindaleth, @burtenshaw, @vicgalle, @CortexPE, @casey-martin, @Leire-aguirre-eguiluz, @mrfakename, @Portias600kNeurons, @nathaliepett, @Filippo

davanstrien nataliaElv ignacioct victor davidberenstein1957 munish0838 0-hero dvilasuero julien-c pacoid plaguss osanseviero mmhamdy alvarobartt gabrielmbmb mihaylovnikitos TuringsSolutions clem hughte BlackKakapo

davanstrien nataliaElv flozi00 davidberenstein1957 julien-c pacoid plaguss giux78 osanseviero mmhamdy gabrielmbmb BlackKakapo abdullahalzubaer clem

davanstrien davidberenstein1957 julien-c pacoid osanseviero mmhamdy gabrielmbmb BlackKakapo clem thomasgauthier

davidberenstein1957 julien-c pacoid osanseviero mmhamdy BlackKakapo clem


MoAI: Mixture of All Intelligence for Large Language and Vision Models (2403.07508)

MoAI

Mixture of All Intelligence for Large Language and Vision Models

https://huggingface.co/papers/2403.07508

The rise of large language models (LLMs) and instruction tuning has led to the current trend of instruction-tuned large language and vision models (LLVMs). This trend involves either meticulously curating numerous instruction tuning datasets tailored to specific objectives or enlarging LLVMs to manage vast amounts of vision language (VL) data. However, current LLVMs have disregarded the detailed and comprehensive real-world scene understanding available from specialized computer vision (CV) models in visual perception tasks such as segmentation, detection, scene graph generation (SGG), and optical character recognition (OCR). Instead, the existing LLVMs rely mainly on the large capacity and emergent capabilities of their LLM backbones. Therefore, we present a new LLVM, Mixture of All Intelligence (MoAI), which leverages auxiliary visual information obtained from the outputs of external segmentation, detection, SGG, and OCR models. MoAI operates through two newly introduced modules: MoAI-Compressor and MoAI-Mixer. After verbalizing the outputs of the external CV models, the MoAI-Compressor aligns and condenses them to efficiently use relevant auxiliary visual information for VL tasks. MoAI-Mixer then blends three types of intelligence (1) visual features, (2) auxiliary features from the external CV models, and (3) language features by utilizing the concept of Mixture of Experts. Through this integration, MoAI significantly outperforms both open-source and closed-source LLVMs in numerous zero-shot VL tasks, particularly those related to real-world scene understanding such as object existence, positions, relations, and OCR without enlarging the model size or curating extra visual instruction tuning datasets.

sugatoray jucamohedano maywell damerajee


Working on a new open-source project to showcase https://huggingface.co/levihsu/OOTDiffusion  🤗

But it's not a Space 🫢  Can you guess what it does?  👀


Can we improve the quality of open LLMs for more languages?

Step 1: Evaluate current SOTA.

The Data Is Better Together community has rated more than 10K prompts for quality. We now want to translate a subset of these to help address the language gap in model evals.

The plan is roughly this:

- We started with https://huggingface.co/datasets/DIBT/10k_prompts_ranked and took a subset of 500 high-quality prompts
- We're asking the community to translate these prompts into different languages
- We'll evaluate the extent to which we can use AlpacaEval and similar approaches to rate the outputs of models across these different languages
- If it works well, we can more easily evaluate open LLMs across different languages by using a judge LLM to rate the quality of outputs from different models.

You can find more details in our new GitHub repo: https://github.com/huggingface/data-is-better-together (don't forget to give it a ⭐!)

dvilasuero clefourrier samusenps sarahooker Joseph717171


I love vision language models 💗
My favorite is KOSMOS-2, because it's a grounded model (it doesn't hallucinate).
In this demo you can,
- ask a question about the image,
- do detailed/brief captioning,
- localize the objects! 🤯
It's just amazing for VLM to return bounding boxes 🤩
Try it here https://huggingface.co/spaces/merve/kosmos2

meryem1232 samusenps osanseviero plmsmile victor GO4code theainerd ShivjiAgnihotri


AV2Wav: Diffusion-Based Re-synthesis from Continuous Self-supervised
  Features for Audio-Visual Speech Enhancement (2309.08030)

🚀🗣🌟 New Research Alert - ICASSP 2024! 🌟🗣🚀
📄 Title: AV2Wav: Diffusion-Based Re-synthesis from Continuous Self-supervised Features for Audio-Visual Speech Enhancement 🌟🚀

📝 Description: Diffused Resynthesis and HuBERT Speech Quality Enhancement.

👥 Authors: Ju-Chieh Chou, Chung-Ming Chien, Karen Livescu

📅 Conference: ICASSP, 14-19 April 2024 | Seoul, Korea 🇰🇷

🔗 Paper: https://huggingface.co/papers/2309.08030

🌐 Web Page: https://home.ttic.edu/~jcchou/demo/avse/avse_demo.html

📚 More Papers: more cutting-edge research presented at other conferences in the
DmitryRyumin/NewEraAI-Papers curated by @DmitryRyumin

🚀 Added to the Speech Enhancement Collection: https://huggingface.co/collections/DmitryRyumin/speech-enhancement-65de31e1b6d9a040c151702e

🔍 Keywords: #AV2Wav #SpeechEnhancement #SpeechProcessing #AudioVisual #Diffusion #ICASSP2024 #Innovation

DmitryRyumin samusenps taufiqdp enzostvs osanseviero merve victor Shir02021


Yi: Open Foundation Models by 01.AI (2403.04652)

DeepSeek-VL: Towards Real-World Vision-Language Understanding (2403.05525)

OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist
  Autonomous Agents for Desktop and Web (2402.17553)

Diaries of Open Source. Part 4!

🌏Cohere and Cohere4AI release Command-R, a 35B model that is multilingual, RAG-optimized, and can manage tools!
Model: https://huggingface.co/CohereForAI/c4ai-command-r-v01
Blog post: https://txt.cohere.com/command-r/

🧑🍳StarChat2: A powerful code model that is conversational
Try it out: https://huggingface.co/spaces/HuggingFaceH4/starchat2-playground
Repos: https://huggingface.co/collections/HuggingFaceH4/starchat2-15b-65f068417b330fafad751fce
Training code: https://github.com/huggingface/alignment-handbook/tree/main/recipes/starchat2-15b

🐲Yi-9B: trained on 3 trillion tokens, this english-chinese LLM is quite good and with a very nice detailed report!
Model: https://huggingface.co/01-ai/Yi-9B
Paper: https://huggingface.co/papers/2403.04652

🐋DeepSeek-VL, 1.3B and 7B VLMs
Paper: https://hf.co/papers/2403.05525
Large model: https://huggingface.co/deepseek-ai/deepseek-vl-7b-chat

✍Writer releases OmniACT: a dataset for multimodal agents for desktop and web.
Dataset: https://huggingface.co/datasets/Writer/omniact
Paper: https://huggingface.co/papers/2402.17553

🍎Apple releases MobileCLIP: fast image-text models! https://github.com/apple/ml-mobileclip

🦙💪LlamaGym - fine-tune LLM agents with RL in just a few lines of code! https://github.com/KhoomeiK/LlamaGym

 🖼New multimodal leaderboard ConTextual https://huggingface.co/blog/leaderboard-contextual

🎁 Design2Code: benchmark for multimodal LLMs for automating front-end development.
Dataset https://huggingface.co/datasets/SALT-NLP/Design2Code
Paper https://huggingface.co/papers/2403.03163
Project https://salt-nlp.github.io/Design2Code/

You can find the previous part at https://huggingface.co/posts/osanseviero/633758457910104

samusenps victor kramp DmitryRyumin adamelliotfields merve mohammedbriman giux78 MichaelFried theainerd avinash02 VanshRana12


You gotta love what Aapple’s mlx team cooked:

- A unified memory model that literally does compute-magic: parallel operations with automatic dependency insertions.
- Supports off-the-shelf use of all the fun stuff in composable func transformations (differentiation,  vectorization, computation graph optimization).
- Houses simplified forms of all the APIs we love and in the languages we adore (python, C++, C) sorry Swift :)
- mlx.nn is a stallion 🔥 simple to use.
- Open-source friendly (who would have thought lol).
- Dynamic graph construction👍🏼
- Supports both CPU and GPU🤖
- Beginner Friendly 👌🏼
- Great examples (clean code💯)
- Good documentation

Well done Awni Hannun et al 👏🏼

Could this be The Transformer of ml frameworks? Well at least for us mac users 😂

Repo: https://github.com/ml-explore/mlx
Examples: https://github.com/ml-explore/mlx-examples
Documentation: https://ml-explore.github.io/mlx/build/html/python/nn.html


🚀 Excited to unveil the Augmented ARC-Challenge Dataset with Chain-of-Thought Reasoning! 🧠✨

📚 Created by enhancing the ARC dataset with AI-generated reasoning from Google's Gemini Pro, this resource aims to improve question answering models' ability to tackle complex science queries.

🔍 Features:
- 1068 training examples
- Detailed reasoning steps for nuanced understanding
- Questions spanning physics, chemistry, biology, & more!

🌟 Ideal for benchmarking QA models, enhancing model interpretability, and studying in-context examples.

🔗 Dive in and help your models learn the art of reasoning!

🔎 Explore more: https://huggingface.co/datasets/Locutusque/arc-cot


Stealing Part of a Production Language Model (2403.06634)

Stealing Part of a Production Language Model

https://huggingface.co/papers/2403.06634

We introduce the first model-stealing attack that extracts precise, nontrivial information from black-box production language models like OpenAI's ChatGPT or Google's PaLM-2. Specifically, our attack recovers the embedding projection layer (up to symmetries) of a transformer model, given typical API access. For under \20 USD, our attack extracts the entire projection matrix of OpenAI's Ada and Babbage language models. We thereby confirm, for the first time, that these black-box models have a hidden dimension of 1024 and 2048, respectively. We also recover the exact hidden dimension size of the gpt-3.5-turbo model, and estimate it would cost under 2,000 in queries to recover the entire projection matrix. We conclude with potential defenses and mitigations, and discuss the implications of possible future work that could extend our attack.

ajibawa-2023 taufiqdp arnavgrg adamelliotfields evdcush abdullahalzubaer bootsbootsboots philipp-zettl


Hello World! This post is written by the Large Action Model framework LaVague! Find out more on https://github.com/mithril-security/LaVague

Edit: Here is the video of 🌊LaVague posting this. This is quite meta

samusenps osanseviero abidlabs akhaliq PeepDaSlan9 julien-c eduagarcia diegotluz OmRajani

samusenps abidlabs akhaliq julien-c diegotluz


How Far Are We from Intelligent Visual Deductive Reasoning? (2403.04732)

Can you beat an AI at Raven puzzles?

https://huggingface.co/spaces/HuggingFaceM4/ai_raven

The most powerful vision+language AI systems like Gemini or GPT4V struggle with this problem when used out-of-the-box (https://huggingface.co/papers/2403.04732).

But when properly trained, a small ~8B model can be very accurate at these IQ tests, solely based on visual inputs!

Raven's Progressive Matrices are visual intelligence tests invented in the 1930s designed to measure abstract reasoning and problem-solving ability. The test consists of a series of matrices or patterns with one part missing. The task for the test-taker is to identify the missing piece from a set of options.

Such puzzles can be procedurally generated at scale. https://huggingface.co/datasets/HuggingFaceM4/RAVEN is one example. The complexity of the puzzles is then controlled by the complexity of the generation procedure.

We fine-tuned an early checkpoint of our upcoming vision-and-language model idefics2 on that dataset. The resulting checkpoint yields ~91% accuracy! No chain of thoughts, no pre-processing of the image, no additional inputs or metadata, just the RAVEN problem fed to the model as a standalone image (and a short instruction to the model “Which figure should complete the logical sequence?”), with the training objective being the standard cross-entropy.

Just another evidence that in a lot of cases, for a given well-scoped problem, you will be better off paying to collect & annotate data, and fine-tune a model on that data (i.e. build your own AI) than wastefully trying to solve that problem with a gigantic general-purpose model you call through a paid API!


Can we align code generation models to be good at chat without compromising their base capabilities 🤔?

This was the question the H4 team asked itself when BigCode released StarCoder2 a bit over a week ago. We knew that code models like https://huggingface.co/deepseek-ai/deepseek-coder-6.7b-instruct and https://huggingface.co/m-a-p/OpenCodeInterpreter-DS-33B get impressive scores on code benchmarks like HumanEval, but they tend to score poorly on chat benchmarks like MT Bench and IFEval. We also knew that the Zephyr recipe we applied to Mistral 7B produced a strong chat model, so we wondered -- could be tweaked to produce a strong coding assistant?

It turns out the answer is yes and I'm happy to share StarChat2, a DPO fine-tune of StarCoder2 15B that scores highly on both HumanEval and MT Bench / IFEval 🌟!

The most interesting lesson for me was that you get better models by blending in more code/math data than chat during the SFT step - in terms of tokens, we found a ratio of 3:1 worked best.

Anyway, here's a demo of the model, along with all the code and datasets we used to train it:

* Demo: https://huggingface.co/spaces/HuggingFaceH4/starchat2-playground
* Collection: https://huggingface.co/collections/HuggingFaceH4/starchat2-15b-65f068417b330fafad751fce
* Recipe: https://github.com/huggingface/alignment-handbook

Hope it's useful to others!

lvwerra kashif osanseviero smangrul jordangong Chunte sasikiran MexIvanov vicgalle medmac01


🎥 🤾 Vid2Persona: talk to person from video clip

A fun project over the last week with @sayakpaul. It has a simple pipeline from extracting traits of video characters to chatting with them.

Under the hood, this project leverages the power of both commercial and open source models. We used Google's Gemini 1.0 Pro Vision model to understand the video content directly, then we used HuggingFaceH4/zephyr-7b-beta model to make conversation!

Try it Hugging Face Space and let us know what you think.
: https://huggingface.co/spaces/chansung/vid2persona

The space application is a dedicated implementation for ZeroGPU environment + Hugging Face Inference API with PRO account. If you wish to host it on your own environment, consider duplicate the space or run locally with the project repository
: https://github.com/deep-diver/Vid2Persona

chansung anonyme789 Chan-Y victor osanseviero xprilion

anonyme789 Srulikbd osanseviero xprilion


If you're trying to run MoE Mixtral-8x7b under DeepSpeed w/ HF Transformers it's likely to hang on the first forward.

The solution is here https://github.com/microsoft/DeepSpeed/pull/4966?_x_tr_sl=auto&_x_tr_tl=en&_x_tr_hl=en-US#issuecomment-1989671378

and you need deepspeed>=0.13.0

Thanks to Masahiro Tanaka for the fix.

nss-ysasaki sitloboi2012 theainerd osanseviero victor muhtasham hangzhang-nlp


Relightable and Animatable Neural Avatars from Videos (2312.12877)

🚀🕺🌟 New Research Alert - AAAI 2024! 🌟💃🚀
📄 Title: Relightable and Animatable Neural Avatars from Videos 🌟🚀

📝 Description: Relightable & animatable neural avatars from sparse videos.

👥 Authors: Wenbin Lin, Chengwei Zheng, Jun-Hai Yong, and Feng Xu

📅 Conference: AAAI, February 20-27, 2024 | Vancouver, Canada 🇨🇦

🔗 Paper: https://huggingface.co/papers/2312.12877

🌐 Github Page: https://wenbin-lin.github.io/RelightableAvatar-page
📁 Repository: https://github.com/wenbin-lin/RelightableAvatar

📺 Video: https://www.youtube.com/watch?v=v9rlys0xQGo

📚 More Papers: more cutting-edge research presented at other conferences in the https://huggingface.co/spaces/DmitryRyumin/NewEraAI-Papers curated by @DmitryRyumin

🚀 Added to the Avatars Collection: https://huggingface.co/collections/DmitryRyumin/avatars-65df37cdf81fec13d4dbac36

📚 Added to the AAAI 2024 Papers: https://github.com/DmitryRyumin/AAAI-2024-Papers

🔍 Keywords: #NeuralAvatar #RelightableAvatars #AnimatableAvatars #3DModeling #PhotorealisticRendering #ShadowModeling #DigitalAvatars #GeometryModeling #AAAI2024 #DeepLearning #Animation #Innovation

DmitryRyumin samusenps osanseviero sitloboi2012 Lician


Gemini 1.5: Unlocking multimodal understanding across millions of tokens
  of context (2403.05530)

671 authors 🤯

https://huggingface.co/papers/2403.05530 Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

osanseviero muhtasham dvilasuero pabloce diwank


New foundation model on document understanding and generation in  transformers 🤩
UDOP by MSFT is a bleeding-edge model that is capable of many tasks, including question answering, document editing and more! 🤯
Demo 👉 https://huggingface.co/spaces/merve/UDOP
It is a model that combines vision, text and layout. 📝
This model is very interesting because the input representation truly captures the nature of the document modality: text, where the text is, and the layout of the document matters!
If you know T5, it resembles that: it's pre-trained on both self-supervised and supervised objectives over text, image and layout.
To switch between tasks, one simply needs to change the task specific prompt at the beginning, e.g. for QA, one prepends with Question answering.
As for the architecture, it's like T5, except it has a single encoder that takes in text, image and layout, and two decoders (text-layout and vision decoders) combined into one.
The vision decoder is a masked autoencoder (thus the capabilities of document editing).
For me, the most interesting capability is document reconstruction, document editing and layout re-arrangement. This decoder isn't released though because it could be used maliciously to fake document editing.
Overall, the model performs very well on document understanding benchmark (DUE) and also information extraction (FUNSD, CORD) and classification (RVL-CDIP) for vision, text, layout modalities.
You can learn more about the model from below resources (h/t to
@nielsr), thanks a lot for reading 🤗
Docs: https://huggingface.co/docs/transformers/main/en/model_doc/udop 📚
Checkpoints: https://huggingface.co/collections/microsoft/udop-65e625124aee97415b88b513
Demo notebooks: https://github.com/NielsRogge/Transformers-Tutorials/tree/master/UDOP 📕

samusenps rajistics not-lain cstr diwank


Information Flow Routes: Automatically Interpreting Language Models at
  Scale (2403.00824)

🔍 Today's pick in Interpretability & Analysis of LMs: Information Flow Routes: Automatically Interpreting Language Models at Scale by @javifer @lena-voita

This work presents a novel method to identify salient components in Transformer-based language models by decomposing the contribution of various model components into the residual stream.

This method is more efficient and scalable than previous techniques such as activation patching, as it only requires a single forward pass through the model to identify critical information flow paths. Moreover, it can be applied without a contrastive template, which is observed to produce results dependent on the selected contrastive example for activation patching.

Information flow routes are applied to Llama 2, showing that:

1. Models show “typical” information flow routes for non-content words, while content words don’t exhibit such patterns.
2. Feedforward networks are more active in the bottom layers of the network (where e.g. subject enrichment is performed) and in very last layer.
3. Positional and subword-merging attention heads are among the most active and important throughout the network.
4. Periods can be treated by the model as BOS tokens by leaving their residual representation mostly untouched during the forward pass.

Finally, the paper also demonstrates that some model components are specialized for specific domains, such as coding or multilingual texts, suggesting a high degree of modularity in the network. The contribution of domain-specific heads obtained by projecting right singular values of the OV circuit to the unembedding matrix show highly interpretable concepts being handled in granular model components.

📄 Paper: https://huggingface.co/papers/2403.00824

🔍 All daily picks: https://huggingface.co/collections/gsarti/daily-picks-in-interpretability-and-analysis-of-lms-65ae3339949c5675d25de2f9

Kukedlc samusenps javifer CristianJD Theli


ShortGPT: Layers in Large Language Models are More Redundant Than You
  Expect (2403.03853)

A recent paper titled "ShortGPT: Layers in Large Language Models are More Redundant Than You Expect" proposes a simple and effective approach to pruning Large Language Models (LLMs) by removing redundant layers.

Key points:
* Discovers significant redundancy across layers in LLMs, with some layers playing a negligible role for the final performance.
* Defines a new metric called Block Influence (BI) to quantify the importance of each layer in an LLM.
* Removes layers with low BI scores, achieving up to 25% reduction in parameters and computation while maintaining 92% of the LLM's performance.

Congrats to the authors for their work!

Paper: https://huggingface.co/papers/2403.03853


ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment (2403.05135)

ELLA

Equip Diffusion Models with LLM for Enhanced Semantic Alignment

https://huggingface.co/papers/2403.05135

Diffusion models have demonstrated remarkable performance in the domain of text-to-image generation. However, most widely used models still employ CLIP as their text encoder, which constrains their ability to comprehend dense prompts, encompassing multiple objects, detailed attributes, complex relationships, long-text alignment, etc. In this paper, we introduce an Efficient Large Language Model Adapter, termed ELLA, which equips text-to-image diffusion models with powerful Large Language Models (LLM) to enhance text alignment without training of either U-Net or LLM. To seamlessly bridge two pre-trained models, we investigate a range of semantic alignment connector designs and propose a novel module, the Timestep-Aware Semantic Connector (TSC), which dynamically extracts timestep-dependent conditions from LLM. Our approach adapts semantic features at different stages of the denoising process, assisting diffusion models in interpreting lengthy and intricate prompts over sampling timesteps. Additionally, ELLA can be readily incorporated with community models and tools to improve their prompt-following capabilities. To assess text-to-image models in dense prompt following, we introduce Dense Prompt Graph Benchmark (DPG-Bench), a challenging benchmark consisting of 1K dense prompts. Extensive experiments demonstrate the superiority of ELLA in dense prompt following compared to state-of-the-art methods, particularly in multiple object compositions involving diverse attributes and relationships.

vladbogo Dlbk samusenps ClayFace Zmu melohux Benson


𝐑𝐮𝐧𝐬 𝐟𝐚𝐬𝐭 𝐚𝐥𝐬𝐨 𝐨𝐧 𝐂𝐏𝐔!

𝐑𝐮𝐧𝐬 𝐟𝐚𝐬𝐭 𝐚𝐥𝐬𝐨 𝐨𝐧 𝐂𝐏𝐔!

Super work from @DeepMount00:

🚀 𝐃𝐢𝐬𝐜𝐨𝐯𝐞𝐫 𝐔𝐧𝐢𝐯𝐞𝐫𝐬𝐚𝐥 𝐍𝐞𝐫: 𝐀 𝐆𝐥𝐢𝐍𝐞𝐫-𝐁𝐚𝐬𝐞𝐝 𝐈𝐭𝐚𝐥𝐢𝐚𝐧 𝐍𝐄𝐑

Introducing 𝐔𝐧𝐢𝐯𝐞𝐫𝐬𝐚𝐥 𝐍𝐞𝐫 𝐟𝐨𝐫 𝐈𝐭𝐚𝐥𝐢𝐚𝐧 𝐋𝐚𝐧𝐠𝐮𝐚𝐠𝐞, a revolutionary Named Entity Recognition (NER) model evolved from the GliNer architecture and meticulously tailored for the Italian language. This advanced model is a beacon of efficiency and versatility, engineered to 𝐫𝐞𝐜𝐨𝐠𝐧𝐢𝐳𝐞 𝐚𝐧𝐲 𝐞𝐧𝐭𝐢𝐭𝐲 𝐭𝐲𝐩𝐞 within the rich nuances of Italian, using a bidirectional transformer encoder. It stands out as an ideal solution for those navigating the challenges of resource-limited environments or seeking an efficient alternative to the cumbersome Large Language Models (LLMs).
𝐑𝐮𝐧𝐬 𝐟𝐚𝐬𝐭 𝐚𝐥𝐬𝐨 𝐨𝐧 𝐂𝐏𝐔!

Experience this Italian-focused innovation live on Hugging Face Spaces:
https://huggingface.co/spaces/DeepMount00/universal_ner_ita

Paper:  https://arxiv.org/abs/2311.08526 Urchade Zaratiana et all. great work!

samusenps osanseviero Dlbk DeepMount00 urchade giux78 tomaarsen MoritzLaurer


Good time to drop some 🤗 in the comments  > https://x.com/elonmusk/status/1767108624038449405

samusenps Dlbk sa8 frntn clefourrier Zmu CKeibel lvalue clem


Some papers deserve a standing ovation after reading, “Direct Preference Optimization: Your Language Model is Secretly a Reward Model” is one such paper:

One major drawback of LLMs is the lack of precise control over their behavior which makes it very difficult to align with desired outcomes. Existing methods to mitigate this involves gathering generated, humanly labeled data and fine-tuning the unsupervised LLM to align with preferences - this is known as Reinforcement Learning From Human Feedback (RLHF).

RLHF is an incredibly complex, usually unstable and computationally costly method. It involves first scaling a suitable reward model that meets human preferences then fine-tuning the language model with RL to maximize the estimated reward while maintaining a major part of the original model.

This paper introduces a new algorithm called Direct Preference Optimization (DPO) that simplifies the whole process. In short, it directly optimizes the LM without explicit reward modeling or reinforcement learning.  This is achieved by leveraging a mapping between reward functions and optimal policies, allowing the constrained reward maximization problem to be optimized exactly with a single stage of policy training.

DPO’s genius lies in its ability to intuitively increase the relative log probability of preferred to "unpreferred" responses.

The amazing thing about this paper is how fundamentally self-proven it is - from clearly stating the problem to explicitly explaining the underlying theory backed with mathematical proofs, it’s just genius.

In my opinion, every academic research paper should follow this approach.  It won the 2023 NeurIPS Outstanding paper award (Category: Outstanding Main Track Runner-Ups).

osanseviero victor stonestack kawinm Theli AI-B


🚀⭐Introducing our new survey "Leveraging Biomolecule and Natural Language through Multi-Modal Learning: A Survey"

arxiv: https://arxiv.org/abs/2403.01528
github: https://github.com/QizhiPei/Awesome-Biomolecule-Language-Cross-Modeling

The integration of biomolecular modeling with natural language (BL) has emerged as a promising interdisciplinary area at the intersection of artificial intelligence, chemistry and biology. This approach leverages the rich, multifaceted descriptions of biomolecules contained within textual data sources to enhance our fundamental understanding and enable downstream computational tasks such as biomolecule property prediction. The fusion of the nuanced narratives expressed through natural language with the structural and functional specifics of biomolecules described via various molecular modeling techniques opens new avenues for comprehensively representing and analyzing biomolecules. By incorporating the contextual language data that surrounds biomolecules into their modeling, BL aims to capture a holistic view encompassing both the symbolic qualities conveyed through language as well as quantitative structural characteristics. In this review, we provide an extensive analysis of recent advancements achieved through cross modeling of biomolecules and natural language.

QizhiPei osanseviero victor samusenps MaziyarPanahi bczhou


CAMixerSR: Only Details Need More "Attention" (2402.19289)

🚀🖼🌟 New Research Alert - CVPR 2024! 🌟🖼🚀
📄 Title: CAMixerSR: Only Details Need More "Attention" 🌟🚀

📝 Description: CAMixerSR is a new approach integrating content-aware accelerating framework and token mixer design, to pursue more efficient SR inference via assigning convolution for simple regions but window-attention for complex textures. It exhibits excellent generality and attains competitive results among state-of-the-art models with better complexity-performance trade-offs on large-image SR, lightweight SR, and omnidirectional-image SR.

👥 Authors: Yan Wang, Shijie Zhao, Yi Liu, Junlin Li, and Li Zhang

📅 Conference: CVPR, Jun 17-21, 2024 | Seattle WA, USA 🇺🇸

🔗 Paper: https://huggingface.co/papers/2402.19289

🔗 Repository: https://github.com/icandle/CAMixerSR

📚 More Papers: more cutting-edge research presented at other conferences in the https://huggingface.co/spaces/DmitryRyumin/NewEraAI-Papers curated by @DmitryRyumin

🚀 Added to the Image Enhancement Collection: https://huggingface.co/collections/DmitryRyumin/image-enhancement-65ee1cd2fe1c0c877ae55d28

🔍 Keywords: #CAMixerSR #SuperResolution #WindowAttention #ImageEnhancement #CVPR2024 #DeepLearning #Innovation

DmitryRyumin samusenps osanseviero Jakaline victor Dimo87


Quantize 7B paramater models in 60 seconds using Half Quadratic Quantization (HQQ).

This game-changing technique allows for rapid quantization of models like Llama-2-70B in under 5 minutes, outperforming traditional methods by 50x in speed and offering high-quality compression without calibration data.

Mobius Labs innovative approach not only significantly reduces memory requirements but also enables the use of large models on consumer-grade GPUs, paving the way for more accessible and efficient machine learning research.

Mobius Labs' method utilizes a robust optimization formulation to determine the optimal quantization parameters, specifically targeting the minimization of errors between original and dequantized weights. This involves employing a loss function that promotes sparsity and utilizes a non-convex lp<1-norm, making the problem challenging yet solvable through a Half-Quadratic solver.

This solver simplifies the problem by introducing an extra variable and dividing the optimization into manageable sub-problems. Their implementation cleverly fixes the scale parameter to simplify calculations and focuses on optimizing the zero-point, utilizing closed-form solutions for each sub-problem to bypass the need for gradient calculations.

Check out the colab demo where you are able to quantize models (text generation and multimodal) for use with vLLM or Timm backend as well as transformers!

AutoHQQ: 👉 https://colab.research.google.com/drive/1cG_5R_u9q53Uond7F0JEdliwvoeeaXVN?usp=sharing
Code: https://github.com/mobiusml/hqq
HQQ Blog post: https://mobiusml.github.io/hqq_blog/

Edit: Here is an example of how powerful HQQ can be: https://huggingface.co/macadeliccc/Nous-Hermes-2-Mixtral-8x7B-DPO-HQQ

Citations:

@misc{badri2023hqq,
	title = {Half-Quadratic Quantization of Large Machine Learning Models},
	url = {https://mobiusml.github.io/hqq_blog/},
	author = {Hicham Badri and Appu Shaji},
	month = {November},
	year = {2023}
}

Warung appoose boqiangliang hiepxanh Lewdiculous cnmoro MH0386


Over the past week, I've been putting Claude through its paces, focusing primarily on productivity tasks (you know, the good old BAU – Business As Usual).

1. Python/Torch/Transformers/AI/ML
Right off the bat, I threw some complex AI/ML tasks at Claude, and I must say, it handled them with finesse. It even caught a few things that GPT missed! However, let's not get too carried away – we're not quite at the auto-code level just yet.

2. Brainstorming
This is where Claude falls a bit short. It seems to be more grounded than its competitors, which might not be ideal for generating novel ideas. If you're looking for a brainstorming partner, you might want to look elsewhere.

3. Attention
Despite the claims of super-large attention in the paper, Claude's "forgetting" mechanism seems to be more pronounced. It tends to miss entire chunks of information rather than just specific details like GPT does.

4. Following / Tasks
I hit a roadblock when Claude couldn't generate a LaTeX document. It's not the best at following complex, multi-step tasks.

5. Hallucinations
Oh boy, does Claude hallucinate! And when it does, it's on a whole new level of nonsense. The hallucinations seem to align with its grounded nature, making them even more convincing within the context of the prompt.

6. Sycophancy
Claude is quite the people-pleaser. I've found that using an adversarial brainstorming approach is more beneficial and time-efficient, as it forces me to highlight Claude's mistakes rather than letting it focus on being a sweet, pleasant minion.

7. Interface / UI
There's definitely room for improvement here. Basic features like stepping back on a prompt and stopping generation with the ESC key are missing. These are essential for extracting and composing content effectively.

Despite these limitations, I firmly believe that Claude is currently the #1

samusenps macadeliccc osanseviero victor delano Nymbo

Иванов Константин


LAST CHANCE TO TRY 🌟STARCODER2

After today it's gone !

actually not - just joking ! it's <3 open source !

 just trying to get folks' attention to my featured "Spaces of the Week" :

https://huggingface.co/spaces/Tonic/starcoder2

drop a like for your boy and join us next week for making fine tunes !


Multi-Instance Generation Controller: Enjoy complete control over position generation, attribute determination, and count!

code link: https://github.com/limuloo/MIGC
project page: https://migcproject.github.io/

MIGC decouples multi-instance generation into individual single-instance generation subtasks within the cross-attention layer of Stable Diffusion.

Welcome to follow our project and use the code to create anything you imagine!

Please let us know if you have any suggestions!

Flowerfan limuloo1999 little-air-dust LeyRio xiaodi sanaka87 wenhaozzu yqqCheergo Whalesong Csplk mingzs2 samusenps taufiqdp Jouryjc osanseviero Warung George-Blaze danniel2023 linoyts Jakaline ariarcus erickdp radames

limuloo1999 xiaodi sanaka87 Shinku Whalesong mingzs2 samusenps vvvvJ osanseviero danniel2023 linoyts shashankbodapati radames jeffboudier


Updating PaperQA Gradio app and Hugging Face Space.
: Link ➡ https://huggingface.co/spaces/chansung/paper_qa
: Standalone repo ➡ https://github.com/deep-diver/paperqa-ui

The final goal is to let ppl have their own paper archive. At the end, You will be able to easily *clone* on local or Hugging Face Space with Google's Gemini API Key (which is free), Hugging Face Access Token. You can just drop arXiv IDs at the bottom, then all the auto analyze papers are automatically archived on Hugging Face Dataset repo.

Here are few updates included, and dig in the source code if you want similar features for your use cases!
🖥 making complex UI + fully responsive
+ making UI as quickly as possible (avoid server-client when possible)
💬 Permanent Chat history management with in-browser local storage
+ Chat history management *per* paper
+ Chat history management in lazy mode (too many paper, impossible to create chat history for every single paper beforehand, hence)

Current plan is to support Gemini and any open source models on Hugging Face PRO account, but will expand it to GPT4 soon.

Any suggestion on this project is welcome! possibly,
- hooking up RAG system (open models' context length is small)
- hooking up Internet search system
- image/figure analysis
....


Keep stacking cool stuff and getting better results! After I changed the standard vision encoder to SigLIP, NLLB-CLIP got a 10% average performance improvement. And now, I added matryoshka layers (https://arxiv.org/abs/2205.13147) to enable smaller embeddings and got another 6% performance boost! Plus, thanks to MRL, 4.5x smaller embeddings retain 90%+ quality.

The large model is finally SoTA for both image and text multilingual retrieval!

The models are available on the hub:
- https://huggingface.co/visheratin/nllb-siglip-mrl-base
- https://huggingface.co/visheratin/nllb-siglip-mrl-large


Retrieval-Augmented Generation (RAG)
Redeemer of the "hallucination problem"

It is fair enough to argue that "hallucinations" in LLMs are just mere reflections of what we humans occasionally do - well it gets worse as we get older, but these models are brain inspired, thus such behaviors are likely inherently unavoidable. After all, we are just dreamers trying make sense of this life.

The best we can do is minimize and control it - but humanly how? By first feeding on relevant facts and then developing a habit that allows us to easily access those facts when needed. This is what RAG is all about - it's just a control mechanism that keeps the LLM aligned with reality and fact.

But How Does RAG Work?

Well, to some extent it is domain-specific but the overall workflow boils down to the following:

1. It makes use of a retrieval mechanism that hunts for facts relevant to a query - this involves an end-to-end backpropagation that leverages a retriever (Query Encoder + Document Index or Source of Truth) with a pre-trained generative model.

2. The generative model then uses the facts retrieved, performs some verification to give a more accurate response.

To summarize, the RAG architecture houses a pre-existing knowledge source model (termed parametric memory), which then utilizes a Source-of-Truth model or vector indexed data (termed non-parametric memory) that is accessed by a pre-trained neural retriever, in order to produce more informed, contextually appropriate and factually correct responses.

Sort of a "Genius Engine" if you might say. If only we humans could harness such, AGI would be much much sooner lol.

In the meantime, I have been Jaward Sesay (Chinese name 苏杰 Sujie) - a young Sierra Leonean, aspiring AI Researcher. I like to read, share and try implementing AI research papers. Also like dunking on big tech while rooting for open-source. My mentor @karpathy, I dream of him following me back on X lol. Thanks.

samusenps osanseviero paralym Jafta rohitdavas


Finetuned Multimodal Language Models Are High-Quality Image-Text Data
  Filters (2403.02677)

A recent paper titled "Finetuned Multimodal Language Models Are High-Quality Image-Text Data Filters" proposes using fine-tuned Multimodal Language Models (MLMs) as high-quality filters for image-text data.

Key points:
* Defines multiple metrics to assess image-text quality from different perspectives like object details, text quality, and semantic understanding.
* Leverages GPT-4 and GPT-4V to construct high-quality instruction data for fine-tuning open-source MLMs as effective data filters.
* Fine-tuned MLM filters generate more precise scores, leading to better filtered data and improved performance of pre-trained models on various downstream tasks.

Congrats to the authors for their work!

Paper:  https://huggingface.co/papers/2403.02677
Code: https://github.com/Victorwz/MLM_Filter
Dataset: https://huggingface.co/datasets/weizhiwang/mlm_filter_instructions
Model: https://huggingface.co/weizhiwang/mlm-filter-llava-13b-gpt4v


Introducing the 🤗 Transformers.js WebGPU Embedding Benchmark! ⚡
👉 https://huggingface.co/spaces/Xenova/webgpu-embedding-benchmark 👈

On my device, I was able to achieve a 64.04x speedup over WASM! 🤯 How much does WebGPU speed up ML models running locally in your browser? Try it out and share your results! 🚀

samusenps omaryshchenko thomwolf osanseviero Seuriin Tonic ozzyonfire amiratatomic abidlabs hogunkim felixdrp leekung Noomam xnohat do-me mikestaub

mvaloatto Tonic kramp abidlabs pogzyb cklam12345


OpenCodeInterpreter: Integrating Code Generation with Execution and
  Refinement (2402.14658)

Diaries of Open Source. Part 3! OS goes to the moon!

💻  OpenCodeInterpreter, a family of very powerful code generation models
Models: https://huggingface.co/collections/m-a-p/opencodeinterpreter-65d312f6f88da990a64da456
Paper: https://huggingface.co/papers/2402.14658
Demo https://huggingface.co/spaces/m-a-p/OpenCodeInterpreter_demo

🔷🔶Zephyr 7B Gemma, Gemma fine-tuned with the Zephyr recipe
Model: https://huggingface.co/HuggingFaceH4/zephyr-7b-gemma-v0.1
Demo: https://huggingface.co/spaces/HuggingFaceH4/zephyr-7b-gemma-chat
GH Repo: https://github.com/huggingface/alignment-handbook

🪆The MixedBread folks released a 2D Matryoshka text embedding model, which means you can dynamically change the embedding size and layer counts
Model: https://huggingface.co/mixedbread-ai/mxbai-embed-2d-large-v1
Release blog post: https://www.mixedbread.ai/blog/mxbai-embed-2d-large-v1

🐋Microsoft released Orca Math, which includes 200K grade school math problems
Dataset: https://huggingface.co/datasets/microsoft/orca-math-word-problems-200k

🥷IBM silently released Merlinite, a cool model trained on Mixtral-generated synthetic data using a novel LAB method https://huggingface.co/ibm/merlinite-7b

🌚 Moondream2 - a small vision language model to run on-device!
Model: https://huggingface.co/vikhyatk/moondream2
Demo: https://huggingface.co/spaces/vikhyatk/moondream2

🏙CityDreamer: 3D City Generation
Demo: https://huggingface.co/spaces/hzxie/city-dreamer
Repo: https://github.com/hzxie/city-dreamer
Model: https://huggingface.co/hzxie/city-dreamer

🌏ML in all languages
Sailor, a family of South-East Asian languages models https://huggingface.co/collections/sail/sailor-language-models-65e19a749f978976f1959825
Samvaad dataset, which includes 140k QA pairs in Hindi, Bengali, Marathi, Tamil, Telugu, Oriya, Punjabi, and Gujarati https://huggingface.co/datasets/GenVRadmin/Samvaad-Mixed-Language-2

You can see the previous part at https://huggingface.co/posts/osanseviero/674644082063278

DmitryRyumin fffiloni samusenps vladbogo unclecode visheratin ajibawa-2023 mvaloatto mikonvergence Makya


I have just published my first blog post.

While FlashAttention has been readily integrated into HuggingFace transformers, there are much higher gains to be had (at least theoretically) for finetuning models with examples of variable sequence lengths in a batch.

For a deeper dive, please go through my blog at https://huggingface.co/blog/mayank-mishra/padding-free-transformer.

Kukedlc osanseviero tomaarsen samusenps eramax mayank-mishra julien-c

damerajee julien-c mayank-mishra jeffboudier


GPAvatar: Generalizable and Precise Head Avatar from Image(s) (2401.10215)

🚀🎭🌟 New Research Alert - ICLR 2024! 🌟🎭 🚀
📄 Title: GPAvatar: Generalizable and Precise Head Avatar from Image(s) 🌟🚀

📝 Description: GPAvatar's objective is to faithfully replicate head avatars while providing precise control over expressions and postures.

👥 Authors: Xuangeng Chu et al.

📅 Conference: ICLR, May 7-11, 2024 | Vienna, Austria 🇦🇹

🔗 Paper: https://huggingface.co/papers/2401.10215

🔗 Github Page: https://xg-chu.github.io/project_gpavatar
🔗 Repository: https://github.com/xg-chu/GPAvatar

🔗 Video: https://www.youtube.com/watch?v=7A3DMaB6Zk0

📚 More Papers: more cutting-edge research presented at other conferences in the https://huggingface.co/spaces/DmitryRyumin/NewEraAI-Papers curated by @DmitryRyumin

🚀 Added to the Avatars Collection: https://huggingface.co/collections/DmitryRyumin/avatars-65df37cdf81fec13d4dbac36

🔍 Keywords: #GPAvatar #MTA #Synthesis #LipSyncing #Expressions #HighResolutionVideos #ICLR2024 #DeepLearning #Animation #Innovation

DmitryRyumin osanseviero samusenps philipp-zettl rreed-pha


Hi!
We release a series of LLMs named GraphWiz https://huggingface.co/GraphWiz/, which is designed for solving different graph problems and outputs their CoT. GraphWiz has surpassed GPT-4 by a large margin in 9 tasks.

Let us have a try!

Nuo97 osanseviero Kukedlc samusenps kramp Androidkit yoeldcd


Last day on Spaces of the Week ,
and we made it to last place on trending.
i really thought it couldnt get any better, but i'm crying ! 😭

The thing i like the most about ZeroGPU , `import spaces` , is that i dont have to always check to see if someone decided to test if i have hard character limits , and it reloads the application flawlessly .

drop a like on my spaces here :
Spaces of the Week : https://huggingface.co/spaces/tonic/starcoder2
9 other ZeroGPU demos :  https://huggingface.co/tonic


I remember very well that about two years ago, 0-shot named entity recognition (i.e. where you can choose any labels on the fly) was completely infeasible. Fast forward a year, and https://huggingface.co/Universal-NER/UniNER-7B-all surprised me by showing that 0-shot NER is possible! However, I had a bunch of concerns that prevented me from ever adopting it myself. For example, the model was 7B parameters, only worked with 1 custom label at a time, and it had a cc-by-nc-4.0 license.

Since then, a little known research paper introduced GLiNER, which was a modified & finetuned variant of the https://huggingface.co/microsoft/deberta-v3-base line of models. Notably, GLiNER outperforms UniNER-7B, despite being almost 2 orders of magnitude smaller! It also allows for multiple labels at once, supports nested NER, and the models are Apache 2.0.

Very recently, the models were uploaded to Hugging Face, and I was inspired to create a demo for the English model. The demo runs on CPU, and can still very efficiently compute labels with great performance. I'm very impressed at the models.

There are two models right now:
* base (english): https://huggingface.co/urchade/gliner_base
* multi (multilingual): https://huggingface.co/urchade/gliner_multi

And my demo to experiment with the base model can be found here: https://huggingface.co/spaces/tomaarsen/gliner_base

osanseviero MexIvanov urchade rajistics samusenps ajibawa-2023 danielus nickprock polodealvarado deepsh2207 luqman2511 chokamp Poulain Godspeed-AI eek louisguitton


Multi-LoRA Composition for Image Generation (2402.16843)

"Multi-LoRA Composition for Image Generation" introduces two new approaches for combining multiple visual elements in text-to-image generation using Low-Rank Adaptations (LoRAs)! 🎨

Key Points:
* Proposes two methods - LoRA Switch and LoRA Composite - that activate/combine LoRAs during the denoising process rather than merging weights
* LoRA Switch cycles through different LoRAs at each step, while LoRA Composite averages guidance from all LoRAs simultaneously

Paper: https://huggingface.co/papers/2402.16843
Project page: https://maszhongming.github.io/Multi-LoRA-Composition

Congrats to the authors for their work!

DmitryRyumin osanseviero samusenps mvaloatto femboysLover


Introducing https://huggingface.co/datasets/davanstrien/cosmopedia_chat (v0.0.1), my first experiment using the new NousResearch Genstruct model https://huggingface.co/NousResearch/Genstruct-7B

This dataset uses a subset of https://huggingface.co/datasets/HuggingFaceTB/cosmopedia, a synthetic textbook-quality dataset, and Genstruct to generate user/assistant response pairs.

My current results are mixed, but I'm excited to see how much work is happening around synthetic data generation in the community.  Most crucial next step is working more on data filtering from cosmopedia.

Massive thanks to @euclaise @teknium and the other NouseResearch folks for sharing this model ❤


Learning to Generate Instruction Tuning Datasets for Zero-Shot Task
  Adaptation (2402.18334)

Create synthetic instruction datasets using open source LLM's and bonito🐟!

With Bonito, you can generate synthetic datasets for a wide variety of supported tasks.

The Bonito model introduces a novel approach for conditional task generation, transforming unannotated text into task-specific training datasets to facilitate zero-shot adaptation of large language models on specialized data.

This methodology not only improves the adaptability of LLMs to new domains but also showcases the effectiveness of synthetic instruction tuning datasets in achieving substantial performance gains.

AutoBonito🐟: https://colab.research.google.com/drive/1l9zh_VX0X4ylbzpGckCjH5yEflFsLW04?usp=sharing
Original Repo: https://github.com/BatsResearch/bonito?tab=readme-ov-file
Paper: https://huggingface.co/papers/2402.18334

samusenps pabloce clem nihalnayak damerajee jayomb nanyy1025


PixArt-Σ: Weak-to-Strong Training of Diffusion Transformer for 4K
  Text-to-Image Generation (2403.04692)

PixArt-Σ

Weak-to-Strong Training of Diffusion Transformer for 4K Text-to-Image Generation

https://huggingface.co/papers/2403.04692

In this paper, we introduce PixArt-\Sigma, a Diffusion Transformer model~(DiT) capable of directly generating images at 4K resolution. PixArt-\Sigma represents a significant advancement over its predecessor, PixArt-\alpha, offering images of markedly higher fidelity and improved alignment with text prompts. A key feature of PixArt-\Sigma is its training efficiency. Leveraging the foundational pre-training of PixArt-\alpha, it evolves from the `weaker' baseline to a `stronger' model via incorporating higher quality data, a process we term "weak-to-strong training". The advancements in PixArt-\Sigma are twofold: (1) High-Quality Training Data: PixArt-\Sigma incorporates superior-quality image data, paired with more precise and detailed image captions. (2) Efficient Token Compression: we propose a novel attention module within the DiT framework that compresses both keys and values, significantly improving efficiency and facilitating ultra-high-resolution image generation. Thanks to these improvements, PixArt-\Sigma achieves superior image quality and user prompt adherence capabilities with significantly smaller model size (0.6B parameters) than existing text-to-image diffusion models, such as SDXL (2.6B parameters) and SD Cascade (5.1B parameters). Moreover, PixArt-\Sigma's capability to generate 4K images supports the creation of high-resolution posters and wallpapers, efficiently bolstering the production of high-quality visual content in industries such as film and gaming.

samusenps clem mvaloatto victor thomwolf ianyeung DeathGodlike


As promised, a video on the new Major TOM dataset on HF: https://youtu.be/KonWxQ1mCpA?si=tp3Cz6-F0exfgdx3

If you prefer audio, there is a podcast too: https://www.satellite-image-deep-learning.com/p/major-tom-expandable-eo-datasets

I think this is just the start for remote sensing datasets on HuggingFace..!


🗄 Massive data release on the HF Hub for 75 languages!

https://huggingface.co/datasets/BramVanroy/hplt_monolingual_v1_2

In December of last year, HPLT (https://hplt-project.org/) released version 1.2 of their dataset. It covers web-crawled data of 75 languages!, in the raw format as well as deduplicated and cleaned sections. In total, we're talking about over 40TB of data! This data was already accessible via their website but I figured the accessibility could be improved by an integration with Hugging Face tooling. 🤗 So I added the dataset here to the Hugging Face hub, enabing direct use in your conventional training pipelines for LLMs or other language technologies. The data will automatically be downloaded and optimised with just one line of code:

```python
load_dataset("BramVanroy/hplt_mono_v1_2", "nl_cleaned")
```
Let's use this big blob of data to build something awesome in our languages! 🥳

stefan-it samusenps ajibawa-2023 clem osanseviero ymoslem damerajee rafaelpierrehf


We just released bitsandbytes==0.43.0 📦 , with these significant new additions:

‣ 🛫 FSDP+QLoRA support (alpha release)
  ◦ now anyone with 2 powerful gaming GPUs can fine-tune 70B param models at home!
  ◦ in collab with Jeremy Howard + team @ answer.ai
  ◦ answer.ai blogpost: https://www.answer.ai/posts/2024-03-06-fsdp-qlora.html
  ◦ example repo: https://github.com/AnswerDotAI/fsdp_qlora/

‣ 🌈⊞ Official Windows support
  ◦ now via simple `pip install bitsandbytes>=0.43.0`

‣ 📄 Huge docs update:
  ◦ https://huggingface.co/docs/bitsandbytes/main
  ◦ Be sure to check out the optimizers and the API docs
  ◦ ... even more upcoming ...

Under the hood there we have many other improvements, due to extensive maintenance activity, community contributions by super active + knowledgable volunteers ✨ 🚀 and the official sponsorship by Hugging Face that makes all this possible 🤗 ❤ 🌍

We would greatly appreciate any further community contributions, be it to help with refactorings, exterminating flaky tests, writing doc-strings, tutorials, new features. Don't be shy, just contact us and we see where this leads us:
https://github.com/TimDettmers/bitsandbytes/discussions

Have a great weekend everyone!

Titus-von-Koeller mdouglas clem osanseviero rganti ajibawa-2023 julien-c t1u1 ybelkada cstr tolgacangoz hiyouga danielus smangrul

Djruffkutz clem osanseviero samusenps julien-c Theli ybelkada hiyouga smangrul