Tencent Releases Open MoE Model Hy3 with 295 Billion Parameters
Translated by a model; the original is in Russian.
The Tencent Hy Team has unveiled a new model, Hy3 — a Mixture-of-Experts (MoE) model with 295 billion parameters, of which 21 billion are active, plus 3.8 billion parameters in the MTP layer. The model is distributed under the Apache 2.0 license.
Hy3 is an evolution of the preview version released in late April. According to the developers, in the intervening time they gathered feedback from more than 50 products and scaled up post-training using higher-quality data.
According to Tencent, the final model outperforms comparably sized peers and approaches the quality of flagship open models that are 2–5 times larger. They also note a marked increase in usefulness across various product scenarios and performance-related tasks.
Technical details
The full version of the model weighs in at 598 GB on Hugging Face, while the quantized FP8 version is 300 GB. The context length is 256 thousand tokens.
Until July 21, the model is available for free on the OpenRouter platform.
The source's author, Simon Willison, traditionally tested the model on his informal benchmark task — generating an SVG image of a pelican riding a bicycle — and got a result from Hy3 via OpenRouter.
Source: Simon Willison
Comments
Sign in, to comment.