First ever  English-Hungarian  GPT3 NEOX 7B modell

Produced by the Hilanco Consortium

  • Language Technology Research Group, Hungarian Research Centre for Linguistics (NYTK)
  • Applied Data Science and Artifical Intelligence Centre, University of Pécs (PTE).

Trained on

a single GDX A100 node (8 GPUs with 80GB RAM each) with ~60 PFlop/day compute power over a period of 92 days, using Microsoft DeepSpeed GPU optimization technology

Architecture

A 2 node cluster (GDX a100 node connected via Mellanox Infiniband to a Supermicro 4xA100 node, with the headnode containing 2Tb of RAM and 256 CPU’s)

Training Data

  • English 102 Bn words from The Pile corpus
  • Hungarian: 25 Bn words, compiled by NYTK from Common Crawl and own sources
  • The corpus was compiled using a Supermicro A100 node (4 GPUs 80GB RAM each).

Model Download

available for bona fide nonprofit research purposes. The inference environment requires 14 GB VRAM, the whole system 95GB

Demo and Download

For further details contact

feldmann.adam[at]pte.hu,
varadi.tamas[at]nytud.hu

HILANCO is a Natural Language Processing Consortium (NLP) in Hungary, between the Hungarian Research Centre for Linguistics (NYTK) and the University of Pécs, Applied Data Science and Artifical Intelligence Centre (PTE). The mission of the Consortium is to provide new NLP technologies and resources in order to build intelligent language applications for AI purposes. We provide cutting edge solutions to serve the whole spectrum of innovation from research and development to marketable business products.
For enquiries contact

Publications

Partners

microsoft

Members

Image

Tamás Váradi
Deputy Director - NYTK

Image

Ádám Feldmann
AI Lead - PTE

Image

Győző Zijian Yang
NLP Scientist - NYTK