HILBERT
HILBERT is the BERT-Large model for Hungarian Trained on the 4 BN NYTI-BERT corpus. One of the pioneers of the revolutionary transformer models, BERT has caused a sweeping success in the field of neural NLP.
HIL-SBERT
Sentence-BERT (SBERT) models are fine-tuned BERT networks aimed at obtaining high-quality sentence embeddings. SBERT was introduced in the original paper by Reimers and Gurevych (2019). We used multilingual knowledge distillation proposed by Reimers and Gurevych (2020) for creating a Hungarian model. The pre-trained hubert-base-cc (Nemeskey 2021) was fine-tuned on the Hunglish 2.0 parallel corpus (Varga et al. 2005) to mimic the bert-base-nli-stsb-mean-tokens model provided by UKPLab. Sentence embeddings were obtained by applying mean pooling to the huBERT output. The data was split into training (98%) and validation (2%) sets. By the end of the training, a mean squared error of 0.106 was computed on the validation set. Our code was based on the Sentence-Transformers library. Our model was trained for 2 epochs on a single GTX 1080Ti GPU card with the batch size set to 32. The training took approximately 15 hours.
To DOWNLOAD the models, please fill out the registration form: » REGISTRATION FORM «