For Romanian, Neurograf v1 makes the case for training a language model stack from scratch: 131 billion tokens, a 635 GB corpus, an 80k byte-level BPE tokenizer, five decoder-only models from 160M to 6B parameters, plus instruction-tuned assistants and contrastive embedding models. On the evaluation side, a first-order formula for masked-text reconstruction predicts how much outputs will drift if the corpus is resampled from the same population, using token-pair counts over the full 50,257-token GPT-2 vocabulary and accounting for document length, context counts, and offset correlations.

Neurograf v1: An End-to-End Romanian-First Language Model Family

For Romanian MT or localisation work, the practical gain is a 635 GB Romanian corpus, 80k byte-level BPE, and 6B-scale base and instruction-tuned models, but diacritic preservation and corpus quality remain the main risks.

We present Neurograf v1, an experimental end-to-end Romanian-first language-model pipeline developed to investigate the feasibility of training language models from scratch for a lower-resource European language. The pipeline comprises RoSfera, a 635 GB Romanian corpus containing 175.7 million documents; an 80k byte-level BPE tokenizer; five decoder-only models ranging from 160M to 6B parameters, trained on 131 billion tokens; three instruction-tuned assistants; and two contrastively trained embedding models. Training and evaluation were conducted using computing infrastructure at ELI-NP/IFIN-HH. The model family exhibits consistent scaling of validation loss and increasing Romanian factual and cultural knowledge with model size. On the 1,000-question WWTBM benchmark, Neurograf-6B achieves 69.89% accuracy on Romanian-context questions, while Neurograf-3.2B reaches 67.38%. Compute-optimal analysis yields a parameter-scaling exponent of approximately 0.423. Neurograf v1 serves as an experimental demonstrator rather than a production system. The study identifies practical challenges involving diacritic preservation, corpus quality, data composition, and tokenization efficiency, while demonstrating the feasibility of developing Romanian-first base models, instruction-following assistants, and semantic embedding models within a unified training pipeline.

Forecasting Corpus-Sampling Disagreement in Masked Text Reconstruction

For corpus-based MT evaluation, disagreement between models trained on two samples from the same data falls only 5–40% when corpus size quadruples, so 1/n uncertainty scaling is too optimistic.

If a second text corpus were collected from the same population, how much would a predictor's outputs change? We study this question for masked-text reconstruction using a smoothed predictor built from nearby token-pair counts and evaluated over the full 50,257-token GPT-2 vocabulary. Treating each source document as one observation, we derive a first-order formula for the expected disagreement between predictors fitted to two independent corpora. The formula accounts for variable document length, random context counts, and correlations across token offsets, and it can be evaluated from a separate reference corpus without access to either target corpus. On TinyStories, WikiText-103, and CNN/DailyMail, growing each corpus from 512 to 2,048 chunks of 64 tokens lowers disagreement by only 5--40%, far short of the 75% implied by 1/n scaling. The forecast tracks this slow decline, which we trace to smoothing of rarely seen contexts. Forecast accuracy depends less on how these correlations are modeled than on the size of the reference corpus: small references systematically underpredict disagreement. A source-split jackknife that uses only reference data reduces this calibration error by 57--59%. After this correction, a simpler smoothed-multinomial approximation is most accurate when the reference corpus is smaller than the target corpus, and a nonlinear source bootstrap is most accurate when it is larger. Finally, small masked diffusion language models trained on the same corpora show that the count-based forecasts partly identify which inputs are most sensitive to the corpus, but miss the neural models' much faster decline in disagreement.

Papers announced by arXiv on Sunday, 11 October 2026, covering submissions from 8 October to 9 October 2026.