Michal Štefánik

AIMO Interpretability Challenge

Standard benchmarks tell us whether a model solves olympiad-level maths problems, but not whether it does so through stable reasoning mechanisms or brittle reasoning shortcuts. This competition asks participants to tell these two apart by looking inside the models.

Building on the resources of the Fields Model Initiative, we provide participants with newly-published problems, their symbolic versions allowing to generate perturbed variants, access to frontier reasoning models, and robustness evaluations. The resulting methods for identifying robust problem-solving will be released as an open-source tools to interpretability and generalisation communities.

Language Models Learn Universal Representations of Numbers and Here's Why You Should Care

By analyzing how LLMs decode numbers, we develop better tools for extracting ordinal information from language models. We show that diverse models represent ordinal information in a universal, systematic, and remarkably accurate way across input contexts. We also provide practical guidelines for studying numerical reasoning in LLMs; showing that universal interpretability tools such as TunedLens can be highly inaccurate for numeric tokens, and interpretability methods for numbers should be trained and evaluated on natural-language inputs rather than synthetic contexts to achieve practical applicability.

Pre-trained Language Models Learn Remarkably Accurate Representations of Numbers

Language models are known to make mistakes in operations involving numbers. This raises an assumption that their distributionally-learnt representations of numbers may be highly inaccurate with respect to actual numeric values.

In this work, we first find that learned input embeddings of language models follow a systematic, distinct sinusoidal pattern. Subsequently, we develop a new probing method that is better suited for this pattern than commonly-used linear probes and contrary to previous work, we show that the numeric representations created in pre-training are surprisingly accurate -- reaching up to 100% of accuracy (Llama 3.1 8B) in distinguishing different integer values.

Can Out-of-Distribution Evaluations Uncover Reliance on Prediction Shortcuts? A Case Study in Question Answering

Out-of-distribution (OOD) evaluations are a go-to standard for evaluating models' generalization. In this work, we look at a specific case of Question Answering, where previous work documents a representative sample of prediction shortcuts, presenting an alternative way of assessing models' generalization.

In the case of Question Answering, we find that the (i) OOD performance and (ii) the reliance on shortcuts, both assessing models' generalization, are vastly uncorrelated. This is due to the fact that different OOD datasets share the same shortcuts, underlining blind spots of OODs' generalization assessments. Nevertheless, we also show that the relationship of datasets and shortcuts is more complex: even datasets good for training shortcut-free models can provide biased evaluation (and vice versa).

Attend or Perish: Benchmarking Attention in Algorithmic Reasoning

To what extent can Language Models comprehend and apply an *algorithm* underlying the training data? We construct an interpretability benchmark with annotated reference attention maps, based on which, we assess to what extent can different Transformers follow the robust, reference attention patterns in out-of-distribution settings.

Via interventional experiment, we show that their Attention mechanism presents a key bottleneck in models' generalization capabilities.

BenCzechMark: A Czech-centric Multitask and Multimetric Benchmark for Large Language Models with Duel Scoring Mechanism

To foster research in lower-resource languages, we present BenCzechMark, the first comprehensive Czech language benchmark encompassing 50 challenging, culturally-dependent tasks from 8 categories. In addition, inspired by social preference theory, we propose and implement a new duel scoring system for language models that can provide more reliable assessment of models ranking.

Self-Training Language Models in Arithmetic Reasoning

Language models achieve impressive results in tasks involving complex multistep reasoning, but scaling these capabilities further traditionally requires expensive collection of more annotated data. In this work, we explore the potential of improving the capabilities of language models using automated feedback based on the validity of their predictions in arithmetic reasoning (self-training).

We find that models can substantially improve in both single-round (offline) and online self-training. In the offline setting, supervised methods are able to deliver gains comparable to preference optimization, but in online self-training, preference optimization shows to largely outperform supervised training thanks to superior stability and robustness on unseen types of problems.

Concept-aware Data Construction Improves In-context Learning of Language Models

Previous work curating in-context learning models assumes that ICL emerges from vast over-parametrization or the scale of multitask training. However, theoretical work attributes ICL emergence to specific properties of training data and creates functional in-context learners in small-scale, synthetic settings.

Inspired by previous theories, we propose a method to construct training data requiring analogical reasoning. We show the surprising efficiency of training on such data, with resulting models significantly outperforming classic instruction tuning in 41 and 45 out of 60 tasks, while performing on par with models trained on over 1600 tasks while using only two (2) QA tasks. Our analyses attribute these improvements to enhanced robustness of our models towards shortcuts of previous ICL models and improved ability to benefit from informative demonstrations.

Think Twice: Measuring the Efficiency of Eliminating Prediction Shortcuts of Question Answering Models

In this paper, we challenge the commonly-used evaluation of robustness of language models through the lens of their out-of-distribution performance. We propose a metric to measure the reliance of model's performance on a specific feature and measure the reliance on a set of features identified as prediction shortcuts.

We find that the same prediction shortcuts are exposed in all assessed QA datasets, whereas some shortcuts identified in SQuAD are even more impactful in other datasets. This means that counterintuitively, a model relying on shortcut more might reach better OOD performance. Finally, we survey three existing debiasing methods and find that indeed, the OOD gains of these methods come in hand with increased, rather than decreased reliance on prediction shortcuts.

Conference talk ⬇️️ & Announcement of results in NAACL DADC ⬇️️

Calc-X and Calcformers: Empowering Arithmetical Chain-of-Thought through Interaction with Symbolic Systems

Language models are notoriously inclined to make factual errors in tasks requiring arithmetic reasoning. To enable language models to circumvent this deficiency and offload a critical computation to a symbolic system, we create a collection of Calc-X datasets that demonstrate the appropriate use of a calculator in reasoning chains.

We survey and unify several existing chain-of-thought datasets into a proposed novel format, resulting in a standard collection of over 300,000 samples requiring arithmetic reasoning. Finally, we use the new collection to train open-source calculator-assisted language models and show that models trained on Calc-X almost double the accuracy of generating correct results compared to baselines, and can outperform larger models of previous work, such as Toolformer, or Llama 2. We make all Calc-X datasets and models publicly available.

Link to Marek's EMNLP talk 🎥

Can In-context Learners Learn a Reasoning Concept from Demonstrations?

Recent work analysing the functioning of in-context learners show that instead of learning new associations from the input, models largely rely on their pre-trained knowledge, such as the sentiment of the labels.

We argue that evaluations using a randomly-chosen demonstrations can not disentangle models' reliance on pre-trained knowledge from the ability to learn new functional relations, as most of such demonstrations can provide only limited information. Hence, we propose to evaluate in-context learners with demonstrations sharing a specific, informative reasoning concept with predicted sample. We find that most of the recent in-context learners can not consistently benefit from the demonstrated concepts, irrespective of the model size. However, some models are more sensitive to concepts than others, such as T0 models, which can benefit from concepts in 7 of 8 evaluation scenarios.

Soft Alignment Objectives for Robust Adaptation of Language Generation

Fine-tuning of pre-trained generative language models weakens their ability to generalize, making the open-ended deployment of these models prone to errors like hallucinations or degradations of output text quality.

In this work, we show that adapting the models with modeling the ambiguity of prediction can largely avoid over 95% of the loss caused by traditional fine-tuning.

Conference talk ⬇️️

Resources and Few-shot Learners for In-context Learning in Slavic Languages

In this work, we collect the infrastructure necessary for training and evaluation of ICL in a selection of Slavic languages: Czech, Polish, and Russian. We link a diverse set of datasets and cast these into a unified instructional format through a set of transformations and newly-crafted templates written purely in target languages.Using the newly-curated dataset, we evaluate a set of the most recent in-context learners and compare their results to the supervised baselines. Finally, we train, evaluate and publish a set of in-context learning models.

We find that the massive multitask training can be outperformed by single-task training in the target language, uncovering the potential for specializing in-context learners to the language(s) of their application.

Applications of deep language models for reflective writings

Social sciences exhibit many cognitively complex and highly qualified problems, whose resolution relies on often subjective expert judgements. One of such problems that we study in this work is a reflection analysis in the writings of student teachers.

We perform a variety of experiments on how to efficiently address data collection for applications exhibiting a great level of annotators' subjectivity. Additionally, we demonstrate a great potential in a cross-lingual transfer of multilingual language models in the analysis of reflective writing. We make our resulting datasets and models freely available.

Methods for Estimating and Improving Robustness of Language Models

Despite their outstanding performance, large language models (LLMs) suffer notorious flaws related to their preference for simple, surface-level textual relations over full semantic complexity of the problem. This proposal investigates a common denominator of this problem in their weak ability to generalise outside the training domain.

We survey diverse research directions providing estimations of model generalisation ability and find that incorporating some of these measures in the training objectives leads to enhanced distributional robustness of neural models. Based on these findings, we present future research directions towards enhancing the robustness of LLMs.

Conference talk ⬇️️

Adaptor: Objective-Centric Adaptation Library for Language Models

This paper introduces Adaptor library, transposing the traditional model-centric approach composed of pre-training + fine-tuning steps to objective-centric approach, composing the training from applications of selected objectives. We survey research directions that can benefit from enhanced objective-centric experimentation in multitask training, custom objectives development, dynamic training curricula, or domain adaptation and demonstrate the practical applicability of Adaptor in selected unsupervised domain adaptation scenarios.

Introduction talk ⬇️️

When FastText Pays Attention: Efficient Estimation of Word Representations using Constrained Positional Weighting

We propose a constrained positional model, which adapts the sparse attention mechanism from neural machine translation to improve the speed of the positional model. Our constrained model outperforms the positional model on language modelling and trains twice as fast.

RegEMT: Regressive Ensemble for Machine Translation Quality Evaluation

This work introduces a simple regressive ensemble for evaluating machine translation quality based on a set of novel and established metrics. We evaluate the ensemble using a correlation to expert-based MQM scores of the WMT 2021 Metrics workshop.

In both monolingual and zero-shot cross-lingual settings, we show a significant performance improvement over single metrics. In the cross-lingual settings, we also demonstrate that an ensemble approach is well-applicable to unseen languages. Furthermore, we identify a strong reference-free baseline that consistently outperforms the commonly-used BLEU and METEOR measures and significantly improves our ensemble’s performance.

Ensembling Ten Math Information Retrieval Systems: MIRMU and MSM @ ARQMath 2021

ARQMath Community Question Answering (CQA) competition challenges open-domain QA systems to find answers to a set of yet non-answered questions on Math Stack Exchange, potentially providing users with responses to all new but answerable questions.

In our submission, we create an ensemble of ten “weak” individual systems that we let vote to provide answers to unseen questions. Our submission, a best-performing among all the automated systems, shows that such ensemble approach can be more robust on unseen questions than single-model neural systems.

NLP Scientist - Team Lead

• Research and productisation of language technologies for diverse specialized applications, involving language generation, classification, entity recognition, search or recommendation.

• On-premise deployments of generative language models: compute optimisation, scalability, high availability assurance.

• Coordinating a diverse team of tenure NLP scientists, software developers, operations engineers, business developers, and copywriters.

• Communication with clients of diverse cultural backgrounds, public talks, and hands-on workshops at tech conferences and companies.



• We developed a paraphraser that can transcribe technical documentation to a more familiar language, enabling easier comprehension and searchability by visitors. Our solution enabled our customer to attract 12% more visitors to their websites.

• We delivered machine translation models specialized for chat conversations across languages of Southeast Asia. Our on-premise deployment enables our customer to save 10k+ USD per month while delivering translation quality comparable to Google Translate.

See our blogs, NLP case studies, HuggingFace models or open-source projects.