To Skip, to Swap or to not Swap? Identifying Step Transition Types in Instructional Manuals
Hsiu-Yu Yang,
Michael Roth,
Andreas Bulling,
Carina Silberer
Proc. International Conference on Language Resources and Evaluation (LREC),
2026.
Abstract
Links
BibTeX
Project
Large language models (LLMs) have been widely used as procedural planners, providing guidance across applications. However, in a human-assistive scenario where the environment and users’ knowledge constantly change, their ability to detect various step types for alternative plan generation is underexplored. To fill this gap, we introduce a novel evaluation task and dataset to assess if models can identify steps that are sequential, interchangeable, and optional in textual instructions across five domains in a step-by-step manner. We compare seven LLM families from both open-source and proprietary spaces across varying sizes to a visually-informed baseline based on procedural knowledge graphs (PKG). Our results suggest that LLMs encode procedural knowledge, enabling them to identify step types with increasing effectiveness as training parameters and data size grow. However, all LLMs exhibit inconsistencies in reasoning on the mutual exclusivity of interchangeable and sequential step pairs. In contrast, the symbolic PKG baseline offers an advantage here. Comprehensive analyses furthermore uncover limitations in LLMs’ procedural reasoning abilities.
@inproceedings{yang26_lrec,
title = {{To Skip, to Swap or to not Swap? Identifying Step Transition Types in Instructional Manuals}},
author = {Yang, Hsiu-Yu and Roth, Michael and Bulling, Andreas and Silberer, Carina},
year = {2026},
doi = {},
booktitle = {Proc. International Conference on Language Resources and Evaluation (LREC)}
}
