LLM based Speech to Speech Translation (S2ST) System – An Overview

This talk presents a technical overview of Large Language Model (LLM)-based Speech-to-Speech Translation (S2ST) systems, exploring how they move beyond traditional cascaded pipelines. We discuss the role of self-supervised feature extraction and the adoption of decoder-only transformer architectures for unified speech understanding and generation. A central theme is the use of Chain of Thought (CoT) reasoning to decompose the translation task into interpretable sub-steps. We also cover how Residual Vector Quantization (RVQ) codes enable efficient discrete audio representations with disentangled semantic and acoustic content, and how it is used as LLM generation target. The talk concludes with practical considerations around data preparation, model training, and inference, highlighting how these design choices come together to support real-time, controllable, and high-fidelity speech translation.