Estonian language in large language models

Value of the Information & Strength of the Argument

The presentation provides valuable insights into the state of Estonian in LLMs, backed by practical experience from the SLLM project. The argumentation is coherent and well-structured, moving from historical context to evaluation methods and then to the rationale for local development. The speaker effectively balances technical details with broader considerations, such as data rights and societal benefits. The use of concrete examples, like the Chatbot Arena and the prototype model, strengthens the credibility of the claims. However, some points, such as the effectiveness of continued training, are based on preliminary results and could benefit from more extensive evidence.

Scientific Rigor, Source Quality, Title Accuracy

The presentation demonstrates scientific rigor through the systematic overview of evaluation methods and the discussion of specific models and projects. The speaker references well-known models (GPT, BERT, Llama) and initiatives (Chatbot Arena), but does not provide formal citations or links to specific papers. The title accurately reflects the content, focusing on the status and development of Estonian in LLMs. The talk is based on the speaker’s expertise and ongoing research, which adds credibility, but it is not a peer-reviewed source. The Q&A session further clarifies technical aspects, showing transparency and depth of knowledge.

The title accurately reflects the content, focusing on the status and development of Estonian in large language models.

8/10

The presentation is given by a qualified associate professor in language technology, based on ongoing research and practical experience. It provides a balanced view of the challenges and approaches, with references to specific models and projects. However, it is an expert opinion rather than a peer-reviewed study, and some claims are not fully detailed.

Markers derived by PSI from the transcript: the creator did not define chapters.

• 00:00Introduction: Kairit Sirts introduces herself and the topic of Estonian in large language models.
• 01:00Timeline of language models: from GPT and BERT in 2018 to multilingual models and GPT-3.
• 03:00Discussion of GPT-3's closed nature and the emergence of ChatGPT in 2022.
• 05:00Introduction of Llama 2 and the SLLM project's goal to improve Estonian in open models.
• 07:00Two key questions: evaluating Estonian quality and whether local efforts are needed.
• 09:00Evaluation methods: subjective impression and automatic testing with examples.
• 12:00Crowd-sourced evaluation: Chatbot Arena and the Estonian variant Tehisaru.
• 15:00Results from Tehisaru: leaderboard and comparison of open models.
• 18:00LLM-as-a-judge method and its use in evaluation.
• 20:00Reasons for local development: data access, competence, and sensitive data handling.
• 22:00Dismissal of training an Estonian GPT from scratch and the SLLM project overview.
• 24:00Q&A: technical details on the prototype, model size, and licensing.

• Chatbot Arena — Mentioned as a platform for crowd-sourced evaluation of LLMs.
• Tehisaru — Estonian version of Chatbot Arena developed by the speaker's team.
• Llama 2 — Open-weight model family from Meta, used as a base for the SLLM project.

• Estonian Language Technology Programme — Funding source for the SLLM project, supporting language technology development in Estonia.