GenAI-1: What is a Language Model?

We all heard about LLMs (Large Language Models), didn’t we ? but before understanding Large Language Models What is Language Model ?

The language model is probability distribution over a sequence of tokens. The tokens can be anything like words, characters or subwords in language. What it mean by probability distribution over sequence of tokens. Let’s understand it with simple example. Consider the words ‘the’, ‘flat’, ‘rat’, ‘ran’, ‘on’. The language model would give higher probability to the more sensible sentence due to greater likelihood of real-world usage and logical coherence.

P(‘the’, ‘rat’, ‘ran’, ‘on’, ‘flat’) = 0.03 P(‘the’, ‘rat’, ‘on’, ‘flat’, ‘ran’) = 0.02 P(‘on’, ‘flat’, ‘the’, ‘rat’, ‘ran’) = 0.01

Language models are applied to a wide range of tasks, from classification to generation.

Now, its time to explore generative language models more. So, the next question is

What is Generative Language Models ?

Thank You 😊, fellow traveler, for joining us on this Que Ans journey.