Bigram language models

# makemore is a character-level language model
# so it models the given sequence of characters, e.g. one example is given as: alyssa, there are 32k such examples
# then it can predict the next character in the sequence

# we will go from character-level(bigram), to word-level(bag of words), mlp, rnn, gru, finally modern transformers(on which gpt-2 is based on)

# first, let's load up the dataset "names.txt"
# open("names.txt", "r").readlines()[5:20]
# .read(5) reads the first 5 characters, not the first 5 names

# a character level language model is predicting the next character in a sequence given already some concrete sequence of characters before it
# for eg. if there is a word named isabella, it says that the character 'i' is very likely to occur first in the new name
# the character 's' is likely to come after i
# the character 'a' is likely to come after is, and so on
# and the last piece of information: after the sequence becomes 'isabella', the word is likely to end

# in a bigram language model, we are always working with only 2 characters at a time
# we are looking only at the one character we are given, and are trying to predict the next character in the sequence
# eg. what characters are likely to follow 'r', or likely to follow 'a', and so on.
# we are just modeling this little structure
# atleast for now, we are ignoring the fact that we have more information through the entire word
# HERE: WE ARE JUST LOOKING AT THE PREVIOUS CHARACTER TO PREDICT THE NEXT ONE
# and this is a very simple and weak language model

# now, let's look at what bigrams in our dataset look like
# bigrams: just 2 characters in a row

# continue from 8:14

# so, e is likely to come first, and a is likely to come last
# we are going to create a special array of characters, and we're going to hallucinate a special start token here, then the word, then the special end

# TO LEARN THE STATISTICS ABOUT WHICH CHARACTERS ARE MORE LIKELY TO FOLLOW OTHER CHARACTERS,
# THE SIMPLEST WAY IN THE BIGRAM LANGUAGE MODELS IS TO SIMPLY DO IT BY COUNTING
# SO WE'RE JUST GOING TO COUNT HOW OFTEN ANY ONE OF THESE COMBINATIONS OCCURS IN THE TRAINING SET
# so we need a dictionary to maintain some counts for every one of these bigrams, let's use a dictionary b
# bigram is a tuple of character 1, character 2

# note that for these 3 words, a was always the ending character everytime so the count is 3 which is what we want to see

# b.items() returns the tuples of key-value
# in this case, the keys are character bigrams and values are the counts
# we want to sort, but the default sort is on the first item of the tuple, but we want to sort by count here which is the second element of the tuple
# so we added key = lambda that takes the key value and returns keyvalue at [1] instead of default [0]
# and we want to do it in descending order by using -ve sign

# it will be convenient for us to keep this information in a 2d-array(table with proper organization of proper rows and columns and a value in a cell) instead of a python dictionary
# rows: first char of bigram, columns: second char of bigram
# each entry tells us how often the second character follows the first character in the dataset
# the array representation / library we are going to use: pytorch
# pytorch: deep learning neural network framework
# but part of it is also torch.tensor that allows us to create multidimensional arrays and manipulate them efficiently

# the problem now is that we have characters a-z, <S> and <E> and they need to be indexed using integers
# so we need some lookup table from characters to integers

# N
# let's visualize it nicely

# NumPy:      N[row, column] -> N[i, j]
# Matplotlib: (x, y)         -> (column, row) -> (j, i)

# upon careful scrutinization, we are not being very clever
# look at the last row and second last column, it is completely filled with zero's, and a 2x2 matrix at the bottom right too(only <S><E> is valid there for empty word)
# that's cuz <E> will never be the first character of a bigram, and <S> will never be the second character of a bigram
# we are using brackets <> cuz it is a convention to do so in NLP

# now, we won't use 2 special tokens. We're only going to use one special token, so N array would be 27x27 instead

# note: first row is the start letter one, i.e. .a, .b, it is equivalent to <S>a, <S>b, etc
# note: first column is the end letter one, i.e. a., b., it is equivalent to a<E>, b<E>, etc

# to sample the first character of the name, we look at the first row, .a, .b.
# don't get confused .a doesn't mean it ends at a.
# here .a means <S>a, so it means, how likely is a to be the first character of a word

# now, we want to sample from these counts
# since we are given raw counts, we have to convert them into probabilities

# now we can try to sample from this distribution
# to sample from this distribution, we are going to use torch.multinomial
# it means: you can give me probabilities and i will give you integers which are sampled according to the probability distribution

# suppose: P = torch.tensor([0.50, 0.30, 0.15, 0.05])
# here, a is most likely the next character, d is the least likely, but any of them can still happen
# A probability distribution tells you how likely each outcome is.
# sampling means: Randomly pick one outcome, but make the randomness follow the given probabilities.
# it's not choosing the largest probability, it's not choosing uniformly
# we let the probabilities themselves determine how likely each outcome is to be picked

# torch.multinomial
# P = torch.tensor([0.50, 0.30, 0.15, 0.05])
# torch.multinomial(P, num_samples=1)
# Here are the probabilities. Randomly choose one index according to them.
# it might return tensor[0] or tensor[1] or tensor[2] or even tensor[3]
# For one sample, anything can happen.
# but if we sample many times
# for _ in range(10):
    # print(torch.multinomial(P, 1).item())
# we will notice that in one run, [0] appears most and [3] appears least
# and why're we doing this? cuz we want it to feel natural
# we don't want every single word to start with the same letter with the highest probability

# prob for first sample, i.e. 0: 61%, for second sample: 30%, for third sample: 9%
# the more we sample, the more accurate it should be

# getting back to topic

# note: we are working on N[0], so we are working to find which character might be the first character to our word

# we got the first character, now we have to draw the next one
# we drew e, so now look at the row e and find the character that is likely to occur after e
# now we write a loop

# the reason these samples are so terrible is that bigram language model is just terrible
# but they are name like a little bit but not name like enough
# but think from the model's perspective, eg. you see x., the model doesn't know that x is the first x of the word. it is just thinking about one character at a time
# so it directly also looks for how likely is x the last character
# to convince yourself that this is not so terrible is by : instead of relying on p(that came from our dataset of 32k words),
# if p was just a uniform probability distribution of all 27 characters making any character to appear equally likely

# saw the output from a completely untrained model? saw how random it could have actually been?

# now, let's fix an efficiency issue
# we are always fetching a row of N from the count matrix up, then converting to float, and then dividing
# this happens in every iteration of the loop
# and we're renormalizing these rows again and again
# so let's prepare a new matrix P based on existing matrix N, but P will have probabilities 
# i.e. same as N, but instead of values, we set probabilities (value/sum of that row) normalized to one
# so it indicates the probability distribution directly for the next character(column) given the character before(row)

# we're doing this not just for efficiency, but also to practice n-dimensional tensors and practice their manipulation
# practice broadcasting
# if we're going to build up to transformers, we have to be good in tensor manipulations, complicated array operations for efficiency

# P.sum()
# We want to divide all the rows value's by the respective sum of the row
# looking documentation for torch.sum
# so, we provide the array we want to sum, and it's dimensions
# but not just that, be careful of keep dimension attribute
# if keepdim = True, output tensor is of the same size as input, except in the dimension along you summed which will become 1
# but if keepdim = False, this dimension is squeezed out, so torch.sum does the sum and also collapses dimension to be of size one, 
# but addition it also does the squeeze, it squeezes out that dimension

# we don't want 1x27 row vector, we don't want the sum of counts across a column
# we want to sum of counts across a row, so we want a 27x1 row vector, and each entry gives corresponding sum of that row

# note that shape is 27 x 1, so it's a column vector, we're going horizontally
# buut, notice that these counts are identical no matter whether we sum across rows or columns
# that's cuz this array of counts comes from bigram statistics and by coincidence or by the way this array is constructed, the sums along rows and columns are identical

# is it possible to take a 27x27 array and divide by 27 x 1 array?
# P / P.sum(1, keepdim=True)
# 27 x 27
# -------
# 27 x  1
# whether or not we can perform this operation is determined by broadcasting rules (explore this in pytorch)
# from documentation, there are a few conditions:
# 1. each tensor has atleast 1 dimension
# 2. when iterating over the dimension sizes, starting at the trailing(later) dimension, the dimension sizes must either be equal, one of them is 1, or one of them doesn't exist.

# so we align them like this
# 27 x 27
# -------
# 27 x  1
# move from right to left
# from the right, they are not equal, but one of them is a 1, so it is fine, and in the other dimension, both are equal (27)
# since they are fine, the operation P = P / P.sum(1, keepdim=True) is broadcastable/allowed
# so what happens internally is that, the dimension 1 is stretched out, it is copied to match 27
# so our column vector(that has the sum) is copied 27 times for each one of those 27 columns(. - z), and internally it becomes 27/27
# finally element wise division is done
# thus this action shall normalize every single row

# now, let's understand where broadcasting could go wrong
# also lookup some tutorials on broadcasting and practice it, cuz you can run into bugs very quickly

# now suppose we erase keepdim=True, and by default it's false
# what happens then? well it just squeezes the dimension 1 out turning it into 1 d array

# you might think that P / P.sum(1) might work as expected, but it doesn't
# let's check if it is broadcastable

# P / P.sum(1)
# 27 x 27
# -------
#      27
# rule 1: align all dimensions on the right
# move from right to left
# either equal, one of them should be 1, or one of them shouldn't exist
# equal, and doesn't exist
# so internally, broadcasting will create a 1 in front, and it will get copied and will broadcast
# 27 x 27
# -------
#  1 x 27
# this won't work and you'll get garbage result

# so why was the keepdim necessary?
# earlier when keepdim was used, the 1-d vector(column) was copied across each column properly, 
# so eg. for a value in fourth row and first column, row[3][0], it was divided by sum of the fourth row[3]
# but now, when keepdim is absent, the 1-d vector(column) was copied across each row instead
# so eg. for a value in fourth row and first column, row[3][0], it was divided by sum of the first row[0]

# so first column was divided by sum of first row, instead of first row being divided by sum of first row and so on
# so first row was divided as: first entry divided by sum of first row ok, but second entry of first row divided by sum of second row, 3rd entry of first row divided by sum of third row and so on.

# in short, from broadcasting rules, a dimension is silently added to the left which caused this issue, so work with it carefully
# what was supposed to be a column vector becomes a row vector instead and it works in opposite manner instead

# we trained a bigram language model just by counting how frequently any pairing occurs and then normalizing so that we can get a nice probability distribution
# the elements of array P are the parameters of our bigram language model giving and summarizing the statistics of these bigrams
# so we trained the model and now we know how to sample from the model
# we just iteratively sample the next character and feed it in each time and get the next character

# now, we want to evaluate the quality of this model
# we want to somehow summarize the model's quality into a single number
# how good is it at predicting the training set, eg. we can evaluate the training loss from the training set
# this training loss tells us the quality of the model in a single number

# side note: if we were to sum all probability values in the matrix P, would we get 27? cuz sum for one row would be 1, and we have 27 rows

# from this graph, we can see the log is a monotonic transformation of probability
# if we pass in probability as x = 1, we get 0
# and as we go lower, i.e. probability or x < 1, we get the log's value as more and more negative until -infinity is reached at x = 0

# the only reason we are doing this stuff with log is for convenience
# keep in mind: we are trying to compute product of all the probabilities, i.e. ..*.a*...*aa*ab*ac...*zz
# note: log(a*b*c) = log(a) + log(b) + log(c)
# since likelihood = product of probabilities, 
# log likelihood = log(product of probabilities) = sum of logs of individual probabilities
# so log likelihood starts at 0, and it accumulates overtime by logs of all individual probabilities

# so how high can log likelihood get? it can go to zero
# this happens when all probabilities are 1, and since log(1) = 0, log(1) + log(1) + ...(for all probabilities) = 0
# but when all the probabilities are very low instead of 1, our log probability gets very very negative, and we don't like this
# but why don't we like this? cuz we'd like to implement a loss function: and loss function has the semantics that low is good (which is opposite of our case) because we're always trying to minimize the loss
# so we'd like to invert this from current situation where 0(desired) and -infinity(undesired)
# to invert it, we do the negative log likelihood: which is simply the negative of log likelihood

# and one final modification people like to do for convenience is instead of just the sum of nll, people compute the average of nll

# so the loss function assigned by our model: the average negative log likelihood is: 2.9981: the quality of our model
# lower the anll better our model and vice versa
# and now, the job of our training is to find the parameters that minimize the negative log likelihood

# SUMMARY:
# GOAL: MAXIMIZE LIKELIHOOD(product of all probabilities) OF DATA W.R.T MODEL PARAMETERS (statistical modeling) (defined in the matrix)
# equivalent to maximizing the log likelihood (because log is monotonic)
# equivalent to minimizing the negative log likelihood
# equivalent to minimizing the average negative log likelihood (guess we can't say normalized since it isn't in range 0-1.)

# currently we are storing all probabilities in a table format, but what's coming up later is that these numbers won't be kept explicitly but be calculated by a neural network

# acc to our model, hx has a 0% probability, so log(0) = -inf, and our anll = inf
# this is unreasonable, cuz we plugged a string that would be somewhat reasonable but it shouldn't be 0% likely
# there's a very simple fix to smooth out our model: model smoothing
# we will add fake counts: eg. add a count of 1 to everything
# the more we add, the more smoother it gets, and we get a more smoother uniform model
# 1 is a decent count to add which will ensure that there will be no 0s in our probability matrix P, this can change our generations by a little bit
# so it is unlikely to get an infinity

# so until now, we've trained a respectable bigram character level language model
# and we've trained the model by looking at the counts of all the bigrams and normalizing the rows to get probability distributions
# we can also use the parameters(probability values inside matrix P) of this model to perform sampling of new words according to the probability distribution
# and we can evaluate the quality of this model: summarized in a single number: negative log likelihood, and the lower this loss, the better our model is 
# because it is giving high probabilities to the actual next characters in all bigrams in our training set
# but what we did was just counts and normalizing these counts
# but now we want to take an alternative approach and end up in a very similar position
# the approach is very different cuz we would like to cast the problem of bigram character level language modeling into the neural network framework
# and in the neural network framework, we're going to approach things differently but end up in a similar spot

# our neural network is still going to be a bigram character level language model
# so it receives a single character as input
# and there's a neural network with some parameters
# and it will output the probability distribution over the next character in a sequence, 
# it will make guesses as to what is likely to follow the char that was input to this model
# ADDITIONALLY, we will be able to evaluate any setting of the parameters of the neural net cuz we have the loss function: the negative log likelihood
# so we'll look at the probability distributions and then look at the labels (identity of next character in that bigram)
# knowing the second character in the bigram allows us to look at how high of a probaility the model assigns to that character
# ofcourse, we want the probability to be very high, ie. loss is low
# then we're going to use gradient-based optimization to tune the parameters of this network cuz we have the loss function and we'll minimize it
# so we'll tune the weights so that neural net correctly predicts the probabilities for the next character

# so, when input is 0, we want the weights to be arranged so that 5 gets a high probability, similar for 5-13, 13-13, 13-1, and 1-0

# caution: be careful of APIs of some frameworks, eg. we used torch.tensor, but there is also a torch.Tensor which is very very similar too
# the official documentation doesn't distinguish much, but in stackoverflow there was a thread that distinguishes the difference

# difference:
# torch.tensor: infers the dtype automatically, also has argument dtype if we want to change it, so stick with this instead(recommended)
# torch.Tensor: returns torch.FloatTensor

# now, how will we feed these examples xs, ys into neural network?
# right now, it's not straightforward, cuz these examples are integers that are index of characters(., a, ..., z), and we can't just plug integer index into neural network
# neural networks has neurons, neurons have weights and they act multiplicatively on the input: wx + b, tanh, so it doesn't make sense to use these 0-26 int directly
# instead, a common way of encoding integers: one-hot encoding: 
# we take an integer eg.13 and create a vector with all zeros except for the 13th dimension(which will be 1), and we feed it into a neural net

# pytorch has a one hot function inside torch.nn.functional that takes a tensor made up of integers
# and it also takes input as num of classes: how long you want your vector to be. in our case 27 = 26(a-z) + 1(.)
# not specifying num_classes might give incorrect result: might take largest value in xs, eg. 13 instead

# understanding this graph: try to read ., e, m, m, a

# we evaluate 5 activations of this neuron on these 5 inputs all in parallel
# we fed 5 inputs into the same neuron simultaneously, and in parallel, pytorch has evaluated wx for all of them
# now instead of a single neuron, we want to have 27 neurons

# so by using matrix multiplication, we can very efficiently evaluate the dot product between lots of input examples in a batch

# so what we did until now was, we fed our 27-dimensional inputs into a first layer of a neural net that has 27 neurons
# i.e. 27 inputs and 27 neurons, these neurons perform wx, they don't have a bias and they don't have a non-linearity like tanh
# we'll leave this layer to be linear
# additionally, we're not going to have any other layers. This is going to be it, the dumbest, smallest, simplest neural net: just a single linear layer

# now we want to explain what we want those 27 outputs to be
# for every input example, we're trying to produce a probability distribution for the next character in the sequence
# we see that some nums are -ve, some are +ve as they are coming out of torch.randn() : normal distribution parameters
# but what we want is something like in previous matrix, where each row told us the counts which we normalized to get probability distributions
# we want something similar to come out of a neural net: but what we have now is some +ve and -ve nums
# probabilities have a structure: are +ve, and sum to 1
# the output of our neural net can't be counts: they are integers and positive, not a good thing to output from a neural net
# so what to output? and how to interpret the 27 numbers: the 27 nums are giving us the log counts instead of giving counts directly like in previous matrix

# to get the counts from the log counts, we have to exponentiate them
# for exponentiation: if x = -ve, fx is always < 1. 
# if x = +ve, fx > 1

# note that all -ve nums turned to 0-1 and all +ve nums are >1.
# these exponentiated outputs can be interpreted (unlike prev -ve, +ve nums) like counts in our earlier matrix example
# so these values are >0 and take values based on settings of W. hmm nice

# what did we achieve?
# note: by 5 examples: we mean 1 word emma: eg 1: .e, eg2: em, eg3: mm, eg4: ma, eg5: a.
# for every single one of our 5 examples, we have a row that came out of a neural net, and because of the transformations, we ensure the outputs are probabilities

# logits = xenc @ W #wx
# counts = logits.exp()
# probs = counts / counts.sum(1, keepdims=True)

# all three operations above are differentiable, and we can backpropagate through them, and we get probability distributions

# for the zeroth example, where we fed in .e of emma. we input a . and expect a e as output with high probability
# we first had one_hot vector of zeroth example, i.e. we fed the dot into a neural net
# it went into the neural net and produced a probability distribution: probs[0]

# continue from 1:26:24

# SUMMARY---------------------------------------------------------------------------------------------------------------------------------------

# FORWARD PASS

# softmax: often used layer in neural net that takes input z(a vector of logits, i.e. raw neural network outputs) into a probability distribution
# It exponentiates each logit and then normalizes by dividing by the sum
# of all exponentiated values.
# Input: any real numbers (positive, negative, or zero).
# Before normalization:
#   - negative logits become numbers between 0 and 1 after exponentiation
#   - zero becomes 1
#   - positive logits become numbers greater than 1
# Output:
#   - all values are positive
#   - all values sum to 1
#   - each value represents the probability of that class

# Softmax is commonly used as the final layer of a classifier to convert
# logits into class probabilities.

# Note: that in the above forward pass, everything is made up of differentiable layers
# we can backpropagate through everything there, eg. matrix multiplication is just multiplication and addition, exponentiation(can be backpropagated through)
# also sum and division are backpropagatable

# we have a loss function: nlls.mean().item()
# and this is made up of only differentiable operations
# and we can minimize loss but tuning the W(weights) by computing the gradients of the loss w.r.t W matrices, i.e. gradient-based optimization

# what we'll do is very similar to what we did in micrograd
# in micrograd we had four input examples and their targets
# xs = [
#     [2.0, 3.0, -1.0],
#     [3.0, -1.0, 0.5],
#     [0.5, 1.0, 1.0],
#     [1.0, 1.0, -1.0]
# ]
# ys = [1.0, -1.0, -1.0, 1.0] # desired targets,
# but our input has xs as tensor([0, 5, 13, 13, 1]) which can be converted to one_hot and be 5x27 just like xs above
# but we won't have a desired ys, 
# instead we will try to minimize our loss upto the point that it is similar to the loss computed from earlier method 1: table approach

# in micrograd, our neural net was n(x) which was a multi layer perceptron
# say we want input layer with 3 neurons, 2 hidden layers each having four neurons and an output unit

# x = [2.0, 3.0, -1.0]
# n = MLP(3, [4, 4, 1])# we want 3 inputs into 2 layers of 4 and one output
# n(x)

# but our neural net is just a single linear layer of matrix multiplication(wx) which is followed by a softmax
# our neural network: logits = xenc @ W, followed by 2 lines of softmax

# in micrograd, loss was mean square error, we just subtracted prediction from ground truth, then squared it and summed it all up
# ypred = [n(x) for x in xs]
# loss = sum([(ygt - yout)**2 for ygt, yout in zip(ys, ypred)])
# here, loss=single num that represented the quality of neural net, and when loss is low, it showed that the neural net was predicting correctly
# then we iterated over all parameters, and set gradient to zero
# and called loss.backward
# backward pass
    # for p in n.parameters():
    #     p.grad = 0.0
    # loss.backward() #initiated backpropagation at final output node of loss, i.e. just filled the gradients that were recently reset to 0
# and during backpropagation, we updated/nudged all the parameters slightly in the negative direction of gradient, i.e. tuning weights, i.e. training
# update: gradient descent
    # for p in n.parameters():
    #     p.data += (-0.05) * p.grad

# we're going to do the same thing here too
# here, probs is our ypred
# to evaluate loss: we don't use MSE, we have negative log likelihood cuz in this case we're doing classification(not regression like in micrograd)

# so we plucked out the probabilities assigned by the neural network to the correct next characters for 5 bigrams we're working with

# note: this loss matches the loss we obtained above(but this is a vectorized form)
# average negative log likelihood, i.e. loss= 3.7693049907684326

# so, we defined our forward pass: we have our probs(ypred) and loss
# now, we do backward pass

# backward pass
# 1. make sure all gradients are reset: 0
# in pytorch, we can set all gradients to zero or None
# setting gradients to None: more efficient as it is interpreted as lack of gradient, and same as zeros too
# then we do loss.backward, but remember that pytorch requires we pass in requires_grad=True while generating the random weights initially

# in our forward pass above, pytorch actually keeps track of all the operations we did under the hood and builds a full computational graph
# those graphs exist inside pytorch, it knows all the dependencies and all the mathematical operations
# this loss.backward() just fills in the gradients of all the intermediates all the way back to W(parameters of our neural net)

# so, W[0,0] =  0.0121 is +ve
# it tells us that slightly increasing it will increase the loss: eg. W[0,0] += h

# run cell of forward pass then backward pass again and again manually to see loss decreasing, or you could just use loop

# getting lower loss: equivalent to: neural network is assigning high probabilities to the correct next characters

# rearranged everything and put together------------------------FROM START-------------------------------------------------------

# what number do we expect to be our desired loss?
# we expect something we got originally when we had optimized just by counting, our loss was roughly 2.47 after smoothing
# getting approximately 2.46 is the goal, and not down to 0.0003, but why?
# because we're not taking any additional information, we're just taking the previous character and trying to predict the next one
# but instead of trying to do it by explicit counting and normalizing, we're doing it with gradient based learning
# this gradient based approach is way more flexible and scalable, we can now expand this approach and complexify the neural net
# currently we're just taking 1 char and predicting next char, but we can substantially/largely iterate on this
# eg. we take multiple previous characters instead of just 1, and feed it into very complex neural nets too
# but fundamentally, the output of the neural nets will just be logits, and the logits will go through the exact same transformation
# we'll take it through softmax, calculate the loss function and the negative log likelihood, and do gradient-based optimization

# as we complexify the neural nets and work all the way upto transformers, the structure is the same fundamentally
# the only thing that will change/be more complex is the way we,
# the way we do the forward pass where we take the previous chars to calculate the logits for the next character

# in our initial table based approach, it is not scalable, there'll be way too many combinations of what previous characters could be
# it works for 1char input, 1 char output, but doesn't work well for 10char input 1 char output, it was an unscalable approach
# but our neural network approach is very scalable and flexible too

# ahh ahh, remember that in the table method case where table=N, we used the first character to index the row of the array
# that row gave us the probability distribution for the next character
# i.e. first character was used as a lookup into matrix N to get probability distribution
# but what happens in neural network case is very similar too

# we take the index by encoding xs as one-hot and multiplying it by W
# then logits becomes the row of W  
# logits then gets exponentiated to create the counts then gets normalized and becomes probability
# so, W is almost the same as N, but W is the log counts, not the counts itself
# thus we can say W.exp = N table matrix
# N was filled by populating the counts of bigram
# but in gradient based framework, we initialized W randomly, then we let the loss guide us to arrive at the exact same array

# qn: in this simple case we got loss from table case and used that to get loss for neural net.
# but what about other complex cases where we might not construct the table based on counts?

# remember the smoothing in N case where we added fake count 1 to every entry to prevent probability from being 0 to any bigram
# but if we increase the count, probability becomes more and more uniform

# and interestingly, our gradient-based optimization of neural network has an equivalent to smoothing
# We initialized W randomly right? but what if we initialized W to be 0?
# if all entries of W are zero, then logits will be all 0 since logits = xenc @ W
# then counts = exponentiating logits becomes all 1
# finally probability will become exactly uniform
# thus when values in W are all equal to each other or all 0, then probabilities come out completely uniform

# IMPORTANT NOTE:
# trying to incentivize W's entries to be near zero's = equivalent to = label smoothing, more the smooth distribution

# SO, THIS BRINGS US TO

# we can like actually add this to our actual loss computation
# regularization strength * regularization loss
# this optimization has 2 components now
# 1. it's trying to work out all the probabilities
# 2. the additional component + 0.01 * ((W)**2).mean() tries to make all W's be zero
# cuz if w's are non-zero, there is high loss, minimizing the loss is trying to make W zero
# adding this: + 0.01 * ((W)**2).mean(), is like pushing or adding a gravity force to make W 0 to reduce the loss
# so W want's to be 0 and the probabilities want to be uniform but simultaneously they also want to match up the probabilities as indicated by the data

# this strength of regularization controls the amount of counts we add eg.1 or 100, in P = (P+1).float()

# IMPORTANT
# adding more counts in count based prediction corresponds to increasing regularization strength in neural nets approach

# result kinda similar: earlier we used to get cexze for table based too, but now after 1 day getting cava idk
# but no worry its due to randomness and sampling and the results don't have to be same too, just similar
# both the models are identical

# SUMMARY:
# WE INTRODUCED THE BIGRAM CHARACTER-LEVEL LANGUAGE MODEL
# WE LEARNT HOW TO TRAIN THE MODEL(based on count/sum and for neural net too)
# WE LEARNT HOW TO SAMPLE FROM THE MODEL
# WE LEARNT HOW TO EVALUATE THE QUALITY OF THE MODEL USING NEGATIVE LOG LIKELIHOOD LOSS
# WE TRAINED THE MODEL IN 2 DIFFERENT WAYS THAT GIVE THE SAME RESULT AND THE SAME MODEL

# WAY 1:
# COUNTED THE FREQUENCY OF ALL BIGRAMS AND JUST NORMALIZED
# WAY 2:
# USE NLL LOSS AS GUIDE TO OPTIMIZE COUNT ARRAY TO MINIMIZE LOSS IN GRADIENT BASED FRAMEWORK

# OBSERVED: BOTH GAVE SAME RESULT
# WAY 2: VERY FLEXIBLE, currently: our neural network is very simple
# we take 1 char, we take 1 char through single linear layer to calculate LOGITS
# we will COMPLEXIFY, take more inputs than just 1 and feed them into the neural net
# the neural net's output will be the same logits which will be normalized in the same way, 
# the process of loss computation, gradient based framework will all be identical
# but more optimized and better output will be received
# and the neural net will complexify to transformers