View on GitHub
GitHub
Neural Networks: Zero to Hero
State of GPT | BRK216HFS
Loading player
Notes
Transcript
1133 segments
0:00
[MUSIC]
0:07
ANNOUNCER: Please welcome
0:08
AI researcher and
0:10
founding member of OpenAI, Andrej Karpathy.
0:21
ANDREJ KARPATHY: Hi, everyone. I'm happy
0:24
to be here to tell you about the state of
0:26
GPT and more generally about
0:28
the rapidly growing ecosystem of large language models.
0:31
I would like to partition the talk into two parts.
0:35
In the first part, I would like to tell you about
0:36
how we train GPT Assistance,
0:39
and then in the second part,
0:40
we're going to take a look at how we can use
0:42
these assistants effectively for your applications.
0:46
First, let's take a look at the emerging
0:48
recipe for how to train
0:49
these assistants and keep in mind that this is all
0:51
very new and still rapidly evolving,
0:53
but so far, the recipe looks something like this.
0:55
Now, this is a complicated slide,
0:57
I'm going to go through it piece by
0:59
piece, but roughly speaking,
1:01
we have four major stages, pretraining,
1:04
supervised finetuning, reward modeling,
1:06
reinforcement learning,
1:07
and they follow each other serially.
1:09
Now, in each stage,
1:11
we have a dataset that powers that stage.
1:14
We have an algorithm that for our purposes will be
1:17
a objective and over for training the neural network,
1:22
and then we have a resulting model,
1:23
and then there are some notes on the bottom.
1:25
The first stage we're going to start
1:27
with as the pretraining stage.
1:28
Now, this stage is special in this diagram,
1:31
and this diagram is not to scale because
1:33
this stage is where all of
1:34
the computational work basically happens.
1:36
This is 99 percent of the training
1:38
compute time and also flops.
1:41
This is where we are dealing with
1:44
Internet scale datasets with thousands
1:46
of GPUs in the supercomputer and
1:48
also months of training potentially.
1:51
The other three stages are finetuning
1:53
stages that are much more along
1:54
the lines of small few number of GPUs and hours or days.
1:59
Let's take a look at the pretraining stage
2:00
to achieve a base model.
2:03
First, we are going to gather a large amount of data.
2:07
Here's an example of what we call a
2:08
data mixture that comes from
2:10
this paper that was released by
2:13
Meta where they released this LLaMA based model.
2:16
Now, you can see roughly the datasets that
2:18
enter into these collections.
2:20
We have CommonCrawl, which is a web scrape, C4,
2:23
which is also CommonCrawl,
2:25
and then some high quality datasets as well.
2:27
For example, GitHub, Wikipedia,
2:29
Books, Archives, Stock Exchange and so on.
2:31
These are all mixed up together,
2:32
and then they are sampled
2:34
according to some given proportions,
2:36
and that forms the training set for the GPT.
2:40
Now before we can actually train on this data,
2:43
we need to go through one more preprocessing step,
2:45
and that is tokenization.
2:46
This is basically a translation of
2:48
the raw text that we scrape from the Internet into
2:51
sequences of integers because
2:53
that's the native representation
2:55
over which GPTs function.
2:57
Now, this is a lossless translation
3:00
between pieces of texts and tokens and integers,
3:03
and there are a number of algorithms for the stage.
3:05
Typically, for example, you could
3:07
use something like byte pair encoding,
3:08
which iteratively merges text chunks
3:11
and groups them into tokens.
3:13
Here, I'm showing some example chunks of these tokens,
3:16
and then this is the raw integer sequence
3:18
that will actually feed into a transformer.
3:21
Now, here I'm showing
3:23
two examples for hybrid parameters
3:26
that govern this stage.
3:28
GPT-4, we did not release
3:30
too much information about how it was trained and so on,
3:32
I'm using GPT-3s numbers,
3:33
but GPT-3 is of course a little bit old
3:35
by now, about three years ago.
3:37
But LLaMA is a fairly recent model from Meta.
3:40
These are roughly the orders of
3:42
magnitude that we're dealing
3:43
with when we're doing pretraining.
3:44
The vocabulary size is usually a couple 10,000 tokens.
3:48
The context length is usually something like 2,000,
3:50
4,000, or nowadays even 100,000,
3:53
and this governs the maximum number of integers that
3:56
the GPT will look at when it's trying to
3:58
predict the next integer in a sequence.
4:01
You can see that roughly the number of parameters say,
4:04
65 billion for LLaMA.
4:06
Now, even though LLaMA has only 65B parameters
4:08
compared to GPP-3s 175 billion parameters,
4:11
LLaMA is a significantly more powerful model,
4:13
and intuitively, that's because
4:15
the model is trained for significantly longer.
4:17
In this case, 1.4 trillion tokens,
4:19
instead of 300 billion tokens.
4:21
You shouldn't judge the power of a model by
4:23
the number of parameters that it contains.
4:26
Below, I'm showing some tables of
4:28
rough hyperparameters that typically
4:31
go into specifying the transformer neural network,
4:34
the number of heads,
4:34
the dimension size, number of layers,
4:36
and so on, and on the bottom
4:38
I'm showing some training hyperparameters.
4:41
For example, to train the 65B model,
4:44
Meta used 2,000 GPUs,
4:46
roughly 21 days of training and
4:48
a roughly several million dollars.
4:52
That's the rough orders of magnitude that you should have
4:54
in mind for the pre-training stage.
4:57
Now, when we're actually pre-training, what happens?
5:00
Roughly speaking, we are going to take our tokens,
5:03
and we're going to lay them out into data batches.
5:06
We have these arrays
5:07
that will feed into the transformer,
5:09
and these arrays are B,
5:10
the batch size and these are all independent examples
5:13
stocked up in rows and B by T,
5:16
T being the maximum context length.
5:17
In my picture I only have 10 the context lengths,
5:20
so this could be 2,000, 4,000, etc.
5:23
These are extremely long rows.
5:24
What we do is we take these documents,
5:26
and we pack them into rows,
5:28
and we delimit them with
5:29
these special end of texts tokens,
5:31
basically telling the transformer
5:32
where a new document begins.
5:35
Here, I have a few examples of documents and then
5:38
I stretch them out into this input.
5:41
Now, we're going to feed all of
5:43
these numbers into transformer.
5:46
Let me just focus on a single particular cell,
5:49
but the same thing will happen at
5:50
every cell in this diagram.
5:52
Let's look at the green cell.
5:54
The green cell is going to take
5:56
a look at all of the tokens before it,
5:58
so all of the tokens in yellow,
6:00
and we're going to feed that entire context
6:03
into the transforming neural network,
6:05
and the transformer is going to try to
6:07
predict the next token in
6:08
a sequence, in this case in red.
6:10
Now the transformer, I don't have
6:11
too much time to, unfortunately,
6:13
go into the full details of this
6:14
neural network architecture is
6:15
just a large blob of neural net stuff for our purposes,
6:18
and it's got several,
6:20
10 billion parameters typically or something like that.
6:22
Of course, as I tune these parameters,
6:23
you're getting slightly
6:24
different predicted distributions
6:26
for every single one of these cells.
6:28
For example, if our vocabulary size is 50,257 tokens,
6:34
then we're going to have that many
6:35
numbers because we need to
6:36
specify a probability
6:38
distribution for what comes next.
6:40
Basically, we have a probability for
6:41
whatever may follow.
6:42
Now, in this specific example,
6:44
for this specific cell,
6:45
513 will come next,
6:47
and so we can use this as
6:48
a source of supervision to
6:49
update our transformers weights.
6:51
We're applying this basically
6:53
on every single cell in the parallel,
6:54
and we keep swapping batches,
6:56
and we're trying to get the transformer to make
6:58
the correct predictions over what
6:59
token comes next in a sequence.
7:02
Let me show you more concretely what this looks
7:03
like when you train one of these models.
7:05
This is actually coming from the New York Times,
7:07
and they trained a small GPT on Shakespeare.
7:11
Here's a small snippet of Shakespeare,
7:12
and they train their GPT on it.
7:14
Now, in the beginning,
7:15
at initialization,
7:17
the GPT starts with completely random weights.
7:19
You're getting completely random outputs as well.
7:21
But over time, as you train the GPT longer and longer,
7:26
you are getting more and more coherent and
7:28
consistent samples from the model,
7:31
and the way you sample from it, of course,
7:32
is you predict what comes next,
7:35
you sample from that distribution and
7:36
you keep feeding that back into the process,
7:38
and you can basically sample large sequences.
7:42
By the end, you see that the transformer
7:43
has learned about words and
7:45
where to put spaces and where to put commas and so on.
7:48
We're making
7:48
more and more consistent predictions over time.
7:51
These are the plots that you are looking at
7:53
when you're doing model pretraining.
7:54
Effectively, we're looking at
7:56
the loss function over time as you train,
7:58
and low loss means that our transformer
8:00
is giving a higher probability
8:03
to the next correct integer in the sequence.
8:06
What are we going to do with model
8:08
once we've trained it after a month?
8:10
Well, the first thing that we noticed, we the field,
8:14
is that these models
8:16
basically in the process of language modeling,
8:18
learn very powerful general representations,
8:21
and it's possible to very efficiently fine tune them
8:23
for any arbitrary downstream tasks
8:25
you might be interested in.
8:26
As an example, if you're
8:27
interested in sentiment classification,
8:29
the approach used to be
8:31
that you collect a bunch of positives
8:33
and negatives and then you train some NLP model
8:35
for that, but the new approach is:
8:38
ignore sentiment
8:38
classification, go off and do large
8:41
language model pretraining,
8:43
train a large transformer,
8:44
and then you may only have a few examples and
8:47
you can very efficiently fine tune
8:48
your model for that task.
8:51
This works very well in practice.
8:53
The reason for this is that basically
8:55
the transformer is forced to
8:56
multitask a huge amount of
8:58
tasks in the language modeling task,
9:00
because in terms of predicting the next token,
9:03
it's forced to understand a lot about the structure of
9:05
the text and all the different concepts therein.
9:09
That was GPT-1. Now around the time of GPT-2,
9:12
people noticed that actually
9:14
even better than fine tuning,
9:15
you can actually prompt these models very effectively.
9:17
These are language models and they want
9:19
to complete documents,
9:20
you can actually trick them into performing
9:22
tasks by arranging these fake documents.
9:25
In this example, for example,
9:27
we have some passage and then we like do QA, QA, QA.
9:31
This is called Few-shot prompt, and then we do Q,
9:33
and then as the transformer is tried to
9:35
complete the document is actually answering our question.
9:37
This is an example of prompt engineering based model,
9:40
making it believe that it's imitating
9:42
a document and getting it to perform a task.
9:45
This kicked off, I think the era of, I would say,
9:48
prompting over fine tuning and seeing that this
9:50
actually can work extremely well on a lot of problems,
9:53
even without training any neural networks,
9:55
fine tuning or so on.
9:56
Now since then, we've seen
9:58
an entire evolutionary tree of
10:00
base models that everyone has trained.
10:02
Not all of these models are available.
10:05
for example, the GPT-4 base model was never released.
10:08
The GPT-4 model that you might be
10:09
interacting with over API is not a base model,
10:12
it's an assistant model,
10:13
and we're going to cover how to get those in a bit.
10:15
GPT-3 based model is available via the API under
10:19
the name Devanshi and GPT-2 based model
10:21
is available even as weights on our GitHub repo.
10:24
But currently the best available base model
10:27
probably is the LLaMA series from Meta,
10:29
although it is not commercially licensed.
10:32
Now, one thing
10:34
to point out is
10:35
base models are not assistants.
10:36
They don't want to make answers to your questions,
10:41
they want to complete documents.
10:43
If you tell them to write
10:44
a poem about the bread and cheese,
10:46
it will answer questions with more questions,
10:49
it's completing what it thinks is a document.
10:51
However, you can prompt them in a specific way for
10:54
base models that is more likely to work.
10:57
As an example, here's a poem about bread and cheese,
10:59
and in that case it will autocomplete correctly.
11:02
You can even trick base models into being assistants.
11:06
The way you would do this is you would create
11:08
a specific few-shot prompt
11:09
that makes it look like there's
11:11
some document between the human and assistant
11:13
and they're exchanging information.
11:16
Then at the bottom,
11:17
you put your query at the end and the base model
11:21
will condition itself into
11:23
being a helpful assistant and answer,
11:26
but this is not very reliable and doesn't work
11:28
super well in practice, although it can be done.
11:30
Instead, we have a different path to make
11:32
actual GPT assistants not base model document completers.
11:37
That takes us into supervised finetuning.
11:39
In the supervised finetuning stage,
11:41
we are going to collect
11:43
small but high quality data-sets, and in this case,
11:45
we're going to ask human contractors to gather data of
11:48
the form prompt and ideal response.
11:52
We're going to collect lots of these
11:54
typically tens of thousands or something like that.
11:56
Then we're going to still do language
11:58
modeling on this data.
11:59
Nothing changed algorithmically,
12:01
we're swapping out a training set.
12:02
It used to be Internet documents,
12:04
which has a high quantity local
12:06
for basically Q8 prompt response data.
12:11
That is low quantity, high quality.
12:13
We will still do language modeling
12:15
and then after training,
12:16
we get an SFT model.
12:18
You can actually deploy these models and
12:20
they are actual assistants and they work to some extent.
12:22
Let me show you what an
12:24
example demonstration might look like.
12:25
Here's something that a human contractor
12:27
might come up with.
12:28
Here's some random prompt. Can you
12:29
write a short introduction
12:31
about the relevance of
12:32
the term monopsony or something like that?
12:34
Then the contractor also writes out an ideal response.
12:37
When they write out these responses,
12:38
they are following extensive labeling
12:40
documentations and they are being
12:42
asked to be helpful, truthful, and harmless.
12:45
These labeling instructions here,
12:48
you probably can't read it, neither can I,
12:50
but they're long and this is people
12:52
following instructions and trying
12:53
to complete these prompts.
12:55
That's what the dataset looks like.
12:57
You can train these models. This works to some extent.
12:59
Now, you can actually continue the pipeline from
13:02
here on, and go into RLHF,
13:05
reinforcement learning from human feedback that
13:07
consists of both reward modeling
13:09
and reinforcement learning.
13:10
Let me cover that and then I'll
13:11
come back to why you may want to go through
13:13
the extra steps and how that compares to SFT models.
13:16
In the reward modeling step,
13:18
what we're going to do is we're now going to shift
13:20
our data collection to be of the form of comparisons.
13:23
Here's an example of what our dataset will look like.
13:25
I have the same identical prompt on the top,
13:28
which is asking the assistant to write
13:31
a program or a function that
13:32
checks if a given string is a palindrome.
13:35
Then what we do is we take the SFT model which
13:38
we've already trained and we create multiple completions.
13:41
In this case, we have three completions
13:42
that the model has created,
13:43
and then we ask people to rank these completions.
13:47
If you stare at this for a while, and by the way,
13:49
these are very difficult things to do to
13:51
compare some of these predictions.
13:52
This can take people even hours for
13:54
a single prompt completion pairs,
13:57
but let's say we decided that one of these is
14:00
much better than the others and so on. We rank them.
14:03
Then we can follow that with
14:04
something that looks very much like
14:06
a binary classification on
14:07
all the possible pairs between these completions.
14:10
What we do now is, we lay out our prompt in rows,
14:13
and the prompt is identical across all three rows here.
14:16
It's all the same prompt, but
14:17
the completion of this varies.
14:19
The yellow tokens are coming from the SFT model.
14:21
Then what we do is we append
14:23
another special reward readout token at
14:26
the end and we basically only
14:28
supervise the transformer at this single green token.
14:31
The transformer will predict some reward
14:34
for how good that completion is
14:36
for that prompt and basically it makes
14:39
a guess about the quality of each completion.
14:42
Then once it makes a guess for every one of them,
14:44
we also have the ground truth
14:46
which is telling us the ranking of them.
14:47
We can actually enforce that some of
14:50
these numbers should be much higher
14:51
than others, and so on.
14:52
We formulate this into a loss function and we
14:54
train our model to make reward predictions
14:56
that are consistent with the ground truth coming
14:58
from the comparisons from all these contractors.
15:01
That's how we train our reward model.
15:02
That allows us to score how good
15:04
a completion is for a prompt.
15:06
Once we have a reward model,
15:09
we can't deploy this because this is
15:11
not very useful as an assistant by itself,
15:13
but it's very useful for the reinforcement
15:15
learning stage that follows now.
15:16
Because we have a reward model,
15:18
we can score the quality of
15:19
any arbitrary completion for any given prompt.
15:22
What we do during reinforcement learning
15:24
is we basically get, again,
15:26
a large collection of prompts and now we do
15:28
reinforcement learning with respect to
15:29
the reward model. Here's what that looks like.
15:32
We take a single prompt,
15:34
we lay it out in rows,
15:35
and now we use basically
15:38
the model we'd like to train which
15:39
was initialized at SFT model
15:41
to create some completions in yellow,
15:43
and then we append the reward token again
15:45
and we read off the reward
15:47
according to the reward model,
15:49
which is now kept fixed.
15:50
It doesn't change any more. Now the reward model
15:53
tells us the quality of every single completion
15:55
for all these prompts and so what we can do is we can now
15:58
just basically apply the same
15:59
language modeling loss function,
16:01
but we're currently training on the yellow tokens,
16:04
and we are weighing
16:06
the language modeling objective
16:08
by the rewards indicated by the reward model.
16:11
As an example, in the first row,
16:13
the reward model said that this is
16:15
a fairly high-scoring completion
16:17
and so all the tokens that we
16:18
happen to sample on the first row are going to get
16:21
reinforced and they're going to get
16:22
higher probabilities for the future.
16:25
Conversely, on the second row,
16:26
the reward model really did not like
16:28
this completion, -1.2.
16:29
Therefore, every single token that we sampled in
16:32
that second row is going to get
16:34
a slightly higher probability for the future.
16:36
We do this over and over on
16:37
many prompts on many batches and basically,
16:39
we get a policy that creates yellow tokens here.
16:43
It's basically all the completions here will
16:46
score high according to
16:47
the reward model that we trained in the previous stage.
16:51
That's what the RLHF pipeline is.
16:55
Then at the end, you get a model that you could deploy.
16:58
As an example, ChatGPT is an RLHF model,
17:02
but some other models that you might
17:03
come across for example,
17:05
Vicuna-13B, and so on,
17:06
these are SFT models.
17:08
We have base models, SFT models, and RLHF models.
17:12
That's the state of things there.
17:14
Now why would you want to do RLHF?
17:16
One answer that's not
17:19
that exciting is that it works better.
17:20
This comes from the instruct GPT paper.
17:22
According to these experiments a while ago now,
17:25
these PPO models are RLHF.
17:28
We see that they are basically preferred in a lot
17:30
of comparisons when we give them to humans.
17:33
Humans prefer basically tokens
17:36
that come from RLHF models compared to SFT models,
17:39
compared to base model that is prompted to be
17:41
an assistant. It just works better.
17:43
But you might ask why does it work better?
17:47
I don't think that there's a single amazing answer
17:49
that the community has really agreed on,
17:51
but I will offer one reason potentially.
17:55
It has to do with the asymmetry between how easy
17:58
computationally it is to compare versus generate.
18:02
Let's take an example of generating a haiku.
18:04
Suppose I ask a model to write a haiku about paper clips.
18:07
If you're a contractor trying to train data,
18:10
then imagine being a contractor
18:12
collecting basically data for the SFT stage,
18:14
how are you supposed to create
18:15
a nice haiku for a paper clip?
18:16
You might not be very good at that,
18:18
but if I give you a few examples of
18:20
haikus you might be able to
18:21
appreciate some of these haikus a lot more than others.
18:24
Judging which one of these is good is a much easier task.
18:27
Basically, this asymmetry
18:29
makes it so that comparisons are
18:31
a better way to potentially leverage
18:33
yourself as a human and
18:34
your judgment to create a slightly better model.
18:37
Now, RLHF models are not
18:40
strictly an improvement on the base models in some cases.
18:43
In particular, we'd notice for example
18:45
that they lose some entropy.
18:46
That means that they give more peaky results.
18:49
They can output samples
18:54
with lower variation than the base model.
18:55
The base model has lots of entropy and
18:57
will give lots of diverse outputs.
19:00
For example, one place where I still
19:03
prefer to use a base model is in the setup
19:06
where you basically have
19:09
n things and you want to generate more things like it.
19:13
Here is an example that I just cooked up.
19:16
I want to generate cool Pokemon names.
19:18
I gave it seven Pokemon names and I asked the base model
19:21
to complete the document and it
19:22
gave me a lot more Pokemon names.
19:24
These are fictitious. I tried to look them up.
19:27
I don't believe they're actual Pokemons.
19:29
This is the task that I think the base model would be
19:31
good at because it still has lots of entropy.
19:33
It'll give you lots of diverse cool
19:35
more things that look like whatever you give it before.
19:41
Having said all that, these are
19:43
the assistant models that are probably
19:45
available to you at this point.
19:47
There was a team at Berkeley that ranked a lot of
19:50
the available assistant models
19:51
and give them basically Elo ratings.
19:53
Currently, some of the best models,
19:54
of course, are GPT-4,
19:55
by far, I would say,
19:57
followed by Claude,
19:58
GPT-3.5, and then a number of models,
20:00
some of these might be available as weights,
20:02
like Vicuna, Koala, etc.
20:04
The first three rows here are
20:07
all RLHF models and
20:09
all of the other models to my knowledge,
20:11
are SFT models, I believe.
20:15
That's how we train
20:17
these models on the high level.
20:19
Now I'm going to switch gears
20:20
and let's look at how we can
20:22
best apply the GPT assistant model to your problems.
20:26
Now, I would like to work
20:27
in setting of a concrete example.
20:29
Let's work with a concrete example here.
20:32
Let's say that you are working on
20:34
an article or a blog post,
20:35
and you're going to write this sentence at the end.
20:38
"California's population is 53
20:40
times that of Alaska." So for some reason,
20:42
you want to compare the populations of these two states.
20:44
Think about the rich internal monologue
20:47
and tool use and how much work
20:49
actually goes computationally in
20:50
your brain to generate this one final sentence.
20:53
Here's maybe what that could look like in your brain.
20:55
For this next step, let me blog on my blog,
20:59
let me compare these two populations.
21:01
First I'm going to obviously need to
21:03
get both of these populations.
21:05
Now, I know that I probably
21:06
don't know these populations off the top of
21:08
my head so I'm aware
21:10
of what I know or don't know of my self-knowledge.
21:12
I go, I do some tool use and I go to Wikipedia and I
21:16
look up California's population and Alaska's population.
21:19
Now, I know that I should divide the two, but again,
21:22
I know that dividing 39.2 by
21:24
0.74 is very unlikely to succeed.
21:26
That's not the thing that I can
21:28
do in my head and so therefore,
21:30
I'm going to rely on
21:31
the calculator so I'm going to use a calculator,
21:33
punch it in and see that the output is roughly 53.
21:36
Then maybe I do
21:38
some reflection and sanity checks in
21:40
my brain so does 53 makes sense?
21:42
Well, that's quite a large fraction,
21:44
but then California is the most
21:45
populous state, so maybe that looks okay.
21:47
Then I have all the information I might need,
21:50
and now I get to the creative portion of writing.
21:52
I might start to write something like "California has
21:55
53x times greater" and then I think to myself,
21:58
that's actually like really awkward phrasing so let me
22:00
actually delete that and let me try again.
22:03
As I'm writing, I have this separate process,
22:06
almost inspecting what I'm
22:07
writing and judging whether it looks good
22:09
or not and then maybe I delete and maybe I reframe it,
22:13
and then maybe I'm happy with what comes out.
22:15
Basically long story short,
22:17
a ton happens under the hood in terms of
22:19
your internal monologue when you
22:20
create sentences like this.
22:21
But what does a sentence like this look like
22:24
when we are training a GPT on it?
22:26
From GPT's perspective, this
22:28
is just a sequence of tokens.
22:30
GPT, when it's reading or generating these tokens,
22:34
it just goes chunk, chunk, chunk,
22:35
chunk and each chunk is roughly
22:37
the same amount of computational work for each token.
22:40
These transformers are not
22:42
very shallow networks they have
22:43
about 80 layers of reasoning,
22:45
but 80 is still not like too much.
22:47
This transformer is going to do its best to imitate,
22:51
but of course, the process here
22:53
looks very different from the process that you took.
22:56
In particular, in our final artifacts
22:59
in the data sets that we create,
23:00
and then eventually feed to
23:01
LLMs, all that internal dialogue was completely
23:03
stripped and unlike you,
23:07
the GPT will look at every single token and
23:09
spend the same amount of compute on every one of them.
23:12
So, you can't expect it
23:13
to do too much work per token and also in particular,
23:21
basically these transformers are
23:22
just like token simulators,
23:23
they don't know what they don't know.
23:26
They just imitate the next token.
23:27
They don't know what they're good at or not good at.
23:29
They just tried their best to imitate the next token.
23:32
They don't reflect in the loop.
23:34
They don't sanity check anything.
23:35
They don't correct their mistakes along the way.
23:37
By default, they just are sample token sequences.
23:40
They don't have separate inner monologue streams
23:43
in their head right? They're
23:43
evaluating what's happening.
23:45
Now, they do have some cognitive advantages,
23:48
I would say and that is that they do actually have
23:51
a very large fact-based knowledge
23:52
across a vast number of areas because they have,
23:55
say, several, 10 billion parameters.
23:57
That's a lot of storage for a lot of facts.
23:59
They also, I think have
24:02
a relatively large and perfect working memory.
24:04
Whatever fits into the context window
24:07
is immediately available to
24:09
the transformer through
24:10
its internal self attention mechanism
24:12
and so it's perfect memory,
24:14
but it's got a finite size,
24:16
but the transformer has a very direct access
24:18
to it and so it can a
24:19
losslessly remember anything that
24:22
is inside its context window.
24:23
This is how I would compare those two and the reason I
24:26
bring all of this up is because I
24:27
think to a large extent,
24:29
prompting is just making up for
24:31
this cognitive difference between
24:34
these two architectures like
24:37
our brains here and LLM brains.
24:39
You can look at it that way almost.
24:41
Here's one thing that people found for example
24:44
works pretty well in practice.
24:45
Especially if your tasks require reasoning,
24:48
you can't expect the transformer
24:49
to do too much reasoning per token.
24:52
You have to really spread out
24:53
the reasoning across more and more tokens.
24:56
For example, you can't give a transformer
24:57
a very complicated question and
24:59
expect it to get the answer in a single token.
25:00
There's just not enough time for it.
25:02
"These transformers need tokens to
25:04
think," I like to say sometimes.
25:06
This is some of the things that work well,
25:08
you may for example have a few-shot prompt that
25:10
shows the transformer that it should show
25:12
its work when it's answering
25:14
question and if you give a few examples,
25:17
the transformer will imitate that template and it
25:20
will just end up working out
25:21
better in terms of its evaluation.
25:24
Additionally, you can elicit this behavior from
25:26
the transformer by saying, let things step-by-step.
25:29
Because this conditions the transformer into showing
25:32
its work and because
25:34
it snaps into a mode of showing its work,
25:36
is going to do less computational work per token.
25:40
It's more likely to succeed as a result because it's
25:42
making slower reasoning over time.
25:46
Here's another example, this one
25:47
is called self-consistency.
25:49
We saw that we had the ability
25:51
to start writing and then if it didn't work out,
25:54
I can try again and I can try multiple times
25:56
and maybe select the one that worked best.
26:00
In these approaches,
26:02
you may sample not just once,
26:03
but you may sample multiple times and
26:05
then have some process for finding
26:07
the ones that are good and then keeping
26:09
just those samples or doing
26:10
a majority vote or something like that.
26:11
Basically these transformers in the process as
26:14
they predict the next token, just like you,
26:16
they can get unlucky
26:18
and they could sample a not a very good
26:19
token and they can go down like
26:21
a blind alley in terms of reasoning.
26:24
Unlike you, they cannot recover from that.
26:27
They are stuck with every single token they
26:28
sample and so they will continue the sequence,
26:31
even if they know
26:32
that this sequence is not going to work out.
26:34
Give them the ability to look back,
26:36
inspect or try to basically sample around it.
26:40
Here's one technique also,
26:43
it turns out that actually LLMs,
26:45
they know when they've screwed up,
26:47
so as an example, say you ask the model
26:50
to generate a poem that does not
26:52
rhyme and it might give you a poem,
26:54
but it actually rhymes.
26:55
But it turns out that especially for
26:57
the bigger models like GPT-4,
26:58
you can just ask it "did you meet the assignment?"
27:01
Actually GPT-4 knows very
27:03
well that it did not meet the assignment.
27:04
It just got unlucky in its sampling.
27:07
It will tell you, "No, I didn't actually meet
27:08
the assignment here. Let me try again."
27:10
But without you prompting
27:12
it it doesn't know to revisit and so on.
27:17
You have to make up for that in your prompts,
27:19
and you have to get it to check,
27:21
if you don't ask it to check,
27:23
its not going to check by itself
27:24
it's just a token simulator.
27:28
I think more generally,
27:29
a lot of these techniques fall into
27:31
the bucket of what I would say recreating our System 2.
27:34
You might be familiar with the System 1 and
27:36
System 2 thinking for humans.
27:37
System 1 is a fast automatic process and I
27:40
think corresponds to an LLM just sampling tokens.
27:43
System 2 is the slower deliberate
27:46
planning part of your brain.
27:49
This is a paper actually from
27:51
just last week because
27:52
this space is pretty quickly evolving,
27:53
it's called Tree of Thought.
27:56
The authors of this paper proposed maintaining
27:59
multiple completions for any given prompt
28:02
and then they are also scoring them along
28:04
the way and keeping the ones that
28:06
are going well if that makes sense.
28:08
A lot of people are really playing
28:10
around with prompt engineering
28:13
to basically bring back some of
28:15
these abilities that we have in our brain for LLMs.
28:19
Now, one thing I would like to note
28:21
here is that this is not just a prompt.
28:22
This is actually prompts that are together
28:25
used with some Python Glue code because you
28:28
actually have to maintain multiple
28:29
prompts and you also have to do
28:30
some tree search algorithm here
28:32
to figure out which prompts to expand, etc.
28:35
It's a symbiosis of Python Glue code and
28:38
individual prompts that are
28:39
called in a while loop or in a bigger algorithm.
28:42
I also think there's a really cool
28:43
parallel here to AlphaGo.
28:44
AlphaGo has a policy for
28:46
placing the next stone when it plays go,
28:48
and its policy was trained
28:50
originally by imitating humans.
28:52
But in addition to this policy,
28:54
it also does Monte Carlo Tree Search.
28:56
Basically, it will play out a number of possibilities in
28:59
its head and evaluate all of
29:00
them and only keep the ones that work well.
29:01
I think this is an equivalent of
29:04
AlphaGo but for text if that makes sense.
29:08
Just like Tree of Thought,
29:10
I think more generally people are
29:11
starting to really explore
29:13
more general techniques of not
29:15
just the simple question-answer prompts,
29:17
but something that looks a lot more like
29:19
Python Glue code stringing together many prompts.
29:22
On the right, I have an example from
29:23
this paper called React where they
29:25
structure the answer to a prompt
29:28
as a sequence of thought-action-observation,
29:32
thought-action-observation, and it's
29:34
a full rollout and
29:35
a thinking process to answer the query.
29:38
In these actions, the model is also allowed to tool use.
29:42
On the left, I have an example of AutoGPT.
29:45
Now AutoGPT by the way is
29:47
a project that I think got a lot of hype recently,
29:51
but I think I still find it inspirationally interesting.
29:55
It's a project that allows an LLM to keep
29:58
the task list and continue to
30:00
recursively break down tasks.
30:02
I don't think this currently works very well and I would
30:04
not advise people to use it in practical applications.
30:07
I just think it's something to generally take inspiration
30:09
from in terms of where this is going, I think over time.
30:12
That's like giving our model System 2 thinking.
30:16
The next thing I find interesting is,
30:19
this following serve I would say
30:20
almost psychological quirk of LLMs,
30:23
is that LLMs don't want to succeed,
30:26
they want to imitate.
30:28
You want to succeed, and you should ask for it.
30:31
What I mean by that is,
30:33
when transformers are trained,
30:35
they have training sets and there can be
30:38
an entire spectrum of
30:39
performance qualities in their training data.
30:41
For example, there could be some kind of a prompt
30:43
for some physics question or something like that,
30:45
and there could be
30:45
a student's solution that is completely wrong
30:47
but there can also be an expert
30:49
answer that is extremely right.
30:50
Transformers can't tell the difference between low,
30:54
they know about low-quality solutions
30:56
and high-quality solutions,
30:57
but by default, they want to imitate all of
30:59
it because they're just trained on language modeling.
31:02
At test time, you actually have
31:04
to ask for a good performance.
31:06
In this example in this paper,
31:08
they tried various prompts.
31:10
Let's think step-by-step was very powerful
31:13
because it spread out the reasoning over many tokens.
31:15
But what worked even better is,
31:17
let's work this out in a step-by-step way
31:19
to be sure we have the right answer.
31:20
It's like conditioning on getting the right answer,
31:23
and this actually makes the transformer work
31:25
better because the transformer doesn't have
31:27
to now hedge its probability mass
31:29
on low-quality solutions,
31:31
as ridiculous as that sounds.
31:33
Basically, feel free to ask for a strong solution.
31:37
Say something like, you are
31:38
a leading expert on this topic.
31:39
Pretend you have IQ 120, etc.
31:41
But don't try to ask for too much IQ because if
31:44
you ask for IQ 400,
31:46
you might be out of data distribution,
31:48
or even worse, you could be in data distribution for
31:51
something like sci-fi stuff and it
31:52
will start to take on some sci-fi,
31:54
or like roleplaying or something like that.
31:56
You have to find the right amount of IQ.
31:59
I think it's got some U-shaped curve there.
32:02
Next up, as we
32:04
saw when we are trying to solve problems,
32:07
we know what we are good at and what we're not good at,
32:09
and we lean on tools computationally.
32:12
You want to do the same potentially with your LLMs.
32:15
In particular, we may want to give
32:18
them calculators, code interpreters,
32:20
and so on, the ability to do search,
32:23
and there's a lot of techniques for doing that.
32:27
One thing to keep in mind, again,
32:28
is that these transformers by default may
32:30
not know what they don't know.
32:32
You may even want to tell the transformer in
32:34
a prompt you are not very good at mental arithmetic.
32:37
Whenever you need to do very large number addition,
32:40
multiplication, or whatever,
32:41
instead, use this calculator.
32:42
Here's how you use the calculator,
32:43
you use this token combination, etc.
32:46
You have to actually spell it out because the model by
32:48
default doesn't know what it's good at or not good at,
32:50
necessarily, just like you and I might be.
32:54
Next up, I think something that is very
32:56
interesting is we went from
32:58
a world that was retrieval only all the way,
33:02
the pendulum has swung to the other extreme
33:03
where its memory only in LLMs.
33:06
But actually, there's this entire space in-between of
33:08
these retrieval-augmented models and
33:10
this works extremely well in practice.
33:12
As I mentioned, the context window of
33:14
a transformer is its working memory.
33:17
If you can load the working memory
33:18
with any information that is relevant to the task,
33:21
the model will work extremely well
33:23
because it can immediately access all that memory.
33:26
I think a lot of people are really interested
33:28
in basically retrieval-augment degeneration.
33:32
On the bottom, I have an example of LlamaIndex which is
33:35
one data connector to lots of different types of data.
33:38
You can index all
33:41
of that data and you can make it accessible to LLMs.
33:44
The emerging recipe there is you take relevant documents,
33:47
you split them up into chunks,
33:49
you embed all of them,
33:50
and you basically get embedding vectors
33:52
that represent that data.
33:53
You store that in the vector store and then at test time,
33:56
you make some kind of a query to
33:57
your vector store and you fetch chunks that
34:00
might be relevant to your task and
34:01
you stuff them into the prompt and then you generate.
34:04
This can work quite well in practice.
34:06
This is, I think, similar to
34:07
when you and I solve problems.
34:09
You can do everything from your memory and
34:11
transformers have very large and extensive memory,
34:13
but also it really helps to
34:14
reference some primary documents.
34:17
Whenever you find yourself going
34:19
back to a textbook to find something,
34:21
or whenever you find yourself going back to
34:22
documentation of the library to look something up,
34:25
transformers definitely want to do that too.
34:27
You have some memory over how
34:30
some documentation of the library
34:31
works but it's much better to look it up.
34:33
The same applies here.
34:35
Next, I wanted to briefly talk
34:38
about constraint prompting.
34:39
I also find this very interesting.
34:41
This is basically techniques
34:43
for forcing a certain template in the outputs of LLMs.
34:50
Guidance is one example from Microsoft actually.
34:53
Here we are enforcing that
34:55
the output from the LLM will be JSON.
34:57
This will actually guarantee that
35:00
the output will take on this form because they go
35:02
in and they mess with the probabilities of
35:03
all the different tokens that
35:04
come out of the transformer and
35:05
they clamp those tokens and then
35:07
the transformer is only filling in the blanks here,
35:09
and then you can enforce additional restrictions
35:11
on what could go into those blanks.
35:13
This might be really helpful, and I think
35:15
this constraint sampling is also extremely interesting.
35:19
I also want to say
35:20
a few words about fine tuning.
35:22
It is the case that you can get really
35:23
far with prompt engineering,
35:25
but it's also possible to
35:27
think about fine tuning your models.
35:29
Now, fine tuning models means that you
35:31
are actually going to change the weights of the model.
35:33
It is becoming a lot more
35:35
accessible to do this in practice,
35:37
and that's because of
35:38
a number of techniques that have been
35:39
developed and have libraries for very recently.
35:43
So for example parameter efficient
35:44
fine tuning techniques like Laura,
35:46
make sure that you're only training small,
35:49
sparse pieces of your model.
35:51
So most of the model is kept clamped at
35:53
the base model and some pieces of it are allowed to
35:55
change and this still works pretty
35:56
well empirically and makes
35:58
it much cheaper to tune only small pieces of your model.
36:02
It also means that because most of your model is clamped,
36:05
you can use very low precision inference
36:07
for computing those parts because
36:09
you are not going to be updated by
36:10
gradient descent and so that
36:12
makes everything a lot more efficient as well.
36:13
And in addition, we have a number of
36:15
open source, high-quality base models.
36:17
Currently, as I mentioned,
36:18
I think LLaMa is quite nice,
36:20
although it is not commercially
36:21
licensed, I believe right now.
36:23
Some things to keep in mind is that basically
36:26
fine tuning is a lot more technically involved.
36:29
It requires a lot more, I think,
36:30
technical expertise to do right.
36:32
It requires human data contractors for
36:34
datasets and/or synthetic data pipelines
36:36
that can be pretty complicated.
36:38
This will definitely slow down
36:40
your iteration cycle by a lot,
36:41
and I would say on a high level SFT is
36:44
achievable because you're continuing
36:47
the language modeling task.
36:48
It's relatively straightforward, but RLHF,
36:50
I would say is very much research territory
36:53
and is even much harder to get to work,
36:55
and so I would probably not advise that someone
36:58
just tries to roll their own RLHF of implementation.
37:00
These things are pretty unstable,
37:02
very difficult to train, not something that is, I think,
37:04
very beginner friendly right now,
37:06
and it's also potentially likely also
37:08
to change pretty rapidly still.
37:11
So I think these are
37:12
my default recommendations right now.
37:15
I would break up your task into two major parts.
37:18
Number 1, achieve your top performance,
37:20
and Number 2, optimize your performance in that order.
37:23
Number 1, the best performance will
37:25
currently come from GPT-4 model.
37:27
It is the most capable of all by far.
37:29
Use prompts that are very detailed.
37:31
They have lots of task content,
37:33
relevant information and instructions.
37:36
Think along the lines of what would you tell
37:38
a task contractor if they can't email you back,
37:40
but then also keep in mind that a task contractor is a
37:43
human and they have
37:44
inner monologue and they're very clever, etc.
37:46
LLMs do not possess those qualities.
37:48
So make sure to think through
37:50
the psychology of the LLM
37:52
almost and cater prompts to that.
37:54
Retrieve and add any relevant context
37:57
and information to these prompts.
37:59
Basically refer to a lot of
38:01
the prompt engineering techniques.
38:02
Some of them I've highlighted in the slides above,
38:04
but also this is a very large space and I would
38:07
just advise you to look
38:09
for prompt engineering techniques online.
38:11
There's a lot to cover there.
38:13
Experiment with few-shot examples.
38:15
What this refers to is, you don't just want to tell,
38:17
you want to show whenever it's possible.
38:19
So give it examples of everything
38:21
that helps it really understand what you mean if you can.
38:25
Experiment with tools and plug-ins to
38:27
offload tasks that are difficult for LLMs natively,
38:30
and then think about not just a
38:32
single prompt and answer,
38:33
think about potential chains
38:34
and reflection and how you glue
38:36
them together and how you can
38:37
potentially make multiple samples and so on.
38:40
Finally, if you think you've squeezed
38:42
out prompt engineering,
38:43
which I think you should stick with for a while,
38:45
look at some potentially
38:48
fine tuning a model to your application,
38:51
but expect this to be a lot more
38:52
slower in the vault and then
38:54
there's an expert fragile research zone
38:56
here and I would say that is RLHF,
38:58
which currently does work a bit
39:00
better than SFT if you can get it to work.
39:02
But again, this is pretty involved, I would say.
39:05
And to optimize your costs,
39:06
try to explore lower capacity models
39:09
or shorter prompts and so on.
39:12
I also wanted to say a few words about the use cases
39:15
in which I think LLMs are currently well suited for.
39:18
In particular, note that there's a large number
39:20
of limitations to LLMs today,
39:22
and so I would keep that
39:24
definitely in mind for all of your applications.
39:26
Models, and this by the way could be an entire talk.
39:28
So I don't have time to cover it in full detail.
39:30
Models may be biased, they may fabricate,
39:32
hallucinate information,
39:33
they may have reasoning errors,
39:35
they may struggle in entire classes of applications,
39:38
they have knowledge cut-offs,
39:40
so they might not know any information above,
39:42
say, September, 2021.
39:43
They are susceptible to a large range of
39:45
attacks which are coming out on Twitter daily,
39:48
including prompt injection, jailbreak attacks,
39:51
data poisoning attacks and so on.
39:52
So my recommendation right now is
39:54
use LLMs in low-stakes applications.
39:57
Combine them always with human oversight.
40:00
Use them as a source of inspiration and
40:01
suggestions and think co-pilots,
40:04
instead of completely autonomous agents
40:05
that are just like performing a task somewhere.
40:07
It's just not clear that the models are there right now.
40:11
So I wanted to close by saying that
40:13
GPT-4 is an amazing artifact.
40:15
I'm very thankful that it exists, and it's beautiful.
40:18
It has a ton of knowledge across so many areas.
40:20
It can do math, code and so on.
40:22
And in addition, there's this
40:24
thriving ecosystem of everything
40:25
else that is being built and
40:27
incorporated into the ecosystem.
40:29
Some of these things I've talked about,
40:31
and all of this power is accessible at your fingertips.
40:35
So here's everything that's needed in terms of
40:37
code to ask GPT-4 a question,
40:40
to prompt it, and get a response.
40:42
In this case, I said,
40:44
can you say something to inspire
40:45
the audience of Microsoft Build 2023?
40:47
And I just punched this into Python and verbatim
40:50
GPT-4 said the following:
40:53
And by the way, I did not know that they
40:55
used this trick in the keynote.
40:57
So I thought I was being clever,
40:59
but it is really good at this.
41:02
It says, ladies and gentlemen,
41:04
innovators and trailblazers Microsoft Build 2023.
41:07
Welcome to the gathering of brilliant
41:08
minds like no other,
41:09
you are the architects of the future,
41:11
the visionaries molding the digital realm
41:13
in which humanity thrives.
41:15
Embrace the limitless possibilities of
41:17
technologies and let your ideas
41:18
soar as high as your imagination.
41:20
Together, let's create a more connected,
41:22
remarkable, and inclusive world for generations to come.
41:25
Get ready to unleash your creativity,
41:27
canvas the unknown, and turn dreams into reality.
41:30
Your journey begins today!