Dear viewer, these subtitles were generated by a machine via the service YouTube and therefore are (very) buggy. If you are capable, please help us to create good quality subtitles: https://c3subtitles.de/talk/3013 Thanks!
[Music]
thank you very much being here and I
hope that in the next hour uh or no half
hour um I'll answer the question how
many wind turbines do you need to power
a large language models and which tricks
exist if you want to run a really large
model on moderately priced um affordable
Hardware if there are pedantic people in
the audience a quick warning I use the
words deep learning machine learning
neural networks fairly interchangeably
in this talk what we will be talking
about is deep learning in the way it's
currently practiced the thing that you
often hear refer to as AI
and these systems are often used um not
as a solution to an existing problem but
more sort of as a thing in itself in
which case the answer to this question
would simply be use a more resource
efficient algorithm however that is also
not the focus of this talk so for the
rest of this we will assume that indeed
um deep learning is uh the best
algorithm for the problem that you are
interested
in so before we get into the details I
thought it would help to reiterate the
two main stages in the life cycle of a
deep learning model um because they're
quite different and that means that the
optimizations that you can do uh later
on are also quite a bit
different so the first stage is the
training stage and this is where where
we start from a blank slate and
basically try to get some insight on a
data set that we have at hand so what we
start with is a description of the
architecture which is basically just a
bunch of linear algebra formulas that
tell us how our parameters are meant to
interact
mathematically and then we have the
actual parameters which at this point is
really just a couple of megabytes to
gigabytes of more or less random
data
and then in order for this to become
useful uh what we have to do is carry
out a training procedure so we take our
data set and for each piece in this data
set we already know what we would like
the answer to be at the end so we feed a
piece of data from our training data set
through the model the model gives us an
answer which in the beginning is likely
not very useful however we can compare
that answer to what we would like the
answer to be and then we can go back and
adjust the parameters in the model very
slightly so that the answer that comes
out gets a little bit closer to what we
would like it to be and if we keep doing
that and keep feeding training data
through the network after a while
hopefully this converges to something
where the
parameters have basically absorbed some
statistical knowledge of our training
data set that then allows it to make
good
predictions um this is a relatively
textbook drawing what's usually missing
in these drawings is that in order for
this to work what you also need is uh
gpus and a lot of money so if you're
doing anything that is beyond basic hell
World type stuff you will typically need
several gpus and if we're talking about
large language models which is always
sort of the worst case example because
they are by far the most uh resource
hungry um we are talking about clusters
of very capable very expensive uh gpus
and this costs for large language models
at the moment somewhere in the tens of
millions of US dollars so even very
large companies don't do this training
process regularly but only every once in
a while so this is very resource
intensive however it also runs very
rarely um um another interesting
property of this procedure is that we
know that it works sort of as a sum of
its
parts but there is not really any
insight um into basically how the
parameters that come out of it interact
to give us the prediction in the end so
if we ask basically a computer scientist
for a rigorous explanation of what's
going on inside this model um you get
mainly just a confused
look now the second important stage is
the inference and this is where we want
to query our model to tell us something
about data that was not in the training
set and that we might be interested in
gaining some insights on so we take our
architecture and the optimized
parameters which constitutes our model
we feed in some previously unseen inputs
and usually what comes out is a
probability distribution of based on the
training set what the model thinks the
most likely answer um for this should
be and then from that we can just
generate a prediction of um yeah what
the model output thinks this uh should
be different from training this is a
very lightweight process um so even a
mediumsized large language model you can
typically run on a GPU for less than
,000 computer vision algorithms you can
often run on microcontrollers or your
phone so most of these models will run
fairly well on even moderately priced um
Hardware the important takeaway here is
that a model is always in one or the
other
state when we go to inference there is
not any live updating of parameters it's
completely Frozen at this point and that
means that we can do more fun
optimizations later
on before we get to the optimizations
what I want to talk about is the energy
consumption now what we have seen uh
since the discovery of deep learning is
that the complexity and the required
compute for these models um has just
been growing absolutely astronomically
and the reason for that is if you can
gather more data and build bigger models
and then train them for longer very
reliably what you will get is better
performance on tasks that are genuinely
interesting and for which there is no
other algorithm
available so here you can kind of see
this scaling up until 2010 roundabout uh
more or less doubling every two years in
line with Mo's law and then after the
discovery of deep learning it's a growth
of of 10x every two
years
so if you want to talk about energy I
thought it would be helpful to have
something um to compare it with and I
think a wind
turbine is an appropriate choice so this
particular one uh produces 12 and a half
gwatt hours on average per year which
can um Supply around 5,000 two person
households for training in a large
language model again large language
models usually the worst case example
you would need to spend between 400
megawatt hours and 1300 megawatt hours
for 200 billion parameter model which is
again at the at the upper end of what is
currently being used in practice and
that translates to 1 and a half to 5 and
a half weeks of the average uh wind
turbine output also tried to come up
with a number for computer vision models
um which are also quite popular and
quite a bit less resource intensive this
is again more the upper bound this is a
fairly large model and I would estimate
that this uh for training comes out to 9
and uh 9 megawatt hours of um energy
consumption just for the training which
is equivalent to 6 hours of wind turbine
output there is also another study um
that put these numbers and um basically
uh expressed them in terms of CO2
equivalent
emissions by also taking into account
the uh carbon intensity of the
electricity Grid in the places where
these models were trained this is also
the reasons they're all roughly the same
size Bloom is way down there because it
was trained in France where the
electricity grid um is less carbon
intensive and the other ones are I think
all of them trained in the US and
besides Bloom they all come to once or
eight up to eight times the life cycle
emissions of an internal combustion
engine car so it's quite a
bit um for inference numbers are even
harder to come by I Tred to put together
an estimate of the amount of energy that
CAD GPT consumes every day and again
nothing of this is really publicly
available so I had to work with
estimates of other
people and um what I come out to is
around 185 megawatt hours um or 5 and a
half days of uh wind turbine
operation so what does all the energy
that we're using actually go um if there
is a single person that is responsible
for this it would probably be this
man this is John Von noyman a famous
computer scientist and his I initial
idea was actually really good um what he
proposed we do is we treat program code
and data exactly the same put it all in
the memory have the computation separate
from that and then whenever any
computation needs to happen basically
what first needs to happen is that the
data moves from the memory through the
bus to the computation then we do the
computation and then the memory travels
back through the bus into the memory
there are two slightly different
definitions as to what the word for nman
bottleneck exactly means in the narrower
one um this means um basically that code
and data share the same bus whereas in
the broader definition which is the one
that I'll be using here it more or less
refers to just having this abstract
architecture where compute and memory
are two separate things
here's a picture of John V Norman
celebrating his
invention the most important information
on this slide is that the number on the
bottom is a lot higher than the number
on the top and in terms of energy the
number on the bottom refers to the
energy required to move data around
whereas the number on the top refers to
doing actual computations with it so
neural networks require a lot of
computation but that's not really the
issue the issue is more that they also
move a lot of data
around all right so now that we've found
out what the issue is namely that we
need to move too much data around in our
neural networks we can think about ways
of making them smaller and often energy
consumption isn't actually the main
concern
here um depending on the application
it's usually more about latency because
latency translates um into savings very
quickly but latency also aligns quite
nicely with energy
consumption which is
helpful so the first uh algorithm and
probably the most popular at the moment
is
quantization in scientific problems they
often have the issue that they don't do
particularly well with small errors in
computation and that means that they
usually require very high numerical
Precision neural networks are different
uh in that they are already more of an
approximately right kind of algorithm
and so they can do with a lot less
Precision which is why the training is
already typically done in floating Point
16 the reason for that is mainly that
this parameter updating procedure
requires quite a bit of precision to
work properly however if we think back
um training and INF are distinct and
that means that if we go from training
to inference we can actually squeeze it
down quite a bit more so in practice
what is often done is you take 16bit
numbers floating point and then convert
them to 8bit integer and that reduces
the accuracy a little bit or the
performance um but it's usually not too
much and what you get is just out of the
box a model that is half the
size the next technique is pruning
that's a little bit like playing yanga
Tower with a neural network basically
we're looking for the parts that are
redundant and then try to eliminate them
in order to end up with something that
looks a bit like a Swiss
cheese so if we look at a parameter
Matrix in a neural network we see that
it's very dense and very regular and
that's not really out of necessity
that's just what tends to work well with
the hardware that we have so it's a fair
assumption to think that not all of this
here is really required uh for good
performance so we can try and eliminate
some of it in order to make our model
smaller and in practice that actually
works quite well now remember we're
actually optimizing a couple of
gigabytes of just a big black box so
there is not really a way of telling
which parameters we need to set to zero
and usually there is some euristic that
needs to be used here also this in
itself doesn't really get you anything
you also need
um Hardware support for basically
compressing this so that you don't have
to move around the zeros and then
decompress it at
runtime but modern gpus uh have started
to add Hardware support for
this the next algorithm is the mixture
of experts this is a little bit like
these people that you sometimes
encounter in shopping areas and that
tell you that you're only using 10% of
your brain and then they want to sell
you poorly written science fiction books
this is basically that but done on
purpose so we are trying to use not all
of our neural network but only parts of
it and in order to do that
architecturally we split it up into a
gating function followed by several
subnetworks uh which are referred to as
the
experts and then at runtime when the
gating function sees an input it looks
at it and decides to which of the
experts this needs to be forwarded to
and then each of them generates an
answer and then usually that's just
averaged
together the nice property of this is
that for every input you get all the
expressivity of the huge parameter space
that all the experts have but you only
need to activate
and calculate the ones that are really
relevant to the final output um
according to this gating function this
is also quite a popular technique at the
moment um this is rumored to be used in
GPT 4 again none of this is publicly
confirmed and Mistral French large
language model startup a couple of weeks
ago also released a model that uses
exactly this
technique the last inference
optimization so so far we've only talked
about training models on data but that's
too boring we need something a little
bit more complicated so now what we're
going to do is train a model using
another
model in knowledge distillation what we
start out with is basically a large
model that is already very good at doing
a specific task however it might be too
large for the inference scenario um or
for the hardware that we want to deploy
it on so we need to shrink it down
somehow what we can do is take a smaller
model and then train it not on the
actual data set but on the output
probability distribution of the larger
model as far as I know there is not
really any rigorous explanation as to
why this works but the sort of handwavy
way of explaining it is that by seeing
the probability
distribution it gets access to more
knowledge about the training data set
that the teacher was able to acquire so
this tends to work a little bit better
than basically just taking the small
model and training it on your original
training data
set and then the smaller model we can
just deploy to our constrained Hardware
[Applause]
platform
um the last section in this for this
what I would like to do is go back to
the training stage and look at how
existing models can be adapted to a new
kind of application there are several
reasons why you would want to do that
for example you have private and
proprietary data that you don't want to
upload to a cloud but also you don't
have an HP PC cluster at home or you
want to have a small model that is
specialized to do a specific task and
not a large model that does everything
but it's also very expensive to run or
you might be interested in doing a task
um that is not covered well by the
Baseline model that you are starting out
with so far all of the techniques that I
have been talking about are
relatively broadly applicable and this
is the point where we have to diverge a
little bit and talk about different uh
styles of model architecture so transfer
learning is something that works more
for more old-fashioned computer vision
type algorithms because these
convolutional neural networks have this
nice property that they start getting
more abstract the more you move uh
through the network so they start with
basically just detecting
shapes and edges and then colors and
then it's more abstract shapes and only
at the very end you have something that
tells you okay this is actually what you
fed in is an image of a car or a ship or
a flower so assume that we've trained a
model like that that's really good at
detecting
pictures um but really what we're
interested in is sea creatures which
unfortunately weren't in the original
data set and we only have a small data
set of SE creatures so what we can do is
just take out this last part of the
network that does this translation from
abstract feature to basically a
descriptive label telling you what's in
the picture and we can replace that with
a new random
layer and that we can then train and
because it's a lot fewer parameters than
where um than the entire thing this
converges much quicker and using less
data and we get to reuse most of the
featur detection stuff that's in the
earlier
layers this does not really work um for
larger architectures that are popular
now so large language models diffusion
models um they don't have this nice
property that when you move through the
network you build up
abstraction and um that's typically at
the moment where something called low
rank adaptation is being used so what we
do for this is we take our original
model we freeze all the
parameters but we also add basically
like a detour path on the side um that
the input also takes and then the
results of these two paths are added
together um low rank is basically just a
fancy linear algebra way of saying that
the path on the right has a lot fewer
parameters which means that we can train
it on smaller
hardware and also using less data and
then basically the learn parameters that
are on the site work as basically an
offset to what the original model would
predict and these uh in the end can be
merged into the original parameter space
or they can also be kept separately so
you can also have basically several
models that all have their own extra set
of small parameters and that can do
different
things all right um so with all these
fancy techniques that we've talked about
did we actually manage to keep up with a
10x scaling every two years and the
answer as you probably have guessed by
now is uh no we have not the main
mechanism that enables this scaling is
more Capital invested into larger
clusters of more powerful
gpus and clever engineering and mo law
can offset that to some point but
certainly not all of
it so if we wonder and I think the
previous talk was quite a nice um quite
a nice introduction to this if we think
about does higher efficiency actually
translate into reduced energy
consumption um this handsome young man
would probably tell us that the answer
is no if we could ask him which we can't
because he's been dead for 150 years but
what he was dis what he discovered and
what was later named after him the Jans
Paradox is that after James wat had made
the steam engine more efficient what
happened is that coal consumption
actually went up
afterwards and this is because if
something becomes cheaper it becomes
more abundant and more of it is being
used and that can lead to the effect
that in the end more of the original
resource is being used than was
previously before was made more
efficient I think we're seeing the same
thing with cars where car manufacturers
have been building more and more
efficient engines in the last years and
now just everyone has an SUV or a Ford
F-150 and I would argue that we are
seeing the same Trend uh with machine
Learning Systems in general and I don't
see any indication that this is going to
change anytime in the near
future so to summarize um I think
looking at the already high energy
consumption for large scale machine
learning systems at the moment and then
also having in the back of your head
that they are growing at the moment at
10x every 2 years I think is a very
scary Prospect and I think it's good um
I think it's good advice to at least be
worried about that and
be be careful
um when looking at those things what I
would also like to add is that
nevertheless this is also an extremely
Dynamic field which changes super
quickly and that means that
extrapolating a current Trend into the
future has a lot of
uncertainty attached to it so anything
that is more than a couple of into the
future I think it becomes very quickly
equivalent to looking into a crystal
ball the other point that I was trying
to make here and that hopefully I
succeeded in making is that some aspects
of these systems are very much in the
hands of only the largest of
organizations however there are other
aspects that are um I think somewhat
surprisingly uh fairly accessible even
to
people with relatively modest
resources and that's the end of my
presentation and we have plenty of time
for questions I
[Applause]
think okay you know the drill we have
four microphones in the room two here
and two over there and also we have the
ability to ask questions on the internet
and we will indeed start with the
internet
now there is question could low range
adaptation instead of adapting to new
data points be used to cancel out parts
of the training set for example to
remove training data to comply with gdpr
requests
um yes to a point so again there is not
really any insight into what exactly
this process does so there is nothing
that guarantees you that after you've
done this you've actually canceled out
every bit of data that you are
interested in what you can use it is to
basically steer the model in a different
direction similar to reinforcement
learning from Human feedback you can use
this as an offset to what the model
would normally spit out but there is not
really any strict guarantee that every
piece of data that might be problematic
in your original data set that uh is
actually is actually gone because you
don't really know where it is in the
parameters in the first
place okay microphone number three
please hi uh thanks for the talk uh so
my question is you mentioned the
transfer learning is not really gener
generalizable for uh language models
right uh but what is it what what's the
fineing then I thought it's the same
thing maybe you can explain it in a few
words
um yeah again the as far as I'm aware
the term fine tuning is a little bit so
there is some overlap and there is not
really a strict definition fine tuning
is more you retrain all the parameters
but at a very low learning rate and
basically that means that most of what
is already in the network you can reuse
and only adapt it a tiny bit um transfer
learning is sort of a more crude version
of that where you keep most of it frozen
and just replace
parts of it
um does that answer the
question microphone number
one hi thanks for the talk so um you
mentioned quantization and also pruning
to reduce the network size um so what's
your take on um making neural networks
more explainable by reducing the
size um
I think that's a very important
field I think if you can reduce it so
much that there are so few parameters
that it becomes explainable I think you
lose a lot of thec capacity that makes
these systems popular and I think there
is not really
anything to the best of my knowledge
that was as yet able to bridge that Gap
so either you have models that are so
small that they are
explainable but can't really do anything
too interesting or you have these huge
gigaby sized models that do cool stuff
and no one knows why and um I think
that's very much an open question as far
as I'm
aware microphone number two um we have
seen with Bitcoin that this kind of
energy consumption can goes uh that can
go out of hand quite quickly so do you
think this is even a technical problem
or is it more like of a regulation
problem
um I think it's both yeah I actually
tried to find numbers on the energy
consumption of the Bitcoin network but
gave up at some point
um I think it it has to go hand in hand
I think there need to be Technical
Solutions and I'm sure there is a lot of
clever engineering that can still be
done to make this more resource
efficient what I also think is that this
is not going to happen if the politic or
if there is no no pressure to do that so
the political landscape
also is in charge of creating the
conditions where that kind of innovation
can happen and is being adopted because
it translates into money savings at the
end of the
day all right microphone number four
please yes so uh this is uh perhaps more
of a technical question uh when you are
talking about in this talk you were
mainly talking about gpus what I'm
curious about is uh CPUs because the the
situation with the memory is different
it's not exactly linear CU if you get it
small enough you might be able to fit it
into some kind of cash right and then
you will get really good uh efficiency
improvements has there been any work
done on like shrinking them down past
some critical
point um there is some work on running
these models on CPUs and for inference
that that works reasonably well um again
there is not
really there is not really sort of this
this dependency of having a larger model
which gets you better performance hasn't
really been resolved yet so in the end
the model that has more parameters
basically always wins at the end of the
day um but gpus have more and more
memory on board these days so there are
basically it's just a compute core in an
in an en Normy of high band with memory
so you could argue that basically the
cach is also be being there but it's on
the GPU now I think it's it's a similar
idea um but at the end of the day it all
comes down to the problem of no one
really knowing how to make these models
smaller and still have them give good
performance thank you all right let's
hear from the
internet internet is uh wondering if uh
you have any thoughts on using brain
inspired Computing Technologies uh for
power efficient neural
networks um yeah I originally planned to
have uh also a Sci-Fi section in this T
which I had to um take out at the end
but um yes there are there is quite a
bit of research going in the direction
of having architectures that are not at
all inspired by this for noyman model
but that somehow have the computation
collocated with the
memory um the problem with these is
generally that we have 50 plus years of
experience in making transistors really
really small and these more um on paper
more efficient architectures we don't
really know how to make them to the size
that we can make transistors and so
that's why sort of the mainstream
research still happens in these um
relatively crude approximations of what
and neuron in the human brain do and
there are models that basically try to
match that a little bit more also in
terms of Energy Efficiency but it's not
easy to build them efficiently in
Hardware All Right microphone number two
please thank you for a very nice talk I
believe people are having problem with
choosing the foundational models so my
question is related to well there are
multiple Frameworks that are related to
evaluating them large language models
for the performance have you seen any
evaluations in the form of the energy
consumptions like the capabilities per
what like you
know um not that I'm aware of no I think
it would be an interesting interesting
um yeah interesting thing to to look for
but no not aware of
anything I think they're also mostly
secret right all right uh microphone
number four uh I had two quick questions
uh the first one regarding the mixture
model that you showed uh choosing
between experts I was thinking if you
can localize uh in the weights where the
color recognition happens or motion
recognition happens couldn't you just
upload those specific weights before
training so the training happens faster
that that way you can save energy
another one was could you send the gpus
to space maybe cuz there's no problem
like solar energy in in space is much
better than here atmosphere dissipation
and uh the heat does not cause
atmospheric problems so would be good
training okay so the first question was
um whether you can basically already
make the decision about the experts at
the training time but then you lose this
property that in the end you have a
large parameter space so how it works is
that the network looks at an input and
for large language
models the intuitive explanation is that
the gating function decides the next
output is going to be a verb or a noun
or punctuation and then it's going to
forward that only to the verb or the
noun or the punctuation expert so this
is and it still needs to learn all of
the
dependencies um the way it's efficient
is that during runtime it can then
select which parts it thinks are more
relevant than others but you still need
all the different parts to get you good
performance and the second question was
whether you can send gpus to space I
think I mean there nothing strictly
saying that you can't I think youd have
to radiation Harden them quite a bit and
as I said um I think the the issue here
is often not as much energy that's just
a nice side
effect what companies generally do care
about a lot more is throughput just
because a smaller model runs faster on
the same Hardware or runs the same speed
at cheaper hardware and that saves you a
lot of money whereas sending things to
space solves the energy problem but it's
a lot more costly and that's why no one
really no one really looks into that I
guess all right uh let's go back to the
internet we are we doomed to use big
Tech AI Services forever or are there
any hopes that smaller local mod models
will ever be competitive
um so in terms of the
training I don't really see it happening
anytime soon that uh regular people with
sort of modest budget will be able to
train models from scratch um unless some
big breakthrough happens at some
point what is happening quite a bit is
that big companies train sort of
foundation models and then don't give
you the training algorithm but it least
the weight so that people can go off and
train them on their own data which is
not an ideal situation but it's also
better than nothing and there is quite a
lot of
experimentation happening in that field
but I think for for training from
scratch it's not looking great at the
moment all right Mike two please hi uh I
was wondering one thing is that uh so
basically last year n actually pop paper
that that allows uh a new algorithm that
can actually Trend Quan models on micro
controllers so I was assumed that maybe
in the future if we instead of training
a huge models in high uh HPC centers
what if we train like 1 million smaller
models on 1 million
microcontroller uh instead of training a
huge models in a know HPC do you thing
that would be more uh energy efficient
and are there any people really do the
evaluation on this thank
you
um I'm wondering I don't have a good
answer what I'm wondering is if we if we
train because like these these HPC
systems are relatively efficient if we
now basically do the same amount of
training but not in HPC but on a
microcontroller
and do that a thousand thousands or
millions of times um whether that not at
the end of the day consumes more
energy and not less
um um yeah but um I'm
not I don't know don't have a good
answer to that all right number one but
make it quick okay thanks for the great
talk hi um I was wondering about the
mixture models and whether that the idea
can be extended in a hierarchical manner
like with divide and conquer where you
could go multiple layers down and then
even need on to compute whatever 100K
parameters or so which would allow it to
go on embedding or
something more more tree likee structure
yeah it's an interesting idea um I mean
this idea is is already fairly old I
think it's like the original paper is 25
30 years
old um
I'm not an expert on model architecture
I know that the big issue that these
things have is that they that it takes a
lot of work to convince the model to not
just converge to always using the same
um expert and ignoring all the others
and then that also gets all the training
I would imagine that this problem gets
even worse if you start stacking
this all right and we will end the talk
perfectly on time please thank Alias
again
[Applause]
[Music]
yeah