Dear viewer, these subtitles were generated by a machine via the service YouTube and therefore are (very) buggy. If you are capable, please help us to create good quality subtitles: https://c3subtitles.de/talk/2710 Thanks!
[Music]
give a warm welcome to
Redford QC
[Music]
okay and Mr
[Applause]
[Music]
trick and it's an honor to announce the
talk breaking DRM in Polish trains
reverse engineering a train to analyze a
suspicious
malfunction
[Applause]
[Music]
[Applause]
hi I'm Redford this is kufri K and Mr TI
not
trick and we'll talk today about trains
we'll do a quick intro tell the story
and then we'll go into technical details
so uh we sometimes play together CFS
with dragon sector and Poland can into
space I work for invisible things lab uh
I mostly do lowlevel security and
reverse
engineering and let's and they they will
introduce themselves in a few slides
let's start with the story as you
already know the story is about
trains and the story actually starts a
long time ago in 2016 when Kon it's a
local polish train operator bought 11
impulse trains which one of them is on
the
photo then after some time the trains
started reaching 1 million kilomet on
the
autometers and by this uh amount you
have to do a uh a big
maintenance and because the
manufacturer's warranty already expired
they started a tender to select the best
offer for Servicing
and the offer was won by SPS it's an
independent train Workshop in
Poland and in the first quarter of 2022
the first train reached the
workshop
so let's see at the at least the public
timeline the servicing started with
uh the number 24
train uh the workshop
to took apart the whole train sent the
the parts to the manufacturers and then
assembled the TR train back and but the
problem was that the train didn't start
afterwards then they took another train
for servicing and it was the same the
trains didn't want to start after
servicing and what's even more
interesting is that in the meantime
another Workshop started Servicing
trains for a different train operator
and they run into exactly the same
problem so it's it's getting a bit
suspicious uh and the story got noticed
by media in Poland because you had like
less trains running so the manufacturer
issued a public press release and they
said that uh among many other
accusations they said that someone
interfered with the security system
whatever that
is and something happened in between and
the the workshop asps started returning
the trains which work so what
happened and what happened in the
meantime uh after the the workshop got
into
troubles the issues didn't look like
normal issues because the the computer
was saying that everything is fine and
they had so
uh pointers into direction of the
manufacturers involvement so but they
didn't have any idea what to do so they
Googled polish hackers and found
us so we we got in contact we got the
trains but that about that later uh in
August we managed to unlock the first
train and by a few months months later
we gathered enough evidence to notice
authorities about this and that's what
we will talk about
today all right I think it's my turn so
hi I'm Mr tick known in Poland as span
clash in Germany as her he been he been
a engrosser ban fan so I want to briefly
introduce
you I want to walk you through some
initial terms here so before I tell you
how to unlock a train uh let's define
what a locked train is so uh we have
basically a train you enter a cabin all
the system reports that the train is
ready to roll there's this device
combined throttle and brake lever so you
push it forward the train releases all
the brakes and then it should accelerate
but it doesn't we have H oh that's
it the
brakes nothing
happens you can you can see that zero on
the
screen so we had a Lo train the workshop
bought additional two CPUs of
the very small yep that's one of them
and we got access to all service and
maintenance documentation for the trains
and yep we started so uh a small detour
to actually see what a train is this is
a very simplified diagram of how most of
Moder trains work nowadays so at the top
we have this three kvdc that's the
Polish Uh current source for trains this
cable that goes above our heads uh the
green box is the PLC the CPU that
controls all behavior of the train then
we have the white or greenish power
converter that actually converts this
high voltage for all devices and power
all smaller devices there uh at the
lower part of diagram we have the stuff
that is responsible to slow down the
train so we have compressor that creates
a pressurized air a lot of systems in a
train works on a pressurized air like
brakes for example more on that later
and in top right Corner we have inverter
and a motor inverter is used to convert
the 3 Kilts to a current that motor can
accept accelerate not burn and Y that's
the thing that makes a nice sound when
you start a train everything is
interconnected with a canabas about
which q3k tell you
more okay I will hi I'm q3k you might
know me from such [ __ ] posts are get as
getting whand to run on iPod Nano but
today I'm not going talk about anything
about that instead let's talk about can
now can is an automotive protocol but
turns out because a train is made out of
multiple cars that means it also uses
can at least that's my
understanding in reality it doesn't even
use can it uses a layer on top of that
called can open so we'll kind of look at
that layer instead and can open
effectively turns your network of or or
network of can into a distributed
mapping of ID into variable data so
different parts of the computer can like
read of the system can read and write
some values and the first thing we
effectively analyzed is like what is the
difference on the can buas what's the
difference in can open between a train
which works and a train which which
doesn't work also know as known as a
trained and what we found out quite
quickly is that the inverter would
actually be told uh slightly different
messages from the PLC instead of being
sent a one which means uh please run
please do not run the the emergency stop
it would be sent a zero which means do
not run and instead of being sent any
power to any of the four inverters the
power would be always set to zero on the
other hand everything from the frontal
lever up to the PLC looked fine so
obviously we needed to look at somewhere
in between the PLC and the inverter
because that was the most likely
corporate so this is what we can reduce
our problem to there is a train it has a
can Network it's actually a bunch of
different buses it's five of them
there's two plcs which are called master
and slave and there's four inverters so
let's just focus on that and in fact
let's focus on the PLC first because I
keep we keep talking about CPU PLC what
is a PLC a PLC is a programmable logic
controller it's like an Arduino but
bigger what's a TCMS that TCMS is the
entire system that you get from a vendor
if you want to control your train
electronically and in this case neac
bought a system from selecton which is a
Swiss brand part of a system called Mass
M83 X which is based around the CPU 831
TG PLC which I'm just going to call CPU
from now on and in in cont in comparison
to Arduino is actually programmed not
via C code but via a weird standard
called IC 61131-3
we'll get back to that later this is the
PLC in this natural habitat there's the
PLC on the right there's a canbas
expander to the to it left and finally
to its full left is a power box notice
that none of these are colored yellow
and this is very important it means
they're not safety critical so we can't
touch them I am not is not a joke the
important parts are yellow so whenever
you look at the train keep that in mind
so how do you actually program one of
these we have the PLC here on the right
we looked into it it's based around an
infinion tc1 130 trick core core it has
a bunch of RAM and has a bunch of flash
which I'm sure you know what it is it
has also n vram which is nonvolatile Ram
which behaves like Ram but keeps across
power cycles and for programming there's
a software suite from selecton called
CAP 1131 where you define your project
in the I standard that gets code
generated into C and then get gets fed
through GCC to actually create a normal
binary and finally that is uploaded over
U UDP or
rs232 so we know how to upload a program
but how do we actually download it
because turns out the software does not
have a button that lets you get the
software from the from the PLC for some
reason but we did take a look closer at
it and we found debugging functionality
and that was basically what how we ended
up acquiring the image from a running
PLC uh the debugger is based around a
subsystem called ciscom which is like
this UniFi dll that is shared across
different programs from selectron like
the ID called CAP and some extra
utilities we looked at that D we look at
some UDP packet capture and we found
more or less a protocol and what we
found is a
that when you want to debug a program
you set you log into the device with
username and password thankfully that
cannot be
changed at least not in the old software
versions from selectron uh and then you
query like hey I would like to get this
variable range like I would like to
monitor this variable range by address
and I like this memory that I previously
requested and turns out yes you just
give it an address so we can just write
a real debugger that just asks for all
memory range from the beginning to end
and that's how you get an image of your
project from a PLC and now I must
apologize but I'm going to the most
boring part of this talk which is
talking about the standard that is used
to program these things let's start easy
this a this is a sample poou there's
going to be a lot of acronyms PO is
program organizational unit and this
kind of PO is an FB which is a
functional block function block which is
implemented in St which is structured
text it's a mouthful don't worry about
it it's going to get worse top part are
the variable definitions bottom part is
the actual code what happens is the code
is just executed every single tick of
the system and we have free variables we
have start stop running these are two
inputs one output and X state which is a
persistent State and the code gets
executed from the beginning to the end
you if x start is set X state is set to
true if x stop is set X state is set to
false and then the output X running is
set from X state so far so good not
terrible let's look at how it looks like
when it's generated to C this is the
structure that gets generated you can
kind of think of it as object
orientation right like you have a
structure for your FB and that's fine
like all the variables we've seen are
embedded as parts of a structure not bad
this is the emitted evalu this is the
emitted evaluation code so this D St
code converted into C you'll see it's
pretty much a one to one mapping right
so again the same code if x start is
true set X state to true if x state is
false sorry if x stop is true set X St
to false and assign the result to X
running
cool uh this is another function that
gets admitted this is called the init
function and that is like any object
orientation as well it's a Constructor
for your struct so that gets called the
first time this thing gets initialized
since everything we start off is false
this is all just zero assignments okay
not too bad now for the it gets worse
part this is another poou this is uh
program which is another kind of Po it's
like an FB but it's top level and this
one is implemented in a language called
fbd which starts for function n blog
diagram and you see this language
doesn't text or at least it's not mostly
text is actually graphical language and
what we see here is we embedded our
instantiation of the previously defined
uh
FB and we wired some some other inputs
and outputs so X start X stop and X
running a and X running B and these are
wired so that effectively our two little
motor controllers control to different
outputs and are crisscross connected to
the start and stop let's see how that
gets generated so structure as expected
the generated structure for the top
level program still has the all the
variables we defined it's missing X
start because we messed up sorry and it
has two s and it has the two embedded po
motor control one motor control 2 but
you'll see the type here is unexpected
it's SC C in C it's not the you would
expect that it might be the the
structure we defined previously but this
just like weic type put a pin in that
let's look at the uh initialization
function it's more or less what we've
seen before things just get set to zero
but again nothing is touching the
embedded pus so what's up with that well
maybe let's look at the body and that
will answer our questions so this is the
generated body code from that diagram we
saw and if you look at the the two
little uh things in Brackets these are
the two evaluations on of the
subordinate pus we set the x start of
the first motor control to the xart of
the top level module xtop to xtop and
then the first second one they
crisscross so xart is xtop xtop is xart
and then for each one of them we
evaluate them and you see the evaluation
goes for macro code o and actually if
you look a bit higher up the getting the
reference to the struct goes for macro
code o so what are these and these turns
out are the worst part of this job
because the macro OD and O are just
defined as the references for a global
table called oop so effectively every
time you have one po accessing another
poou it does so for a global array that
is effectively untyped and sometimes the
indices into this array are Dynamic so
this successfully breaks any sort of
attempt of doing doing static analysis
the traditional way with traditional
tool link just for complete completeness
this is an oopt we generated and if you
remember in this code sample we had o of
12 to get the function
uh to evaluate both motor controls and
here the index number 12 is indeed the
SC body function of the motor control so
you can follow that but it is definitely
not pleasant and the SC Inc types within
the structures we have we still haven't
seen how they're actually set turns out
there's another Global Constructor that
sets all of them and we'll just skip
that because again you already have
falling
asleep so we more or less know how the
Cen looks like we know how the generated
SE sources look like now uh Redford is
going to talk a bit how this looks like
the other way around starting from
binary back to
C so let's look at the same toy project
but now looking at something closer to
what we'll have from a real train so the
binary uh and it will get much worse
because we first has to have to
disassemble it or the compile and we had
to use gidra because other tools just
didn't work well for trior and actually
G also didn't work but that that will'll
discuss in a moment it's going to get
worse yes it's going to get worse much
worse uh for the toy example we had
symbols but of course for a real binary
we didn't have any
symbols uh and let's see that's the the
compiled code for our to toy example and
you can see in the first lines that
there are even more casts and here it's
not that bad because we have symbols so
it means that the compiler knew the
types so it's not that bad but it's
still annoying and also you can see that
there are some troubles with function
pointers you see this end f f f f FF
this shouldn't look like
that so the problems with gidra and
trior there were a lot of them for
example trior is a bit weird it has
separate registers for data and
addresses and as you can see on this
example they're reflected in the calling
convention if a function has data
arguments that are passed through data
registers and pointers are passed
through address registers and the
problem is that gidra doesn't implement
this so we had all the arguments order
messed up so we had to script gidra to
fix
that then you have bugs in the uh
semantics of trick or instructions this
is an example nor dot had this end one
in a wrong place so gidra actually
thought that this instruction work
slightly differently then it really did
and we found a bunch of bags like
that so what to do uh we used a lot of
scripting because uh you've seen this
OKO PT type
which basically erased types from
everything so yeah a lot of scripting to
try to undo the damage then some binary
analyzis remember we have this spare
computer we can upload the code we
downloaded from a train and just do some
experiments on the desk for example we
can poke some variables and see what
happens and this is quite quick because
we can do it at
home uh then another idea is to look at
the differences between trains just div
the software you may ask how is this
possible why aren't they all running the
same software it turns out that I we
don't know why but basically all the
trains had slightly different software
so we had I think now we have 26
different versions of the
software for 30 trains
yes so that's a lot
and maybe maybe there's something
interesting in the differences between
them and we could also look at the can
traffic and try to find the
corresponding place in the
code and also we had some motivation in
the form of a deadline because the
workshop called us after some time
passed hey guys no pressure but like you
have a
week because it turned out that the
train operator after like running
without the trains I think they they
they were six of the trains were locked
already and it was problematic for them
so they decided that next week they will
cancel the servicing contract and we'll
ask the manufacturer to do the servicing
because the manufacturer claimed that
they can fix
that so we had quite a short short
deadline and in the end it turned out
that the ding was the the most fruitful
way because it as we noticed there are a
lot of changes in the code responsible
for locking the trains and we actually
found the exact place where there's a
logic which blocks the trains on
purpose uh it was basically the the the
the first part we found was operating on
values which were almost like a million
so our guess was this looks like a like
odometer because we knew that the trains
were supposed to be serviced at 1
million so the values look like odometer
readings uh and we we notice that it
does something with the values and then
if something is wrong it swaps one bit
in nvam and then we found more bits like
that with different conditions which can
switch them and we did a quick test
because it looks interesting we can do a
test on our desk let's oh and what one
one very important thing we noticed that
the those bits differ between locked up
and running
trains so this is a bit suspicious let's
change the bits to be the same as the
the trains which are healthy and we
uploaded this to the train
controller
[Applause]
and
[Applause]
okay but that that was just a test we
need to actually understand what's
happening there before we like allow
those trains to run like it's all nice
to just flip some bits and like confirm
it does the thing but it's slightly
weird to just like be fine with that we
wanted to actually understand what's
going on so there are two very important
questions the first question is what are
the mechanism which cause the train to
actually not run and the second question
is what triggers them so let's start
with the Locking
mechanisms the first mechanism you
already seen in the message to uh power
converters so you've seen those four
bytes each of them corresponds to one
power inverter and if the bit uh here
you can see if uh this is very
simplified like the code like we wanted
to post the code hours to fit the code
in the slide it just doesn't fit so this
is a simplified data flow diagram still
treat this with a grain of salt I
actually tried I gave up at I don't know
four slides or something like that with
code so it's something like that uh you
have a bunch of different checks which
are those triggers we'll talk in a
moment they got ended and if if they
decide that the train should be disabled
then the uh throttle is gated through
this and it's always zero if the checks
don't pass and that influences the can
open
message another way which you also have
seen in the that message is this bit
which is uh usually it's documented as
reserved for but in some documentation
we found uh the description as emergency
stop
uh so we are not sure how exactly it
works but the locked trains had zero and
the working ones had one and here we can
actually show you the code we had this
function which decided whether the train
should run or not and it was basically
directly written into this scan message
the result of the check you can see this
the building of this one bite of the
message you have four four bits which is
in the message you see one s f o f f is
those four ones which decide which power
inverter should be enabled and which
should be disabled and then you have
this magic bit which comes from a
function FB call which we call sneaky
checks it's a technical
term but that did a lot of the
calculation for the like required
conditions like this gets piped into a
whole bunch of other parts in the code
but this is one place where you could
show it like almost immediately flow
into the the C
messages all right my turn so uh another
nice failure how we can lock a train is
to tamper with some aor stuff so we have
this pantograph it's this thingy at roof
of the train that connects to the uh
wire that basically provides all the
electricity that train needs but this
device is operated on a pressurized air
so if the train is cold and dark we need
to generate some initial air pressure uh
for the pantograph or uh electricity
pickup to go up uh but we can't use the
main compressor because it's really a
power hungry device and we have very
limited batteries in the train so
there's a secondary compressor uh
installed in the train that can be uh
started in a cold train then it
generates some small amount of pressure
that is just enough to raise the
panograph then we have enough
electricity to engage the main
compressor pressurize everything else
and we even have a brakes then so uh the
idea here was to it would be a real
shame if the secondary compressor just
stopped working yes so one of the look
mechanism was to disable secondary
compressor and after a while if it's
disabled and it should be enabled it's
reported as a compressor failure and we
can't start the
train so another another very important
question is how to trigger all these
lock mechanisms so we found a few ways
to do that first that is one of the most
common in the trains we investigated is
what we call lack of movement or idle
timer so I want to mention here that all
these trains are Suburban kinds of
trains so like asban they run for like
20 hours per day to generate any Revenue
so there is a check that verifies if a
train was running over 60 km/ hour for
over 3 minutes and if this condition is
not met the train would just Perman
permanently
lock so if you may ask yourself a
question when the train doesn't move for
10 days when it should move 20 hours
each day probably either something is
broken and it needs servicing and yep
somebody has to do that servicing
or it's not used but this is not really
a case except for few cases for example
uh before uh the trains went to the
workshop to get serviced uh they were
just withdrawn from the service they
waited few days at the sidet tracks and
it was enough to trigger the lock so
colon the operator asked manufacturer to
fix those trains and they fixed it by
extending the lock mechanism from 10 to
21 days and they've added a very clever
mechanism Jo fencing so the train would
only lock if it stays
[Applause]
so you can the train would look only if
it states in these random locations so
let's try to draw this random locations
on map
so the first location is the main
competitor of neak PESA bidos that's the
second biggest manufacturer of trains in
Poland and their workshops that's the
third Workshop owned by PESA in M maski
that's the SPs Workshop that was still
in construction when those stuff was
implemented that's that's called future
proof that's another competitor fabl
from very nice sound
city of
shanu then that's the SPs we were hired
by but wait there's more and there is
the manufacturer Workshop in nov but it
has an additional condition which was
disabled all of the train so for
debugging purposes they could enable
also the geofencing trap at their own
Workshop to
test if it really
works
[Music]
there are also some another additional
locking mechanism that were not present
in all the trains so for example we had
a um you saw it in one of the initial
pictures the can 831 that's the model
that extends number of can buses if the
serial number of the uh extension device
doesn't match the one is stored in the
CPU the train will lock itself
permanently there was also a similar
change uh similar check to check the c
number of WTB that's the bus that
connects multiple trains with each other
then there was a check if the inverter
firmware inverter firmware version is
correct that's actually a same one
because we want to uh have a quite
aligned inverter firmware and the main
software there was a hardcoded check for
1 million kilometers after which the
train will not run and there were some
additional checks for odometer
consistency so uh values of odometer
were uh compared from few different
devices and if they were off by more
than 100 kilm the train would break yeah
but one thing to remember here is that
what as as we said before there were
many different versions of the software
so it's not like every every train had
all of this like every train had a
subset of those checks and except for
the first one the least ones are way
less
common so we also had a very nice date
check in one of the trains so that's the
train that had these secondary
compressor problems so uh the train was
supposed to be serviced on 21st November
of
2022 so21 2021 yep so
yep if you spend a few seconds staring
at this line of code you realize that
this doesn't exactly do what they
intended probably unless they intended
to break the train repeatedly from the
21st to the 30th of November and 21st of
to 31st of December granted this was
only in one train but it's still H
because all of these that's not how you
compare dates I mean it is apparently
the good news here is that in year uh
201100 this lock will not work anymore
for 21 years for 21
years this is actually one of those
examples where we can show you the code
because we did manage to fit it on two
slides this time so we can kind of
follow along on the left hand one you'll
see that we are running the function dfb
to get the current time we do the
comparison as in the previous slide and
we use that to set a set reset latch
which we then save to local storage code
is date after and then in the right hand
Pine you'll see there's a a bit more uh
combining of that with other inputs into
a trigger L variable then this goes
through two other set reset latches and
finally that ends up in an nvrm variable
called nvrm lock enabled so you can kind
of follow it from here to the to this
nvrm lock enabled of course the names
are ours symbol and like none of this
was um had any names so and then we keep
keep keep following and we go actually
to the function that decides to enable
the compressor in general like does the
compressor actually need to run and one
of the conditions is that it's not
locked in in vram and you can see at
right at yes the second statement that
it actually set the output line of one
of the can IO extenders so we can use
these free Snippets of code to like go
from date directly into the line on the
can IO extender and as you can probably
guess spara pomot is secondary
compressor
yes this is on one train yep that train
is now famous because it did indeed
break on the 21st of December this year
yep
[Applause]
but don't worry New Year's is going to
run just
fine what else did we find there
is a secret key
combination you can input in Cabin that
disables each of these locks it's not
actually an economic code it's much more
boring than that but you know yep this
one is the famous one so at some point
Unfortunately they removed it
so oh
well then there was this mysterious
device that was bridging one of the can
buses can 3 uh with uh passenger
information system which has a direct
access to the internet so in theory
there is a possibility to somehow uh
gather Telemetry of the locked values
because we know that this device
received
uh some metadata about the locks in few
cases it was able to the trains were
able to lock by themselves using values
received by this device but uh analysis
of the device itself was not yet
performed that's what basically we're
doing
now there's a few other things we
discovered because like as we were
interacting with follow these operators
some of the trains ended up going to nag
and back so we did the you know kind of
sensible thing of of
doing comparisons before and after but
also not just we didn't just get our
diffs back from what they changed but we
also got a few gifts for us including an
update to dhmi which is the software
that runs on the cabin that would
display the scary message about
intellectual property violations like as
any other system alert like oh you know
your breaks are engaged oh you're
violated copyright uh I am not going to
bother translating all of this it's all
legal [ __ ] but let's actually find
out because maybe there's some some kind
of like hidden legal check behind this
it wasn't like showing all the time
something was triggering this message
let's see thankfully the HMI is just
Linux plus cute so like reverse
engineering it's like just free F to
seconds so we found out that what
actually does supposedly violate nx's
intellectual property laws is stopping
the train for 21 days and then getting
it to run
[Applause]
again now I am not a lawyer but I'm not
exactly sure about this
interpretation okay there's a few
there's two case studies that we still
want to show because they were somewhat
interesting one of them is poo K which
is a Transport company and with their
workshop in Su bitka yes so so quick
context after the train started from
successfully from the from our Workshop
we got a lot of other companies having
troubles asking us to look at their
trains yeah so one of them we we we
learned about four trains that just
wouldn't start and turns out you could
unlock them but if you power cycled them
they would lock up again and by our
understanding this was just a
programming bug so because we didn't
want to actually modify the code to fix
it like all of the things we did we
never modified any code um we they the
trains just got sent ended up being sent
to neak and neak fixed them so of course
we looked at the difference and what
they did is they fixed their bug so they
wouldn't lock up again they extended the
deadline to 21 days just in case and
they unfortunately removed the unlock
code in the
cabin another case studing was poio
stretchin where uh we found that the
blackbox kind of external device that's
kind of used as a date and odometer
subsystem a lot of different things yes
it's kind of like this external
subsystem that's just you see it's
orange one so it's important right and
it would report that is currently the
year 2037 and as we all know 13 15 years
is more than 10 days so the train just
locked
up uh quick summary of all the trains we
had like we tried so hard to make like a
vent diagram or a table of like all the
different lock mechanism it it just
didn't work so here's a quick summary we
analyzed 30 trains 24 of them had
mechanisms the most popular was that
like the train not running for a few
days mechanism that was in nearly all of
the ones that had any lock in the first
place the kind of combination like the
kind of registering against a can 831
seral that was also very common the more
spicy ones were more rare the G the GPS
Geo fening was only two trains the
compressor failure simulation was on the
one train and the UDP can converter
possibly remote lock was also one TR but
we have a few more interesting leads on
that one more thing when you once you
build one of these projects on the SDK
with you the the cap11 31 software a
local file on your Windows computer gets
updated an iny with additional build
metadata like the build time and also
the build path and that then gets
embedded in the in the software that
gets uploaded to the PC and then if
everything goes well a log entry is
created on the PLC when the software is
updated so we can
actually look at some of these log
entries in those PC's and the ones that
we can fairly sure that we can trust we
can cross reference against other
documents and other timelines and we can
reconstruct the story of some of these
trains so let's look back at our
timeline in August 2021 the tender was
one by SPS and then in January 1st 20
Jan sorry January 24th the first time I
was supposed to go to to SPS for serving
and just 3 days before that it got a
software update another train on this on
the 26th of February was to leave to get
to SPS and just two days before that it
got an
update I'm not implying
anything but it's I would like to take a
look at this if I was um Kad speaking of
suspicious things let's talk about n's
response uh so these are it's very
difficult to kind of understand what
they're even trying to say because none
of their statements make any sense so
we're kind of trying to piece together
an argument from their side just so that
we can bring it down ourselves it's it's
really bad but this is more or less what
they're saying first of all the hackers
are someone else did it and we've looked
at the code and it's obvious that
whoever did it had access to the source
code like you would have so many side
effects of trying to Binary patch like
it's obviously integrated whoever had
the source code and I don't know who has
the source codes other than neak maybe
someone stole the trains and stole the
code from n and changed it and upload it
just doesn't make sense second point
this is slander and
defamation truth is I haven't heard of
neak before this
so why would I do that first perect
there's no proof have you seen the
slides fourth point oh we haven't
touched these distraint since 2018 so
what like you can write code ahead of
time what
what fifth point if I'm counting
correctly this a violation of Ip laws
our lawyer backs to differ we'll see you
in court about
that
next point the software was interfered
with and those trains are now unsafe we
either use the unlock combinations that
you left in the code or when you remove
the actual reading of the HMI interface
we just trigger the same functions we
just use your code if your code is
unsafe I don't think that's our problem
right Next Step they actually came up
with a new defense l oh SPS was didn't
have the right documents or the right
software to fix these trains oh you
didn't have this special
servicing software how did you expect to
get them running well when the tender to
buy the trains from neak to KD was
published was was uh won by neak neak
was supposed to give all the docks
required to get the train service so
maybe as maybe neak is now saying that
they didn't fulfill the part of their
procurement and finally the last point
they like to make oh SPS are incompetent
idiots they just keep breaking the
trains and nobody can be trusted and
like it's a train give us the docks and
like SPS are competent they they're able
to fix it like we've seen what they've
done we'll still we're still waiting for
more from neac because right now none of
the recommendation makes sense of course
they don't admit to any of this next
steps we were supposed to release a
technical report about this it's much
more work than we expected because just
collating all the information from
different firmware version kind of
having comparison tables between all of
them being able to like talk precisely
about the data flow like getting all the
code samples in a row it's a lot of work
sorry we'll try to get this done in in
in
January second next step we haven't been
sued yet uh
[Music]
if I would like to say hi to all the nag
lawyers on the room and you know where
to find us if you need
us to be a bit more serious the we have
we have been in touch with the various
different uh governmental agencies about
this basically since we learned this the
Polish antitrust office W is aware of
this since forever we've been talking to
them the there's a public prosecutor's
office case against neak probably right
now going on but we don't have much
comes with that it's it's like they only
said that they are looking into the case
we're not really kept in the loop but we
did speak to all them and we did present
our case and finally oh yeah there's
also like a whole bunch of free letter
agencies that we are more or less in
touch with and finally we are likely to
speak about this to a political audience
at the Polish parliamentary uh
Commission
I I think that's a talk thank you for
listening do you have any questions wow
woo woo yo yo
yo
yeah
yeah so cheat codes for trains so what a
world um I have heard we have a lot of
questions uh in the room and also from
the internet we start with the
internet okay um the internet wants to
know how did the trains that got the new
a software update that inserted the
locks um how did they get there where
that was that done remotely oh yeah uh
that's something we should have spoken
about in the presentation uh not they
weren't done remotely just to make it
clear we are not aware of any mechanism
to up to update the train remotely all
the updates were performed by n n
service people or at Nag's service
Workshop yeah so like when we said we
compared the software it was like the
operator sending the trains to the
manufacturer and we did the dump right
before and right after the train was
sent to them so we could see like what
they
changed yeah microphone phone
4 are you uh implying that these idiots
hooked up a canbas transceiver to the
internet and to the train uh
controls
maybe there is to be a bit more prec
it's a bit more complicated because
there's the UDP to can converter box I
mean it's UDP to modb rtu and can and
the can the can side of it is connected
to one of the can networks and the UDP
side is connected to a local telematics
network for the passenger information
system and that passenger information
system c matics network also has a GSM
modem on it yeah but for sure it it
wasn't just like an open Internet that
anyone can send it there's probably some
VPN or something like that but that's
something we plan to analyze in upcoming
months so maybe there will be an update
presentation someday
you then the next question from the
internet uh yeah you mentioned that
there were some software updates um a
few days before the trains got sent into
service um how did those updates get
there um do they actually have 24/7
access to the trains or was this done
remotely as far as I know they they had
before this whole thing got more public
they they got access to the
trains my understanding is that there's
like shared workshops between a lot of
these comp sometimes they they just have
like shared Workshop or some free access
like when they request access the
at least that was in the past now it
surprisingly it
changed then microphone one please um
it's the 27th of December did you travel
here by
train I didn't unfortunately there isn't
good train
service microphone too uh so my question
would be about the I call it Orange Box
the autom autometer thing uh with a
weird year like 2037 was that thing
affected by the year 2038
bug would the train actually run again
after a year box that you're supposed to
send back to the manufacturer every so
often to get it fixed otherwise it
breaks like that or is that a different
box we we heard some rumors but that
that's not confirmed we just heard
rumors that those particular Red Box had
a back with dates and it's it wasn't the
only case it's probably just back in the
code some integer overflow somewhere or
something like that but we don't have
any like more concrete data then the
next question from the internet uh yeah
the internet wants to know if you
actually found any um legitimate locking
mechanisms so for example if there's
something like I don't know battery
damage or whatever and that prevents the
train from starting or is it just um
workarounds yeah like for example the uh
if the fir if there's a firmer mismatch
between the inverters and the PLC I
think it's it's a legit thing to like
block the train because like the PLC
software is present a very particular
version of the firmware so that's a
safety reason to like not run against it
however this fed into the same mechanism
of just like silently stopping the train
I would have expected all of this would
just like have a documented Behavior
same for things like replacing some of
the parts like I could argue okay this
needs to be done uh you know not have
Hazard so this should be
like explicitly resynchronized
afterwards but no docks so even if we
argue that some of those checks were
legit and I think some of them we could
actually call Legit the way they
manifested was sketchy yeah there were
more more checks which skipped which
were like very legit like testing the
brakes and stuff like that or if doors
are closed so we had we had a theory
that it started with like a legitimate
system for stopping the train if
something is wrong but then they started
adding more conditions to it which were
less
legitimate yeah then uh microphone
number one please
um you said that you are only using
functions the train manufacturer
provided uh does that mean the firmware
running on those trains has remained
unchanged yes we have not changed any of
the code and that's the reason some of
the trains we analyzed we couldn't
unlock for example the one with the comp
the compressor failure it was uh there
was a key me like key cheat code to
unlock it but it was bagged and if you
power cycle the train it will break
again yep we are very careful because uh
altering in our understanding altering
CPU firmware would require a new
certification of that train to be
allowed to run on a public tracks but
yep so that we didn't touch manufacturer
did
but then microphone number two please
yeah so I've got a question about the
origins of all of this uh I don't know
about uh train maintenance but I know in
car maintenance they look at fall codes
and they look at OBD data and stuff like
that I'd imagine it to be uh the same or
similar with trains um how did SPS get
so suspicious that they reached out to
polish hackers that they needed to
Google that because normally they've got
helplines service stocks whatever what
made them jump to this well you have to
understand that SPS and neac might have
some conflicts of interest in helping
each other so there might not be a way
to just call Ne and ask yo what's up if
the next thing that neac says oh you
messed with our security system so like
kind of any sort of you know hoping to
call someone and get this resolved area
kind of wasn't was dead on arrival for
them yeah the manufacturer wasn't really
helpful and as as you've seen it almost
ended up like manufacturer taking over
the tender yeah and in in terms of like
debug codes and debug utility
We There is some utility some very
limited Diagnostics utility yeah but it
does it doesn't show the the locks so
it's not very useful for
us thank you okay then microphone number
four uh have you looked at if there are
any valid safety locks which lock and
report an error compared to the ones you
described which just silently
fail plenty of eror codes like for examp
example even with the compressor failure
like it would simulate an actual failure
by not running the compressor and then
another subsystem would report oh
there's no pressure so something
probably went wrong and all of these
have actual effects on the HMI that are
you know understandable by humans so
these exist like there's nothing that
should have prevented this from also
being a human readable error code
instead it was this silent failure mode
of like don't report anything is
wrong microphone number one please train
was actually saying ready to start yeah
ready to start but it
didn't then ready to start for
microphone one yes so after your story
got a bit of news coverage did any other
train operators reach out to you already
maybe even from Germany about some dodgy
train
Behavior no
[Applause]
comment
[Music]
yes I have to laugh um uh microphone
number two please Perfect um hi at first
thank you for the nice talk it was very
delightful just to watch you talk about
it um did you at any point in this whole
journey of unlocking the train regret
the decision to accept it that's a very
good question um personally I've been a
bit like you know there was this one day
before deadline where nothing was
working yeah so there's that part for me
personally just even like this is
probably going to go forever for a long
time it's going to be in the media for
while it's going to get political so
there's always like maybe we shouldn't
have gone public with it but at the end
of the day that's the correct thing to
[Applause]
[Music]
do and yeah like when when it was like
one day before the deadline my two
friends here were like crunching really
hard it was a 24 shift system to
discover this so we we actually forgot
to tell tell this trivia in the story we
managed to unlock the first train 43
minutes before the the train operator
arrived to cancel the contract so it was
really really
[Applause]
tough my microphone too
please am my microphone too oh yeah
that's everything thank you perfect uh
microphone one hi you did mention that
uh you have to get the trrain
recertified when you change the firmware
but uh doesn't that need to be certified
when they change the uh date range to 20
days or is the software OB curent for
the certifier it it sounds suspicious no
that they just can update the firm like
that without recertifying doesn't
it and
that's that's part of the that's part of
the low that we like not 100% confident
to comment on the law isn't
exactly uh clean on this it I think it
says that if the change is significant
or something like that so it's I I don't
know this is something more for a train
lawyer on the other hand that the change
was made and it was not mentioned in any
paperwork from that service yeah of
course they didn't mention that they
updated the locking mechanism so or even
update that they updated the software
right that wasn't loed anywhere some in
some cases the train operators didn't
even know that the software was
changed but is the software obscurify to
the
certifier that's another good question
uh we don't know
okay um microphone number
three um you mentioned that the code is
executed at every tick or something um
does this mean that the kill switch
conditions are also evaluated every now
and then or only at the startup so like
if the train during operation reaches
the 1 million
kilometers did uh would it emergency
break or or something that's another
very good question yes so I can tell uh
the condition for 1 million Kil kilom
was in the same train that had the
secondary compressor problem so it would
just disable the secondary compressor
that is not needed in a normal train run
just to bring it up from a cold and dark
State yeah normally when when the train
was running you had pressure in the main
pressure tank but what about the other
trigger mechanisms and Trigger the other
mechanism like standing in place uh had
more checks to Ure sure that it didn't
it won't engage when the train is
running uh then signal Angel the next
question from the
internets yeah the internet wanted to
know if that train manufacturer actually
sells elsewhere besides Poland yes
yes then microphone number
two uh yes uh besides this sheet code
code did you find any other uh
delightful Easter eggs or any anything
of that sort in the
code not
really nothing immediately comes to mind
yeah uh I think the the best Easter egg
was just the date checking because we
now are observing twice per year the
international compressor failure
day
then microphone number four uh hi you
mentioned that the cheat code wasn't the
Konami Code are you able to reveal what
it actually was and if it was documented
anywhere it was not documented it was
very simple we're not revealing the
particular code because we're afraid of
people like just randomly pressing the
buttons in the cabin because there was
more than one code like every trigger
mechanisms yeah you you could do
different stuff with trains with the
codes and we don't want to hint people
about the pattern how the code looks
look like so they won't start randomly
experimenting and they they also
differed between some trains so better
not experiment with them we always like
check the code and only then we use them
but it's like something that you could
press with two hands within seconds it's
not like you have to sit there and like
type in something with binary code it
was uh a very quick
combination all right thanks then the
next question from the internet yeah the
internet wants to know what was or is
the um relation between you and SPS and
if you got anything from it like besides
publicity and maybe free rides in
trains good question we should have
declared our possible conflicts of
interest we were contracted to do this
we were paid money changed hands uh yeah
we were basically paid to analyze and
unlock the trains although not for
publishing this we just got like a green
light if you want you you can make it
public but the the workshop was
interested mostly in checking why the
trains
broke and just providing a a technical
report describing what
happened then microsome
3 hello uh given the ramifications are
you concerned about any reprisals be
they political or otherwise and hello
from
Ben hi Ben uh I mean we are mostly
worried about just being dragged through
courts forever uh that's it because
that's like the easiest way to just
annoy us I don't want to give NE lawyers
any ideas but oops too late I
guess otherwise no we are we are like
100% sure of of that we are in the right
we're 100% sure we're acting the public
interest it's naak that should be scared
not us and we are
[Applause]
100% and additionally we are 100%
certain of our findings presented here
yeah so unless they start sending ninja
squads our way I think we're
safe Microsoft number one is there any
chance that nag is going to get her
hands on those trains and would try to
destroy evidence or do something like
that could you repeat the question
please if if there is any chance that
naag will get their hands on those
trains which were like with those locks
and we'll try to update and destroy that
evidence like update firmware removing
those locks yeah we we are prepared for
that we have done dumps already some of
them are audited by big four companies
so even if they try to like the worst
thing they can do now is start try to
hide the evidence because I don't think
the court is going to like
that yeah we are running out of time
which means uh you can reach uh here all
the colleagues
Redford
[Music]
[Applause]
qck and Mr
tick
[Music]
yeah