1 00:00:15,799 --> 00:00:20,360 Herald: So could you give a warm welcome of applause to Stephan Gronke who will be 2 00:00:20,360 --> 00:00:23,329 talking to you in one minute. 3 00:00:23,799 --> 00:00:31,277 *Applause* 4 00:00:31,277 --> 00:00:35,290 Stephan: So hi everybody my name is 5 00:00:35,290 --> 00:00:39,420 Stephan Gronke. I'm a software developer since about 15 years, working in solo 6 00:00:39,420 --> 00:00:44,499 projects and larger teams and smaller teams. So mostly my development stack was 7 00:00:44,499 --> 00:00:50,260 JavaScript and you will find some of the tools that I mentioned coming from this 8 00:00:50,260 --> 00:00:55,829 world, but I'm very sure you can also find something for your project that applies 9 00:00:55,829 --> 00:01:02,249 here. Here's my email address, my PGP key and my favorite social network account. 10 00:01:02,249 --> 00:01:09,350 Yeah, so a little spoiler what will happen today: I will talk about development 11 00:01:09,350 --> 00:01:14,060 process exploitation. So that means if you are developing your software and somebody 12 00:01:14,060 --> 00:01:18,460 joins your team and since you code for review it could happen that it executes 13 00:01:18,460 --> 00:01:23,560 code on your machine without your knowledge. There are a few things that are 14 00:01:23,560 --> 00:01:27,969 really hard to catch or I found hard to catch and I want to share. Maybe you have 15 00:01:27,969 --> 00:01:32,170 the same problems and you find that the same mitigations apply for your project as 16 00:01:32,170 --> 00:01:40,929 well. I will then continue and... let's start with a software development process 17 00:01:40,929 --> 00:01:44,799 that's a small cycle. So first of all it starts with an operating system: you need 18 00:01:44,799 --> 00:01:48,600 to have a computer to write a software and that's something you need to trust first 19 00:01:48,600 --> 00:01:59,720 off. If you... yes, so your operating system contains keys and credentials, it 20 00:01:59,720 --> 00:02:03,159 contains the source code you want to develop and the tools that you have in 21 00:02:03,159 --> 00:02:09,180 place. And the major risk is that tools are vulnerable to some exploitation or 22 00:02:09,180 --> 00:02:13,330 that your host is already compromised and you write a software, commit it to your 23 00:02:13,330 --> 00:02:20,530 coworkers and it isn't what you intended to write. That's a larger problem here. 24 00:02:20,530 --> 00:02:25,611 After you start writing code, the editor is kind of the interface that I have to 25 00:02:25,611 --> 00:02:33,250 write the files and edit the code. I find it kind of complex to use an editor. On 26 00:02:33,250 --> 00:02:36,420 the left you can see that many of the editors come with a package manager 27 00:02:36,420 --> 00:02:41,940 included, which is a good sign for the complexity that these tools have. And I 28 00:02:41,940 --> 00:02:45,150 don't know what tools you need but they support you in development so it's very 29 00:02:45,150 --> 00:02:49,490 good to, for example, have code linters and auto-completion in place to write 30 00:02:49,490 --> 00:02:56,750 better code. At the same time it can be a problem because they can execute code 31 00:02:56,750 --> 00:03:02,472 unattendedly. We will see in a moment. The mitigation I came up with for the editor 32 00:03:02,472 --> 00:03:06,680 part is that you have a virtualized environment where you run your editor. So 33 00:03:06,680 --> 00:03:11,080 when something happens and and it is compromised, not your root system is also 34 00:03:11,080 --> 00:03:15,880 compromised as well. You want to monitor all your config files that you have in the 35 00:03:15,880 --> 00:03:20,590 project and you want to get awareness what exactly happens on my system when I run 36 00:03:20,590 --> 00:03:22,450 and view this code. 37 00:03:24,510 --> 00:03:28,430 The next part you will probably use as a shell integration. So as 38 00:03:28,430 --> 00:03:33,340 soon as you open your repository some of the shells I saw just tell you what 39 00:03:33,340 --> 00:03:36,840 branch you're working in and what files were changed and so on. So that's 40 00:03:36,840 --> 00:03:41,790 something that comes very neat if you're developing but it can be a risk as well. 41 00:03:43,430 --> 00:03:47,240 So my opinion on the shell integrations is mostly that it's made for software 42 00:03:47,240 --> 00:03:50,980 development on your own system. So when you write the code and you can trust it 43 00:03:50,980 --> 00:03:55,760 it's not a problem to use those tools at all, but as soon as you get sources from 44 00:03:55,760 --> 00:04:05,760 foreign developers it can be a problem. So choose your tools wisely and don't execute 45 00:04:05,760 --> 00:04:12,290 code from others if possible. The versioning system that you commit your 46 00:04:12,290 --> 00:04:20,040 code to is also a very good choice. For example Git and yeah Git can execute hooks 47 00:04:20,040 --> 00:04:24,070 on different occasions. For example when you check out new code, if you commit and 48 00:04:24,070 --> 00:04:30,040 so on. That means if you managed to clone a repository and a ".git" folder is 49 00:04:30,040 --> 00:04:34,410 included or a ".hg" folder is included, it could mean that your operating system 50 00:04:34,410 --> 00:04:39,800 decides to execute whatever is in the hooks. It's not possible to store a ".git" 51 00:04:39,800 --> 00:04:43,130 folder within a Git repository but it's possible to store it in a Mercurial 52 00:04:43,130 --> 00:04:48,480 repository or in a SVN or something, and then your shell integration won't know 53 00:04:48,480 --> 00:04:53,210 what the original source was and will execute it anyway. One thing that was 54 00:04:53,210 --> 00:04:57,990 introduced for example from Visual Studio Code, this October, is that they now 55 00:04:57,990 --> 00:05:00,890 support git hooks - which is a great feature, right? *chuckles* 56 00:05:01,974 --> 00:05:03,090 The mitigations 57 00:05:03,090 --> 00:05:07,030 against this are pretty easy: you can either set a different hooks path, which 58 00:05:07,030 --> 00:05:12,790 is not within this project repository so that you don't execute git hooks at all, 59 00:05:12,790 --> 00:05:16,850 or you can use that little wrapper here that you see, to for example check at 60 00:05:16,850 --> 00:05:21,210 least that there is no file that is a git hook within that folder before you execute 61 00:05:21,210 --> 00:05:25,210 git. It's a very good choice if you want to protect yourself from that 62 00:05:25,210 --> 00:05:27,995 vulnerability. 63 00:05:29,745 --> 00:05:33,913 So after you committed the code and shared it to the versioning 64 00:05:33,913 --> 00:05:38,410 server, you probably are going to build it automatically. So some services like 65 00:05:38,410 --> 00:05:44,230 Travis CI will run it, will run it for you. So they will run tests, they will 66 00:05:44,230 --> 00:05:49,090 compile the software and also they do the package versioning and deployment to some 67 00:05:49,090 --> 00:05:53,880 other places. It becomes a problem if you can't reproduce the results from your 68 00:05:53,880 --> 00:05:59,740 built runner, because it's an system you don't control sometimes. And as soon as 69 00:05:59,740 --> 00:06:04,400 you get the binary result from it - if you compile the software that compiles to 70 00:06:04,400 --> 00:06:09,030 binary - you need to check that result somehow, because somebody could have 71 00:06:09,030 --> 00:06:15,230 altered it without your knowledge and then you will ship it to your users. Also a 72 00:06:15,230 --> 00:06:19,280 problem on many of this build workers is: you want to have this process very fast. 73 00:06:19,280 --> 00:06:23,130 So that means you don't want to wait until all the dependencies are installed and the 74 00:06:23,130 --> 00:06:28,290 great service is that you have caching in between these projects. This means that, 75 00:06:28,290 --> 00:06:33,030 for example, if somebody managed to inject the version to the cache of some CI 76 00:06:33,030 --> 00:06:37,919 system, then it will eventually show up in other projects as well and you can pivot 77 00:06:37,919 --> 00:06:43,940 across the projects. Usually if you have a build environment it has access to some 78 00:06:43,940 --> 00:06:48,840 kind of development key. Mostly if you get pull requests from externals, the keys are 79 00:06:48,840 --> 00:06:52,450 stored encrypted and you don't have access to them, but as soon as somebody has write 80 00:06:52,450 --> 00:06:58,110 access to your repository also the keys could be leaked. Let's make an example: 81 00:06:58,110 --> 00:07:01,520 you have somebody offering you a software and you don't give permission to edit the 82 00:07:01,520 --> 00:07:05,930 master branch of the repository but as soon as you open an open a branch anywhere 83 00:07:05,930 --> 00:07:10,530 and make a pull request Travis CI or other build runners will use that and decrypt 84 00:07:10,530 --> 00:07:14,110 the passwords for you and give you access to the credentials which you can then 85 00:07:14,110 --> 00:07:16,942 print or do whatever you intend to. 86 00:07:18,235 --> 00:07:21,500 Yeah, and for me the best option here would be 87 00:07:21,500 --> 00:07:26,420 to have reproducible builds because then you can use different of the build workers 88 00:07:26,420 --> 00:07:30,300 and compare results somehow so that you see if one gets compromised the other two 89 00:07:30,300 --> 00:07:33,519 will tell you: hey, there's a different result, have a look please. 90 00:07:34,220 --> 00:07:35,800 That would be great. 91 00:07:36,250 --> 00:07:40,040 And also the build steps; I mentioned building, testing and packaging 92 00:07:40,040 --> 00:07:43,940 the software are totally different steps so what you can do is you can have one 93 00:07:43,940 --> 00:07:48,930 compartment per step so that you can have a data at a finer level and see what 94 00:07:48,930 --> 00:07:55,669 happens here. After you compiled the software, you built the software, you need 95 00:07:55,669 --> 00:07:59,840 to ship it to the user somehow so either you store it in your own server or most 96 00:07:59,840 --> 00:08:04,740 often you use a CDN. You just put it there and it's the asset that's lying around. 97 00:08:04,740 --> 00:08:09,460 Your users will come around, download it from here and execute it, so what is the 98 00:08:09,460 --> 00:08:13,669 problem here? The problem is, that if you have an URL it's very hard to prove that 99 00:08:13,669 --> 00:08:19,110 it's actually from the real maintainer. If you call your software like if - if you 100 00:08:19,110 --> 00:08:23,260 call your account like a different project, then people won't be able to 101 00:08:23,260 --> 00:08:28,320 notice the difference somehow. What you can do to mitigate this, is to publish the 102 00:08:28,320 --> 00:08:34,319 URLs that you legitly using and also sign your assets so that users can check is 103 00:08:34,319 --> 00:08:38,370 that something that the developer intended to give me or is it something that is 104 00:08:38,370 --> 00:08:42,740 really ... that is really not intended. 105 00:08:43,204 --> 00:08:44,986 So, ... 106 00:08:46,176 --> 00:08:49,359 Yes. And the next part is you need to reach out to your users 107 00:08:49,359 --> 00:08:52,139 so you make people aware, that there is a project they can 108 00:08:52,139 --> 00:08:57,160 check out, they can clone and usually you have the package registries. A few slides 109 00:08:57,160 --> 00:09:03,899 back you saw that the package managers are also included in the editors, so that's 110 00:09:03,899 --> 00:09:08,290 also something where you can ship the software but the package manager I was 111 00:09:08,290 --> 00:09:12,790 mostly looking at was for example NPM. There was an interesting occasion where 112 00:09:12,790 --> 00:09:19,139 somebody had a project called Kik.The company Kik then tried to take it down and 113 00:09:19,139 --> 00:09:23,259 the person just ignored it for the moment but then Kik reached out to NPM directly 114 00:09:23,259 --> 00:09:27,860 and they deleted the repository. In consequence the developer removed all his 115 00:09:27,860 --> 00:09:31,920 projects from the versioning server and a few hours later malware showed up with the 116 00:09:31,920 --> 00:09:35,779 same project names, so that means if you have a software that uses that 117 00:09:35,779 --> 00:09:40,649 dependencies and somebody freed up names it would affect your repository as well 118 00:09:40,649 --> 00:09:46,670 and compromise it. That's something that needs mitigation. I think the best idea 119 00:09:46,670 --> 00:09:51,480 here is to not only identify the project by a unique identifier but also 120 00:09:51,480 --> 00:09:58,319 have a GUID or an or a unique identifier per project that does not change, so that 121 00:09:58,319 --> 00:10:01,700 you can make a difference. That's something that's up to the package 122 00:10:01,700 --> 00:10:05,310 registries to implement. That's not something we can do as a user but it's a 123 00:10:05,310 --> 00:10:11,370 very common case - it's a very common case that these packages fluctuate. 124 00:10:11,370 --> 00:10:14,750 So for example, if somebody deletes it, you don't have a backup of that. 125 00:10:14,750 --> 00:10:18,495 A very good idea is also to store offline backups of every package that you 126 00:10:18,495 --> 00:10:22,422 check out and that you install to your software because it's very bad 127 00:10:22,422 --> 00:10:26,597 if you want to maintain your software and you figure out there's something, there's 128 00:10:26,597 --> 00:10:29,372 something missing and I can't recover because it's deleted. 129 00:10:33,165 --> 00:10:38,110 Yes, software developers have some needs during their work. 130 00:10:38,110 --> 00:10:42,730 I want my tooling to perform if my code editor for example is in the VM and 131 00:10:42,730 --> 00:10:49,139 the VM is slow, that's something that's annoying all over the process. So then on 132 00:10:49,139 --> 00:10:53,110 the other hand the velocity is something that your manager will require from you if 133 00:10:53,110 --> 00:10:59,089 you write commercial software or you try to get something done and you can't spend 134 00:10:59,089 --> 00:11:03,019 all day to work on chores and improve the repository, the versioning and so on. 135 00:11:03,019 --> 00:11:09,579 So that's something you need to deal with. Another big factor for me is the 136 00:11:09,579 --> 00:11:13,480 reliability. So as soon as your software goes down and you are in holiday or 137 00:11:13,480 --> 00:11:18,240 something, anybody else from the company or from your team should be able to 138 00:11:18,240 --> 00:11:26,329 recover what was there before, also known as the bus-factor and, yeah, if you have 139 00:11:26,329 --> 00:11:30,779 convenience like for example Ruby on Rails gives you. It gives you a very good, 140 00:11:30,779 --> 00:11:34,510 very easy start in the projects and that's something you don't want to break by 141 00:11:34,510 --> 00:11:38,839 making it too complicated with a development environment. And also 142 00:11:38,839 --> 00:11:45,490 something I've found to be more annoying than helpful is, if you want to pair- 143 00:11:45,490 --> 00:11:49,579 program and you have a very compartmentalized environment, it's very 144 00:11:49,579 --> 00:11:53,269 hard to share the resources that you need to talk about with other developers, 145 00:11:53,269 --> 00:11:56,699 expecting you're not in the same room but working remotely, what is for me most 146 00:11:56,699 --> 00:11:58,479 often the case. 147 00:12:02,110 --> 00:12:06,601 A large problem that I saw is, if you underhand somebody code, 148 00:12:06,601 --> 00:12:13,030 if you go ahead and and check out code from any online resources, it's 149 00:12:13,030 --> 00:12:17,372 sometimes very hard to tell if the code that you see in your, e.g. Git diff, 150 00:12:17,372 --> 00:12:22,689 is what you really would expect to see. I have some examples here, which can 151 00:12:22,689 --> 00:12:25,959 show how this could work and how this could look like if somebody tries 152 00:12:25,959 --> 00:12:30,459 to inject code to your repository, that you don't see. First of all, let's start 153 00:12:30,459 --> 00:12:37,529 with something easy, that's phishing. What you see here on the slide, on the left 154 00:12:37,529 --> 00:12:42,999 side maybe you see the cursor. That's not a full path, that is just a domain name. 155 00:12:42,999 --> 00:12:49,680 The slashes in here are UTF-8 characters so that thing here resolves to a hostname, 156 00:12:49,680 --> 00:12:54,089 and if you control this host, you can get a certificate for it and then the example 157 00:12:54,089 --> 00:12:59,959 below you see, how it would look like, if you install it. First I have a host that's 158 00:12:59,959 --> 00:13:04,480 just running a web server on port 80, so that you can see the result. Okay, I was 159 00:13:04,480 --> 00:13:09,120 cheating a little bit. I was putting the domain in the /etc/hosts so that I don't 160 00:13:09,120 --> 00:13:15,519 have to buy it, for just showing it. It's strange that dot zip is a domain actually, 161 00:13:15,519 --> 00:13:21,599 but then if you install it, you would see that you can send somebody a very 162 00:13:21,599 --> 00:13:25,140 nice-looking link which looks like a totally different project, but it's 163 00:13:25,140 --> 00:13:30,649 pointing to your server instead; and I found many of the package managers having 164 00:13:30,649 --> 00:13:34,899 the nice feature of executing PostScript hooks, so that means, if you have 165 00:13:34,899 --> 00:13:38,574 installed it, it will run some commands afterwards for you. 166 00:13:42,105 --> 00:13:46,499 Then there is invisible code. If you go online somewhere, find in a forum or in a 167 00:13:46,499 --> 00:13:50,760 blog, you find an article and see, "hey, that code is actually solving my problem", 168 00:13:50,760 --> 00:13:54,509 you go ahead and copy/paste it. So, on the left you see the source code how this 169 00:13:54,509 --> 00:14:00,290 would look like in HTML for the blog. On the right there's the result. So you can 170 00:14:00,290 --> 00:14:03,980 go ahead, you can copy/paste from it and if you paste it to a text area, you will 171 00:14:03,980 --> 00:14:07,759 see, that the result is something that you didn't expect. For example, if you copy a 172 00:14:07,759 --> 00:14:11,509 large chunk of code, you won't go ahead and review it on your local system again 173 00:14:11,509 --> 00:14:15,649 and that could be the compromise for your project. 174 00:14:26,869 --> 00:14:31,141 Another example here is, you can use ASCII characters, 175 00:14:31,141 --> 00:14:34,957 the control characters to influence the output in your terminal. 176 00:14:34,957 --> 00:14:38,379 So if your terminal also supports the legacy of ASCII control 177 00:14:38,379 --> 00:14:42,540 characters, you can use that to just revert the line and override it with 178 00:14:42,540 --> 00:14:47,850 something you wouldn't expect. What you see on top here, that harmless script is 179 00:14:47,850 --> 00:14:52,059 the file. It's a little larger than you would expect for just a echo foo, but not 180 00:14:52,059 --> 00:14:57,009 something you would notice when you just see it. Looking at it from a hex editor 181 00:14:57,009 --> 00:15:01,060 you can see, that there is something more going on than just the foo and if you 182 00:15:01,060 --> 00:15:05,509 actually execute it, it will not print something, it will create the pwned text, 183 00:15:05,509 --> 00:15:11,498 which is a good example for you that your host was compromised in this moment. 184 00:15:15,049 --> 00:15:20,089 Another example I found online, so credit to Ariel for this, so there is a byte 185 00:15:20,089 --> 00:15:23,100 sequence you can use so that this even works in a Git diff. So when you're 186 00:15:23,100 --> 00:15:27,110 working exclusively in your terminal and you're not doing reviews on GitHub or some 187 00:15:27,110 --> 00:15:32,710 graphical tool, it could be the case that you don't notice what was going on. What 188 00:15:32,710 --> 00:15:37,529 you can see here on the left is, I created an empty repository, I added a small 189 00:15:37,529 --> 00:15:43,129 script and in the next step down here, I added some improvement to the script, 190 00:15:43,129 --> 00:15:50,569 which is actually the malicious commit that's here in red. Afterwards, I just ran 191 00:15:50,569 --> 00:15:55,690 a Git diff on the code and I see that there is only no backdoor, oh sorry that 192 00:15:55,690 --> 00:16:00,809 should be okay in the updated slides. So, you don't see the evil.sh that it's 193 00:16:00,809 --> 00:16:05,559 executed as well, if you run it. That's something I consider very dangerous. 194 00:16:10,893 --> 00:16:16,179 So, some mitigations: the best thing you can do, is to make it expensive for your 195 00:16:16,179 --> 00:16:21,089 attackers to compromise or try to. So as soon as you have the chance to notice 196 00:16:21,089 --> 00:16:27,009 what is going on, also retrospectively, you can at least burn the capabilities and 197 00:16:27,009 --> 00:16:31,889 tell others how your project was attempted to compromise and that's something 198 00:16:31,889 --> 00:16:35,499 that is, in my opinion, the best mitigation against this complexity. 199 00:16:36,919 --> 00:16:41,750 What you can also do, is you can test your software from external services directly, 200 00:16:41,750 --> 00:16:47,509 which will tell you, if some compromise happened. For example, Git has it newly 201 00:16:47,509 --> 00:16:52,399 introduced. They will check your packages, the dependencies and will warn you about 202 00:16:52,399 --> 00:16:59,499 some vulnerabilities that are commonly known. The best thing you can do on your 203 00:16:59,499 --> 00:17:02,470 local system is to build small compartments, so that if some compromise 204 00:17:02,470 --> 00:17:08,709 happens, it doesn't affect your full host. Also not all your projects you have 205 00:17:08,709 --> 00:17:12,360 access to. And it's very important, that you have backups on a different system 206 00:17:12,360 --> 00:17:16,270 than the hosts you're working on. So if the compromise happens you still have 207 00:17:16,270 --> 00:17:20,849 access to the original data and can compare it and do some forensics on this. 208 00:17:23,589 --> 00:17:27,640 So, the intrusion detection forensics; there are some great tools available. 209 00:17:27,640 --> 00:17:32,600 For example my favorites are DTrace and Opensnoop. You can monitor changes and 210 00:17:32,600 --> 00:17:37,101 access on the file system or on your system at all. And you can e.g. set some 211 00:17:37,101 --> 00:17:40,120 rules for your projects that are specifically matching. So I am 212 00:17:40,120 --> 00:17:44,359 not going to share some rules that match for all projects, but you will figure out 213 00:17:44,359 --> 00:17:48,779 what is e.g. important. Very good start is e.g. to Opensnoop for 214 00:17:48,779 --> 00:17:54,029 /etc/passwd if there was some access, then you can e.g. say that's not something 215 00:17:54,029 --> 00:18:00,770 what my software would do. And again it's very important to have the backups of this 216 00:18:00,770 --> 00:18:04,530 because in the moment where you execute it, you can't trust your host at all. 217 00:18:06,894 --> 00:18:11,919 The idea how to achieve this is, if you have a VM per project for example, 218 00:18:11,919 --> 00:18:15,610 you let it run for half a year, you don't approve the situation. Instead 219 00:18:15,610 --> 00:18:19,309 of having one system that you need to update the software to, you need to update 220 00:18:19,309 --> 00:18:23,950 afterwards all the projects that you're working on frequently and that's something 221 00:18:23,950 --> 00:18:28,360 that's easy to forget. So it's dangerous. If you assume that every time you run some 222 00:18:28,360 --> 00:18:31,940 command or every time you work in a project, you spin up a new server entirely 223 00:18:31,940 --> 00:18:36,090 from scratch, install the dependencies and so on that's something that's not a risk 224 00:18:36,090 --> 00:18:43,240 for you. Also, if you have for example a virtualized server environment you can 225 00:18:43,240 --> 00:18:48,380 have memory dumps at any time, you can monitor the network and you can also diff 226 00:18:48,380 --> 00:18:51,770 the filesystem. For example, if you stop the server and just compare it to a 227 00:18:51,770 --> 00:18:55,540 previous version and see, "hey, here is something that was changed that I didn't 228 00:18:55,540 --> 00:18:58,970 plan". It's great to know. 229 00:19:01,960 --> 00:19:05,800 Very important is also to separate your accounts. E.g., if you see large GitHub 230 00:19:05,800 --> 00:19:09,317 accounts, people are making contributions every day since years. 231 00:19:11,759 --> 00:19:17,410 So it shows that the people have access to very, to many projects from 232 00:19:17,410 --> 00:19:22,250 the same machine. And the permission model from GitHub for example, allows you to 233 00:19:22,250 --> 00:19:26,669 store an SSH key for write access. But it automatically has access to all the 234 00:19:26,669 --> 00:19:30,940 repositories you control. So the best that you can have here, is to make a 235 00:19:30,940 --> 00:19:35,459 new GitHub account for, ... or to make a new account on that versioning system 236 00:19:35,459 --> 00:19:39,020 that only has exclusively write access to that single repository. 237 00:19:39,740 --> 00:19:43,580 So when you work in your compartmented system and you want to 238 00:19:43,580 --> 00:19:50,260 upload or pull changes, you can't influence other repositories. That means 239 00:19:50,260 --> 00:19:54,580 compromise doesn't spread across all your projects and so on, which would be an 240 00:19:54,580 --> 00:20:00,330 invitation for malware somehow or ransomware. And you get a better 241 00:20:00,330 --> 00:20:06,049 permission model if you create a GitHub organization. In this case you can also 242 00:20:06,049 --> 00:20:11,649 limit your own access in a better way. So my recommendation is not to work on your 243 00:20:11,649 --> 00:20:15,119 personal GitHub account but create an organization for your project. 244 00:20:18,809 --> 00:20:26,100 Something many projects are missing are to find responsible persons for security 245 00:20:26,100 --> 00:20:33,310 and to clearly communicate what is the plan for incident response. Small example: 246 00:20:33,310 --> 00:20:36,770 If you have a new project and you find a vulnerability, you would like to 247 00:20:36,770 --> 00:20:40,571 commit it, but you don't open an issue publicly, because then everybody, every 248 00:20:40,571 --> 00:20:44,490 user would be affected. You try to reach out to some developers and if you don't 249 00:20:44,490 --> 00:20:52,570 have any clue how to securely achieve this, that can get you into trouble. And 250 00:20:52,570 --> 00:20:58,080 there are quite a few projects which don't communicate this and some of them don't 251 00:20:58,080 --> 00:21:01,955 even respond to their security@ email address, which is bad. 252 00:21:05,660 --> 00:21:12,210 In this case I told you what I saw from my experiences of working on the projects. 253 00:21:12,210 --> 00:21:20,699 So that's basically my summary of what can be harmful 254 00:21:20,699 --> 00:21:23,210 and what can be good for your project. 255 00:21:23,220 --> 00:21:33,089 *Applause* 256 00:21:33,089 --> 00:21:38,179 Herald: Thank you and we now have time for Q&A. In the room you can line up behind 257 00:21:38,179 --> 00:21:43,070 the microphones and I can see we have a question from the internet already. 258 00:21:44,410 --> 00:21:50,029 Signal Angel: What about Git signed commits? Any thoughts on that? 259 00:21:50,029 --> 00:21:53,620 Answer: So as soon as you have signed commits and I find that you also email 260 00:21:53,620 --> 00:21:57,073 with the same PGP key, it's very interesting that you have the PGP key on 261 00:21:57,073 --> 00:22:01,329 the same host probably, then you have your Git executable. So if somebody executes 262 00:22:01,329 --> 00:22:06,520 Git hooks, they can steal your PGP keys from this. I didn't find any tutorial 263 00:22:06,520 --> 00:22:11,590 online which explains you, how to make it manually, so that you don't use the Git 264 00:22:11,590 --> 00:22:17,049 for signing the commits. But I think it can be very good to sign the commits, 265 00:22:17,049 --> 00:22:23,210 but it can be also dangerous, because your email communication can be compromised. 266 00:22:23,210 --> 00:22:28,980 Herald: Microphone number four. Q: In the Git diff you showed us, there 267 00:22:28,980 --> 00:22:36,874 were some control characters. I think Git diff pipes to less by default, so 268 00:22:36,874 --> 00:22:41,360 shouldn't they appear there somewhere? A: No, they don't. I just checked with the 269 00:22:41,360 --> 00:22:47,290 latest version today. So, that's something that, well, we can also click on the blog 270 00:22:47,290 --> 00:22:49,814 and see, if there is the video available. 271 00:23:04,734 --> 00:23:08,449 Yeah, it's very hard to show from my HTML slide how this works. 272 00:23:08,449 --> 00:23:11,337 So this video animation, ... maybe we can enlarge it a bit. 273 00:23:12,777 --> 00:23:14,226 That's how it would work. 274 00:23:30,260 --> 00:23:34,523 So most often, yes, if you pipe to less or you use a hex editor to review, 275 00:23:34,523 --> 00:23:36,410 then you would notice, yes. 276 00:23:38,270 --> 00:23:44,299 Q: I somehow remember that, maybe it only shows for longer diffs, but I think 277 00:23:44,299 --> 00:23:46,740 when I type "git diff", I can scroll around. 278 00:23:46,740 --> 00:23:53,000 A: Ah, that's interesting. I need to try. Herald: We have a question from microphone 279 00:23:53,000 --> 00:23:58,539 number one. Q: You mentioned Travis having access to 280 00:23:58,539 --> 00:24:07,470 hidden variables and you being able to leak those variables during pull requests. 281 00:24:07,470 --> 00:24:10,409 What are your suggestions to mitigate that? 282 00:24:10,409 --> 00:24:15,220 A: Don't give people write access to your repository, not even to branches that you 283 00:24:15,220 --> 00:24:18,710 don't trust. So, as soon as they have write access, they would also know the 284 00:24:18,710 --> 00:24:23,409 secrets behind the variables in this case. I like the security model, because if you 285 00:24:23,409 --> 00:24:26,779 for example get contributions from outside, nobody can trigger that and steal 286 00:24:26,779 --> 00:24:29,250 your keys. But as soon as you build it on your own 287 00:24:29,250 --> 00:24:32,440 branch somewhere in the repository that changes. 288 00:24:32,440 --> 00:24:38,940 Q: Yes, but if you submit a pull request, you don't necessarily have to have write 289 00:24:38,940 --> 00:24:43,460 access to that repository. A: Yes, that's what I mean. If you come 290 00:24:43,460 --> 00:24:48,240 from outside and it's not within the same repository, the secrets are not decrypted. 291 00:24:48,240 --> 00:24:52,350 So you can't run the steps. For example, you would not like to deploy directly from 292 00:24:52,350 --> 00:24:57,759 a foreign branch, somewhere. Herald: We have a question from microphone 293 00:24:57,759 --> 00:25:02,700 number four. Q: You mentioned the problem with 294 00:25:02,700 --> 00:25:07,409 different compartments and how to exchange those environments without people. I think 295 00:25:07,409 --> 00:25:10,980 that problem has already been solved with Vagrant and some kind of provisioning 296 00:25:10,980 --> 00:25:14,899 software like Ansible. Do you have any experience with checking 297 00:25:14,899 --> 00:25:20,480 those results of those Vagrant boxes that are automatically provisioned, like 298 00:25:20,480 --> 00:25:25,830 having some server spec software to check those environments afterwards, or having 299 00:25:25,830 --> 00:25:31,269 some kind of hashing, how to find out, if they have been reproduced the same way and 300 00:25:31,269 --> 00:25:37,039 or if they have been any exploit used in that process of setting up the Vagrant 301 00:25:37,039 --> 00:25:40,250 environments. A: Yes, so different levels you can look 302 00:25:40,250 --> 00:25:46,150 at this. There was some, ..., I try to find it, yes. You can for example memory 303 00:25:46,150 --> 00:25:50,120 dump at any time, if you have the hosts trying somewhere or was your question 304 00:25:50,120 --> 00:25:55,150 exactly that you want to check, if your environment that was spun up was not 305 00:25:55,150 --> 00:25:58,870 compromised yet? Q: Yeah, there has to be some kind of 306 00:25:58,870 --> 00:26:05,830 process, how to verify that the produced environments are the ones you expect them 307 00:26:05,830 --> 00:26:10,990 to be, or if they have been compromised and the problem is, I have used those 308 00:26:10,990 --> 00:26:16,610 environments and tried, first I tried the full disk encryption for the Vagrant boxes 309 00:26:16,610 --> 00:26:26,960 but the problem is, it's always the same key for the encryption, so that doesn't 310 00:26:26,960 --> 00:26:30,530 work and even as you mentioned, you can have a memory dump, so you can read out 311 00:26:30,530 --> 00:26:37,110 that key so there's no real possibility to set up a Vagrant box that can't be 312 00:26:37,110 --> 00:26:46,350 tampered with afterwards. So there has to be some kind of hash sum to compare those 313 00:26:46,350 --> 00:26:49,151 produced results. A: Yeah, so as soon as you have a 314 00:26:49,151 --> 00:26:53,350 reproducible build and the result that you, for example script languages are much 315 00:26:53,350 --> 00:26:57,650 easier to achieve, because then you can just diff the filesystem directory and 316 00:26:57,650 --> 00:27:02,030 see, if there was some change. What I would do in this case, is to run multiple 317 00:27:02,030 --> 00:27:05,559 services and compare the results, if that's possible. For example, you have 318 00:27:05,559 --> 00:27:10,630 these reproducible builds, then run it on a few servers which are independent and 319 00:27:10,630 --> 00:27:14,500 compare what you have. Herald: We have two more questions from 320 00:27:14,500 --> 00:27:17,960 microphone number one and only a few minutes left. Microphone number one. 321 00:27:17,960 --> 00:27:22,770 Q: So, what's your recommendation for handling credentials in application 322 00:27:22,770 --> 00:27:28,789 configuration files? We need often some database user and password or something 323 00:27:28,789 --> 00:27:33,840 like this in, say Spring Boot Application YML, or things like that? 324 00:27:33,840 --> 00:27:40,640 And is there any best practice or any framework which can handle such things or 325 00:27:40,640 --> 00:27:46,919 we need to explicitly encrypt these credentials in this application and then 326 00:27:46,919 --> 00:27:51,710 decrypt for itself in the application, but then you need symmetric keys, or? 327 00:27:51,710 --> 00:27:56,250 A: Yes, so Ansible for example comes with a mechanism that's called Ansible Vault, 328 00:27:56,250 --> 00:28:00,000 which encrypts that with a passphrase that you can enter in your command line as soon 329 00:28:00,000 --> 00:28:03,990 as you touch the file. For example, if you want to run Ansible then, it will ask you 330 00:28:03,990 --> 00:28:07,679 for the password when starting up. So, if you want to share that password with your 331 00:28:07,679 --> 00:28:11,220 developers everybody has access to the same keys, I would prefer to give 332 00:28:11,220 --> 00:28:16,000 everybody, so every person in this team or even every device a different key, if 333 00:28:16,000 --> 00:28:21,100 that's possible somehow. That's what I was trying to mention with 334 00:28:21,100 --> 00:28:24,830 the GitHub accounts; that you don't use one GitHub account but you use many of 335 00:28:24,830 --> 00:28:27,049 them. If you, ... yeah. 336 00:28:28,099 --> 00:28:31,860 Herald: We have one more question from microphone number one and then a question 337 00:28:31,860 --> 00:28:35,090 from the internet. Q: Yeah, my question was more about, I 338 00:28:35,090 --> 00:28:38,919 mean some of your recommendations are low- hanging fruits, but some of them it's 339 00:28:38,919 --> 00:28:42,850 like, it's just impossible. I mean it's not sustainable, like it's very hard to 340 00:28:42,850 --> 00:28:46,960 maintain and so I'm wondering, if you use all of them every day or just part of them 341 00:28:46,960 --> 00:28:50,190 or do you just leave like an ??? at the end? 342 00:28:50,190 --> 00:28:54,399 A: It depends on the project. So what I try to do on my development system is to 343 00:28:54,399 --> 00:28:59,280 have these compartment, so that one compromised project would not affect 344 00:28:59,280 --> 00:29:04,289 others. Because I'm not the only person checking and merging the code, so and 345 00:29:04,289 --> 00:29:08,820 that's something that gets quickly too much for one person to review. So, I 346 00:29:08,820 --> 00:29:13,150 can't review all the code that I'm running currently on my computer, that's true. But 347 00:29:13,150 --> 00:29:16,540 I can try to mitigate what the impact of this will be. 348 00:29:16,890 --> 00:29:20,617 Herald: And the question from the internet. Signal: What tool would you recommend 349 00:29:20,617 --> 00:29:31,850 for diffing a file system? A: Diff. *giggles* Well, it worked for me 350 00:29:31,850 --> 00:29:38,620 so far. Or what exactly is the question about? Um, maybe you want to see, if did 351 00:29:38,620 --> 00:29:42,179 the hash change in the files? So when you have e.g. the script file one and 352 00:29:42,179 --> 00:29:45,669 the script file B and they have a different hash sum, that's something I 353 00:29:45,669 --> 00:29:49,760 would consider something I would look up manually. So as soon as I have an 354 00:29:49,760 --> 00:29:53,440 indication that there was something wrong, I would look it up manually and use any 355 00:29:53,440 --> 00:29:57,059 tool that I have. Hex editor or whatever is available. 356 00:29:57,059 --> 00:30:01,200 Herald: Good. We have less than one minute left. Are there any final remarks? 357 00:30:01,200 --> 00:30:03,620 Stephan: Thank you. Herald: Thank you very much. 358 00:30:03,620 --> 00:30:05,927 *Applause* 359 00:30:10,147 --> 00:30:15,860 *postroll music* 360 00:30:15,860 --> 00:30:28,000 subtitles created by c3subtitles.de in the year 2019. Join, and help us!