0 00:00:00,000 --> 00:00:30,000 Dear viewer, these subtitles were generated by a machine via the service Trint and therefore are (very) buggy. If you are capable, please help us to create good quality subtitles: https://c3subtitles.de/talk/684 Thanks! 1 00:00:15,160 --> 00:00:17,229 OK, our next talk is going 2 00:00:17,230 --> 00:00:19,629 to be by her book 3 00:00:19,630 --> 00:00:21,729 in search of evidence based 4 00:00:21,730 --> 00:00:23,829 I.T. Security, and he wants 5 00:00:23,830 --> 00:00:25,779 to do most of the introduction himself. 6 00:00:25,780 --> 00:00:27,819 So this is a very short and brief moment 7 00:00:27,820 --> 00:00:29,069 for me up on stage. 8 00:00:29,070 --> 00:00:30,760 Enjoy and give it up for Hanno. 9 00:00:38,910 --> 00:00:41,189 Yeah, hello, so I 10 00:00:41,190 --> 00:00:43,589 cannot work I am working as a journalist, 11 00:00:43,590 --> 00:00:45,659 and I like to say 12 00:00:45,660 --> 00:00:47,759 I like to avoid the term security 13 00:00:47,760 --> 00:00:49,949 researcher and I hope during my 14 00:00:49,950 --> 00:00:51,299 target will become obvious. 15 00:00:51,300 --> 00:00:52,869 Why that's the case. 16 00:00:52,870 --> 00:00:55,379 Um, I write articles mostly 17 00:00:55,380 --> 00:00:57,779 for Gulam, uh, usually about 18 00:00:57,780 --> 00:00:59,429 I.T. security topics. 19 00:00:59,430 --> 00:01:01,859 And I ran the fuzzing project where 20 00:01:01,860 --> 00:01:03,929 I try to improve the security of free 21 00:01:03,930 --> 00:01:05,249 and open source software. 22 00:01:05,250 --> 00:01:06,869 And this is funded by the Linux 23 00:01:06,870 --> 00:01:08,429 Foundation's Core Infrastructure 24 00:01:08,430 --> 00:01:09,479 Initiative. 25 00:01:09,480 --> 00:01:11,489 And I also write a monthly newsletter 26 00:01:11,490 --> 00:01:12,490 about this. 27 00:01:14,010 --> 00:01:16,079 So, um, as I work 28 00:01:16,080 --> 00:01:18,239 in I.T. Security, I occasionally go 29 00:01:18,240 --> 00:01:20,489 to security conferences and 30 00:01:20,490 --> 00:01:22,559 not just conferences like this one, 31 00:01:22,560 --> 00:01:24,779 but also conferences where you have 32 00:01:24,780 --> 00:01:26,909 a vendor area where people are trying to 33 00:01:26,910 --> 00:01:28,769 sell it, security products. 34 00:01:30,180 --> 00:01:32,219 Um, so I have a few pictures here. 35 00:01:32,220 --> 00:01:34,169 Here someone is selling Next Generation 36 00:01:34,170 --> 00:01:35,339 Apte defense 37 00:01:37,170 --> 00:01:39,329 because, um, someone is selling 38 00:01:39,330 --> 00:01:41,459 something with artificial intelligence. 39 00:01:42,540 --> 00:01:45,089 Um, someone is asking if everything 40 00:01:45,090 --> 00:01:46,409 is moving to the cloud. 41 00:01:46,410 --> 00:01:47,819 Why isn't your security. 42 00:01:51,970 --> 00:01:54,099 And this vendor is saying the 43 00:01:54,100 --> 00:01:56,619 only vendor with guaranteed protection 44 00:01:56,620 --> 00:01:57,620 from ransomware. 45 00:01:59,800 --> 00:02:02,289 And when I see these things, 46 00:02:02,290 --> 00:02:04,899 I am a bit skeptical. 47 00:02:04,900 --> 00:02:07,149 Um, I'm not sure I 48 00:02:07,150 --> 00:02:09,189 feel many of the towns don't have a real 49 00:02:09,190 --> 00:02:09,788 meaning. 50 00:02:09,789 --> 00:02:11,409 They feel like marketing terms. 51 00:02:11,410 --> 00:02:13,449 I don't really know what they are doing 52 00:02:13,450 --> 00:02:15,549 or if if I know what they're 53 00:02:15,550 --> 00:02:18,519 doing, it doesn't feel right. 54 00:02:18,520 --> 00:02:20,739 Um, and I'm not 55 00:02:20,740 --> 00:02:22,659 the only person skeptical about its 56 00:02:22,660 --> 00:02:24,669 security products, so I don't know if, 57 00:02:24,670 --> 00:02:26,679 you know, this guy was Travis Omondi. 58 00:02:26,680 --> 00:02:28,899 He's working for Google and lately 59 00:02:28,900 --> 00:02:31,239 he's been looking at security 60 00:02:31,240 --> 00:02:32,289 products. 61 00:02:32,290 --> 00:02:34,779 And what he found was that many security 62 00:02:34,780 --> 00:02:37,089 products are not very secure. 63 00:02:37,090 --> 00:02:39,309 So, for example, he found 64 00:02:39,310 --> 00:02:41,439 out that Hamas was using some open 65 00:02:41,440 --> 00:02:43,669 source code and they replaced the 66 00:02:43,670 --> 00:02:46,149 stolen copy with copy 67 00:02:46,150 --> 00:02:48,219 and introduced a buffer overflow. 68 00:02:48,220 --> 00:02:49,629 But it's faster. 69 00:02:49,630 --> 00:02:51,239 So great. 70 00:02:51,240 --> 00:02:53,419 Um, Trent Microaggression 71 00:02:53,420 --> 00:02:55,419 did accidentally left a remote debugging 72 00:02:55,420 --> 00:02:57,909 server running, um, 73 00:02:57,910 --> 00:02:59,859 or Palo Alto Network had a memory of 74 00:02:59,860 --> 00:03:01,479 corruption because they shipped a Web 75 00:03:01,480 --> 00:03:04,009 server that was no longer supported. 76 00:03:04,010 --> 00:03:06,309 Um, and here he was trying to 77 00:03:06,310 --> 00:03:07,539 contact Avichai. 78 00:03:07,540 --> 00:03:09,669 And also your code makes zero 79 00:03:09,670 --> 00:03:11,149 sense. 80 00:03:11,150 --> 00:03:13,629 Um, yes. 81 00:03:13,630 --> 00:03:15,969 And we have headlines like this 82 00:03:15,970 --> 00:03:18,429 for, uh, PC versus antivirus 83 00:03:18,430 --> 00:03:20,589 software could make your company more 84 00:03:20,590 --> 00:03:21,609 vulnerable. 85 00:03:21,610 --> 00:03:23,889 And on the upper right, 86 00:03:23,890 --> 00:03:26,099 antivirus tools are useless. 87 00:03:26,100 --> 00:03:27,669 Box ticking exercise. 88 00:03:27,670 --> 00:03:29,949 First, Google security check. 89 00:03:29,950 --> 00:03:32,169 And here are two 90 00:03:32,170 --> 00:03:34,419 tweets where there was recently a quite 91 00:03:34,420 --> 00:03:35,949 heated debate about the value of 92 00:03:35,950 --> 00:03:37,229 antivirus software. 93 00:03:37,230 --> 00:03:39,879 A trust issue is a Chrome developer. 94 00:03:39,880 --> 00:03:43,149 He compared antivirus to homoeopathy. 95 00:03:43,150 --> 00:03:45,369 And, um, April 96 00:03:45,370 --> 00:03:47,819 King, who is from Firefox, said antivirus 97 00:03:47,820 --> 00:03:50,049 because of security issues for Firefox. 98 00:03:51,170 --> 00:03:54,429 Um, and this is from, uh, 99 00:03:54,430 --> 00:03:57,039 where Google asked, um, 100 00:03:57,040 --> 00:03:58,749 users and I.T. 101 00:03:58,750 --> 00:04:01,359 security experts what they think is 102 00:04:01,360 --> 00:04:03,699 are the most important things to do about 103 00:04:03,700 --> 00:04:04,989 it. Security. 104 00:04:04,990 --> 00:04:06,999 And you can see the users had antivirus 105 00:04:07,000 --> 00:04:09,669 software as the very first thing. 106 00:04:09,670 --> 00:04:11,859 Uh, and the security experts don't seem 107 00:04:11,860 --> 00:04:13,179 to think that's so important. 108 00:04:13,180 --> 00:04:15,879 It doesn't even show up in the top five. 109 00:04:15,880 --> 00:04:16,880 So 110 00:04:18,130 --> 00:04:20,379 we can conclude that there's considerable 111 00:04:20,380 --> 00:04:22,479 disagreement, whether it is security 112 00:04:22,480 --> 00:04:24,279 products and especially antivirus 113 00:04:24,280 --> 00:04:27,189 software is actually a good idea. 114 00:04:27,190 --> 00:04:28,720 Um, so 115 00:04:30,220 --> 00:04:32,229 how do we know actually if these things 116 00:04:32,230 --> 00:04:34,419 work and 117 00:04:34,420 --> 00:04:37,089 to investigate, then I'd like to 118 00:04:37,090 --> 00:04:38,499 talk about something completely 119 00:04:38,500 --> 00:04:39,909 different. 120 00:04:39,910 --> 00:04:42,159 So this industry likes 121 00:04:42,160 --> 00:04:43,939 to use medical analogies. 122 00:04:43,940 --> 00:04:45,669 We're talking about viruses. 123 00:04:45,670 --> 00:04:48,309 Viruses are usually something 124 00:04:48,310 --> 00:04:50,470 from medicine which affects people. 125 00:04:52,120 --> 00:04:54,939 And here's another form of antivirus. 126 00:04:54,940 --> 00:04:57,399 It's a vitamin C pill. 127 00:04:57,400 --> 00:04:59,709 And here's a person having a common cold 128 00:04:59,710 --> 00:05:00,849 sneezing. 129 00:05:00,850 --> 00:05:02,979 And yeah, many people think 130 00:05:02,980 --> 00:05:04,659 it's a good idea if you have a common 131 00:05:04,660 --> 00:05:06,999 cold that you should take vitamin 132 00:05:07,000 --> 00:05:08,000 C pills. 133 00:05:08,920 --> 00:05:11,649 Um, unfortunately, 134 00:05:11,650 --> 00:05:13,629 it's probably not very useful. 135 00:05:14,920 --> 00:05:17,049 Um, and why 136 00:05:17,050 --> 00:05:19,239 do we actually know that? 137 00:05:19,240 --> 00:05:21,309 Um, obviously we know this 138 00:05:21,310 --> 00:05:23,559 because we have science. 139 00:05:23,560 --> 00:05:25,389 We're using science to investigate 140 00:05:25,390 --> 00:05:26,390 whether things work. 141 00:05:27,910 --> 00:05:30,489 And for the vitamin C, 142 00:05:30,490 --> 00:05:33,099 here's some quotes from a study 143 00:05:33,100 --> 00:05:34,839 which says, OK, regular ingestion of 144 00:05:34,840 --> 00:05:36,939 vitamin C had no effect on common 145 00:05:36,940 --> 00:05:38,469 cold incidents in the ordinary 146 00:05:38,470 --> 00:05:39,519 population. 147 00:05:39,520 --> 00:05:42,129 So if you're like an average adult person 148 00:05:42,130 --> 00:05:44,379 and take vitamin C, 149 00:05:44,380 --> 00:05:46,479 you're just as likely to get a cold than 150 00:05:46,480 --> 00:05:48,189 everybody else. 151 00:05:48,190 --> 00:05:50,499 However, it may be that it broadens 152 00:05:50,500 --> 00:05:53,049 the duration of your cold a little bit. 153 00:05:53,050 --> 00:05:55,239 But if you take the vitamin C only once 154 00:05:55,240 --> 00:05:57,369 you already got a cold and it has 155 00:05:57,370 --> 00:05:58,370 no use at all. 156 00:06:00,130 --> 00:06:02,199 And the study here is from the 157 00:06:02,200 --> 00:06:04,269 Cochrane Collaboration, which is an 158 00:06:04,270 --> 00:06:06,789 organization that's doing a so-called 159 00:06:06,790 --> 00:06:07,899 meta analysis. 160 00:06:07,900 --> 00:06:09,759 And I will come back later what that is. 161 00:06:09,760 --> 00:06:12,429 But it's generally an organization. 162 00:06:12,430 --> 00:06:14,289 I think there's widespread agreement that 163 00:06:14,290 --> 00:06:16,449 the Cochrane Collaboration is creating 164 00:06:16,450 --> 00:06:18,909 some of the highest quality scientific 165 00:06:18,910 --> 00:06:20,139 evidence in medicine. 166 00:06:22,500 --> 00:06:24,599 Um, so if we want to 167 00:06:24,600 --> 00:06:26,729 know if, uh, medicine 168 00:06:26,730 --> 00:06:28,499 or also like something like a food 169 00:06:28,500 --> 00:06:30,749 supplement, like about pill works, 170 00:06:30,750 --> 00:06:32,969 what's usually done is a so-called 171 00:06:32,970 --> 00:06:35,009 randomized controlled trial. 172 00:06:35,010 --> 00:06:37,079 And that is where you take a group of 173 00:06:37,080 --> 00:06:39,779 people that may have some 174 00:06:39,780 --> 00:06:41,009 some illness. 175 00:06:41,010 --> 00:06:42,989 And then we split them randomly into 176 00:06:42,990 --> 00:06:45,509 groups. And it's it's crucial 177 00:06:45,510 --> 00:06:47,669 that this is done randomly because we 178 00:06:47,670 --> 00:06:49,799 don't want to have some 179 00:06:49,800 --> 00:06:51,929 statistical thing that we chose one 180 00:06:51,930 --> 00:06:54,239 group that is maybe 181 00:06:54,240 --> 00:06:56,069 more sick than the other group to begin 182 00:06:56,070 --> 00:06:58,589 with and then screws 183 00:06:58,590 --> 00:07:00,449 with our results. So we need to split 184 00:07:00,450 --> 00:07:02,219 them randomly into groups. 185 00:07:02,220 --> 00:07:04,289 And a simple way would be, OK, 186 00:07:04,290 --> 00:07:06,119 one group gets medication, the other 187 00:07:06,120 --> 00:07:07,829 group gets a placebo, and then we see 188 00:07:07,830 --> 00:07:08,830 what happens. 189 00:07:09,630 --> 00:07:12,059 In reality, it's usually more complicated 190 00:07:12,060 --> 00:07:14,189 because usually we have a situation 191 00:07:14,190 --> 00:07:15,929 where we already have a known good 192 00:07:15,930 --> 00:07:18,419 medication and we have a new medication 193 00:07:18,420 --> 00:07:19,859 and we just want to know if the new 194 00:07:19,860 --> 00:07:21,539 medication is better. 195 00:07:21,540 --> 00:07:24,419 So we compare an old and a new medication 196 00:07:24,420 --> 00:07:26,609 and we also may have an 197 00:07:26,610 --> 00:07:28,799 alternative to medication like 198 00:07:28,800 --> 00:07:31,049 exercise or dietary changes. 199 00:07:31,050 --> 00:07:33,179 And you may want to know, OK, maybe 200 00:07:33,180 --> 00:07:35,159 we have a medication that works, but 201 00:07:35,160 --> 00:07:37,349 doing exercise works even better and 202 00:07:37,350 --> 00:07:39,509 maybe taking both the medication and 203 00:07:39,510 --> 00:07:41,429 exercise at the same time works even more 204 00:07:41,430 --> 00:07:43,799 better. Um, but this the general 205 00:07:43,800 --> 00:07:46,079 idea. So we randomly split people 206 00:07:46,080 --> 00:07:48,029 into groups and test what happens, 207 00:07:49,920 --> 00:07:52,019 but then we usually don't really care 208 00:07:52,020 --> 00:07:54,149 about a single study because they are far 209 00:07:54,150 --> 00:07:56,009 too many things that can go wrong. 210 00:07:56,010 --> 00:07:58,169 So what we usually care about is all 211 00:07:58,170 --> 00:08:00,119 the scientific evidence we have as a 212 00:08:00,120 --> 00:08:02,249 whole. And that's why we're doing a 213 00:08:02,250 --> 00:08:04,319 meta analysis, which is we're 214 00:08:04,320 --> 00:08:06,149 trying to search for all the studies that 215 00:08:06,150 --> 00:08:07,979 have been done on a particular topic, 216 00:08:07,980 --> 00:08:10,409 ideally randomized control trials, 217 00:08:10,410 --> 00:08:12,119 and we combine the results. 218 00:08:12,120 --> 00:08:14,369 This is obviously sometimes 219 00:08:14,370 --> 00:08:16,949 complicated because we might have studies 220 00:08:16,950 --> 00:08:20,159 for different groups of the population. 221 00:08:20,160 --> 00:08:22,379 They cannot always be easily compared, 222 00:08:22,380 --> 00:08:23,789 but that's the ideal idea. 223 00:08:23,790 --> 00:08:25,859 So we have many studies and then 224 00:08:25,860 --> 00:08:27,869 we combine the results and look at the 225 00:08:27,870 --> 00:08:29,310 whole body of evidence. 226 00:08:31,380 --> 00:08:33,869 Um, yeah. So we call that evidence 227 00:08:33,870 --> 00:08:34,779 based medicine. 228 00:08:34,780 --> 00:08:36,869 So ideally, we want 229 00:08:36,870 --> 00:08:38,699 to make all decisions based on high 230 00:08:38,700 --> 00:08:40,769 quality scientific evidence, 231 00:08:40,770 --> 00:08:42,928 which very often a meta 232 00:08:42,929 --> 00:08:44,750 analysis qualifies. 233 00:08:46,630 --> 00:08:48,759 And now 234 00:08:48,760 --> 00:08:51,099 I want to point out that 235 00:08:51,100 --> 00:08:53,259 one shouldn't have a too idealized view 236 00:08:53,260 --> 00:08:55,059 on science because there are many 237 00:08:55,060 --> 00:08:57,000 problems, um, 238 00:08:58,150 --> 00:09:00,849 here. The top one is actually 239 00:09:00,850 --> 00:09:03,069 the most popular open access paper 240 00:09:03,070 --> 00:09:05,379 of all times, which is why most 241 00:09:05,380 --> 00:09:07,749 published research findings are false. 242 00:09:07,750 --> 00:09:10,119 It has been published in 2005, 243 00:09:10,120 --> 00:09:11,919 and that's actually not very 244 00:09:11,920 --> 00:09:15,369 controversial. So, um, 245 00:09:15,370 --> 00:09:17,529 and the middle one points to an issue 246 00:09:17,530 --> 00:09:19,839 that has been debated in recent years 247 00:09:19,840 --> 00:09:21,999 a lot, where there was 248 00:09:22,000 --> 00:09:24,159 a big experiment to try to replicate 249 00:09:24,160 --> 00:09:26,139 studies in psychology. 250 00:09:26,140 --> 00:09:28,269 And they found out that they were 251 00:09:28,270 --> 00:09:30,699 only able to replicate 252 00:09:30,700 --> 00:09:33,639 the result of 37 percent of the studies. 253 00:09:33,640 --> 00:09:35,919 So the majority of studies, it seems 254 00:09:35,920 --> 00:09:38,199 either they are wrong or the replication 255 00:09:38,200 --> 00:09:39,309 was wrong. 256 00:09:39,310 --> 00:09:41,619 But it seems there's a problem. 257 00:09:41,620 --> 00:09:43,179 But this is not only affecting 258 00:09:43,180 --> 00:09:44,469 psychology. 259 00:09:44,470 --> 00:09:46,389 The you have the same problem in many 260 00:09:46,390 --> 00:09:47,439 fields of sciences. 261 00:09:47,440 --> 00:09:48,849 For example, there's a very similar 262 00:09:48,850 --> 00:09:51,309 debate in cancer research. 263 00:09:51,310 --> 00:09:53,319 Um, and the lowest one points to 264 00:09:53,320 --> 00:09:55,479 something that many clinical trials 265 00:09:55,480 --> 00:09:57,549 findings never get published, which 266 00:09:57,550 --> 00:09:59,139 is also a very important thing to 267 00:09:59,140 --> 00:10:01,299 consider, that the science we're 268 00:10:01,300 --> 00:10:03,429 seeing is not all the science that 269 00:10:03,430 --> 00:10:04,659 has happened. 270 00:10:04,660 --> 00:10:06,549 We very often have a situation where 271 00:10:06,550 --> 00:10:09,069 people do a study and then 272 00:10:09,070 --> 00:10:11,229 based on the result, they decide 273 00:10:11,230 --> 00:10:12,939 whether it's interesting and gets 274 00:10:12,940 --> 00:10:15,189 published or whether it just gets 275 00:10:15,190 --> 00:10:16,349 thrown away. 276 00:10:16,350 --> 00:10:18,099 So, yeah. 277 00:10:20,420 --> 00:10:22,489 So if you want to 278 00:10:22,490 --> 00:10:25,369 evaluate what's good or bad science, 279 00:10:25,370 --> 00:10:27,499 then there are some things we 280 00:10:27,500 --> 00:10:28,459 can look at. 281 00:10:28,460 --> 00:10:30,319 Something very obvious is if we have a 282 00:10:30,320 --> 00:10:32,779 very small number of research subjects. 283 00:10:32,780 --> 00:10:34,939 So sometimes you see studies where people 284 00:10:34,940 --> 00:10:36,829 say, OK, we've tested this with 10 285 00:10:36,830 --> 00:10:37,879 people. 286 00:10:37,880 --> 00:10:40,099 And I say, OK, that's maybe not very 287 00:10:40,100 --> 00:10:42,289 meaningful. It could be just coincidence. 288 00:10:42,290 --> 00:10:43,850 Could be a statistical glitch 289 00:10:45,470 --> 00:10:47,569 then, which is a very common thing, not 290 00:10:47,570 --> 00:10:49,639 only with the quality of the science 291 00:10:49,640 --> 00:10:52,069 itself, but also with the reporting 292 00:10:52,070 --> 00:10:53,719 about science, like when the media 293 00:10:53,720 --> 00:10:55,100 reports about it, is that 294 00:10:56,240 --> 00:10:58,429 correlations are reported as 295 00:10:58,430 --> 00:11:00,079 if they were causal results. 296 00:11:00,080 --> 00:11:02,329 So what's happening here is if 297 00:11:02,330 --> 00:11:04,489 we have a set of data and we may find 298 00:11:04,490 --> 00:11:07,399 out all the people who have property 299 00:11:07,400 --> 00:11:09,589 also have property B, then 300 00:11:09,590 --> 00:11:12,499 we could conclude, OK, A causes B, 301 00:11:12,500 --> 00:11:15,529 but it could also be that B causes A, 302 00:11:15,530 --> 00:11:17,629 or it could also be that there's 303 00:11:17,630 --> 00:11:19,609 a so-called confounder, which means we 304 00:11:19,610 --> 00:11:21,739 have something completely different that 305 00:11:21,740 --> 00:11:23,959 we may not even know about that's causing 306 00:11:23,960 --> 00:11:26,899 both A and B, so 307 00:11:26,900 --> 00:11:28,969 and this is generally a problem 308 00:11:28,970 --> 00:11:30,649 in all studies where you're using an 309 00:11:30,650 --> 00:11:32,899 existing dataset and trying to find 310 00:11:32,900 --> 00:11:33,859 something in it. 311 00:11:33,860 --> 00:11:35,209 And that's why we are doing these 312 00:11:35,210 --> 00:11:37,039 controlled trials where we are split in 313 00:11:37,040 --> 00:11:39,709 groups randomly into two groups 314 00:11:39,710 --> 00:11:41,839 so we can exclude that there are some 315 00:11:41,840 --> 00:11:44,059 some other factor that's happening 316 00:11:44,060 --> 00:11:45,060 here. 317 00:11:46,520 --> 00:11:48,649 Um, and then, yeah, sometimes we 318 00:11:48,650 --> 00:11:50,509 only have a single study or very few 319 00:11:50,510 --> 00:11:51,469 studies. 320 00:11:51,470 --> 00:11:53,719 So we usually want 321 00:11:53,720 --> 00:11:55,789 good science to be based on many 322 00:11:55,790 --> 00:11:58,189 studies. We want science to be replicated 323 00:11:58,190 --> 00:11:59,190 independently. 324 00:12:00,200 --> 00:12:02,419 Um, and 325 00:12:02,420 --> 00:12:03,799 then we have a thing that's called 326 00:12:03,800 --> 00:12:05,059 publication bias. 327 00:12:05,060 --> 00:12:06,799 And that's what I mentioned earlier, is 328 00:12:06,800 --> 00:12:08,929 we don't see all the studies that are 329 00:12:08,930 --> 00:12:10,369 done. 330 00:12:10,370 --> 00:12:12,559 And we may have a situation where 331 00:12:12,560 --> 00:12:15,049 a pharmaceutical company makes a trial 332 00:12:15,050 --> 00:12:17,329 on a medication and it turns 333 00:12:17,330 --> 00:12:19,399 out the medication doesn't really help 334 00:12:19,400 --> 00:12:21,799 and then they don't publish the trial 335 00:12:21,800 --> 00:12:23,239 and then they do another trial. 336 00:12:23,240 --> 00:12:25,069 And there it seems like the medication 337 00:12:25,070 --> 00:12:26,480 helps and then they publish it. 338 00:12:27,860 --> 00:12:29,929 And you can see that, like if you only 339 00:12:29,930 --> 00:12:32,059 see the positive studies and you 340 00:12:32,060 --> 00:12:34,669 don't see the negative studies, then 341 00:12:34,670 --> 00:12:36,349 and then you try to combine these 342 00:12:36,350 --> 00:12:38,419 results, like in a meta analysis, 343 00:12:38,420 --> 00:12:39,799 then you get a skewed result. 344 00:12:40,850 --> 00:12:43,039 And another problem is 345 00:12:43,040 --> 00:12:44,569 called outcome switching. 346 00:12:44,570 --> 00:12:46,639 And it's kind of related to fishing for 347 00:12:46,640 --> 00:12:49,039 results, which is you may have collected 348 00:12:49,040 --> 00:12:51,109 some data and but 349 00:12:51,110 --> 00:12:53,029 it doesn't match your theory. 350 00:12:53,030 --> 00:12:54,079 But then you could try. 351 00:12:54,080 --> 00:12:56,239 OK, maybe if I 352 00:12:56,240 --> 00:12:58,429 just use a selection of 353 00:12:58,430 --> 00:13:00,529 my data, I might may 354 00:13:00,530 --> 00:13:02,449 maybe I can prove something similar to my 355 00:13:02,450 --> 00:13:03,469 theory. 356 00:13:03,470 --> 00:13:05,839 And if you look long enough, you can take 357 00:13:05,840 --> 00:13:07,429 some random data and you will find 358 00:13:07,430 --> 00:13:09,049 something that looks like a scientific 359 00:13:09,050 --> 00:13:10,050 result. 360 00:13:11,180 --> 00:13:13,249 So it's not generally a 361 00:13:13,250 --> 00:13:14,689 problem to do this, but it should be 362 00:13:14,690 --> 00:13:16,039 transparent about it. 363 00:13:16,040 --> 00:13:18,229 If you were first searching for something 364 00:13:18,230 --> 00:13:19,699 and then later you're searching for 365 00:13:19,700 --> 00:13:21,889 something else, you should at least make 366 00:13:21,890 --> 00:13:23,629 clear that you did that. 367 00:13:23,630 --> 00:13:26,389 And ideally, you would want all 368 00:13:26,390 --> 00:13:28,399 of these studies that are somehow based 369 00:13:28,400 --> 00:13:30,349 on statistics, that are empirical 370 00:13:30,350 --> 00:13:32,059 studies. You want them to be 371 00:13:32,060 --> 00:13:34,189 preregistered, which would mean that 372 00:13:34,190 --> 00:13:36,109 you would publish before you even start 373 00:13:36,110 --> 00:13:38,179 collecting the data what you're about 374 00:13:38,180 --> 00:13:40,279 to do. So you could say, yeah, I want 375 00:13:40,280 --> 00:13:42,079 to study this medication. 376 00:13:42,080 --> 00:13:44,149 I'll do a randomized control trial with 377 00:13:44,150 --> 00:13:45,829 these groups. 378 00:13:45,830 --> 00:13:47,689 And then I publish that in a trial 379 00:13:47,690 --> 00:13:49,879 register because then if 380 00:13:49,880 --> 00:13:52,819 you later changed what you were studying, 381 00:13:52,820 --> 00:13:54,349 then other people can see that. 382 00:13:54,350 --> 00:13:56,989 So it's transparent, 383 00:13:56,990 --> 00:13:58,849 but we're very far from that in medicine. 384 00:13:58,850 --> 00:14:01,249 This is happening usually it's 385 00:14:01,250 --> 00:14:02,419 still not ideal. 386 00:14:02,420 --> 00:14:03,769 There are still a lot of problems with 387 00:14:03,770 --> 00:14:06,499 this, but in many other fields, this is 388 00:14:06,500 --> 00:14:07,500 not happening at all. 389 00:14:09,170 --> 00:14:10,879 OK, now let's get back to I.T. 390 00:14:10,880 --> 00:14:12,319 Security. 391 00:14:12,320 --> 00:14:14,629 Um, here's an empty slide and 392 00:14:14,630 --> 00:14:16,279 it's not a mistake. It's intentionally 393 00:14:16,280 --> 00:14:18,349 empty because it's also the complete 394 00:14:18,350 --> 00:14:20,689 list of all randomized control trials 395 00:14:20,690 --> 00:14:22,159 that have ever been done on security 396 00:14:22,160 --> 00:14:23,160 software. 397 00:14:30,990 --> 00:14:33,399 Yeah, um, 398 00:14:35,070 --> 00:14:36,689 there are some people who are doing 399 00:14:36,690 --> 00:14:38,879 something that may look a bit like 400 00:14:38,880 --> 00:14:41,190 scientific tests of antivirus software, 401 00:14:42,330 --> 00:14:44,459 but I fear the methodology 402 00:14:44,460 --> 00:14:46,739 that's used there is extremely flawed. 403 00:14:46,740 --> 00:14:48,989 Um, so what they usually do 404 00:14:48,990 --> 00:14:51,539 is they they have a collection 405 00:14:51,540 --> 00:14:53,789 of malware, which is hopefully 406 00:14:53,790 --> 00:14:56,189 somewhat representative of real malware, 407 00:14:56,190 --> 00:14:58,379 and then they try which software 408 00:14:58,380 --> 00:15:00,339 detects it and which not. 409 00:15:00,340 --> 00:15:02,489 Um, this is a lot of problems 410 00:15:02,490 --> 00:15:04,589 because, for example, if you detect them 411 00:15:04,590 --> 00:15:06,749 measure, that does not mean 412 00:15:06,750 --> 00:15:08,729 if you wouldn't detect them ever, that it 413 00:15:08,730 --> 00:15:10,319 would infect the user. 414 00:15:10,320 --> 00:15:12,449 It could be that the ever tried to 415 00:15:12,450 --> 00:15:14,609 use a browser exploit and 416 00:15:14,610 --> 00:15:16,529 the exploit is only in an altered version 417 00:15:16,530 --> 00:15:18,329 of the browser that the user is no longer 418 00:15:18,330 --> 00:15:19,559 using. 419 00:15:19,560 --> 00:15:21,989 And they usually completely fail 420 00:15:21,990 --> 00:15:24,059 to to consider the idea that you 421 00:15:24,060 --> 00:15:26,099 could do something else then antivirus 422 00:15:26,100 --> 00:15:28,529 software to protect yourself, like 423 00:15:28,530 --> 00:15:30,449 you could use regular updates and 424 00:15:30,450 --> 00:15:32,309 application wirelessly. 425 00:15:32,310 --> 00:15:34,409 Um, so these tests kind of 426 00:15:34,410 --> 00:15:36,569 have the idea that antivirus is 427 00:15:36,570 --> 00:15:38,429 the only thing you can do and the only 428 00:15:38,430 --> 00:15:40,409 thing that matters is comparing different 429 00:15:40,410 --> 00:15:42,209 products against each other. 430 00:15:42,210 --> 00:15:44,279 Um, they usually do not consider that 431 00:15:44,280 --> 00:15:45,929 antivirus software itself could be a 432 00:15:45,930 --> 00:15:47,219 security risk. 433 00:15:47,220 --> 00:15:49,529 But most important of all, they 434 00:15:49,530 --> 00:15:51,539 usually don't test with real users. 435 00:15:51,540 --> 00:15:53,099 They are testing in some kind of lab 436 00:15:53,100 --> 00:15:54,359 condition where they say they are 437 00:15:54,360 --> 00:15:56,579 simulating what a real user is doing, 438 00:15:56,580 --> 00:15:58,349 but they are not testing with your users. 439 00:15:59,740 --> 00:16:02,019 Um, and it's it's quite widespread. 440 00:16:02,020 --> 00:16:04,139 That is the question of forms of 441 00:16:04,140 --> 00:16:05,429 statistics in I.T. 442 00:16:05,430 --> 00:16:06,489 security. 443 00:16:06,490 --> 00:16:08,669 Uh, one very notorious example is also 444 00:16:08,670 --> 00:16:10,989 CV counting when people say so 445 00:16:10,990 --> 00:16:13,349 severe is ah, I don't know if everybody 446 00:16:13,350 --> 00:16:15,299 knows that these are identifiers for 447 00:16:15,300 --> 00:16:17,279 security vulnerabilities. 448 00:16:17,280 --> 00:16:19,589 And what some people tend to do is say, 449 00:16:19,590 --> 00:16:21,929 OK, Windows had that many CBD, Linux 450 00:16:21,930 --> 00:16:23,069 had that many CBD. 451 00:16:23,070 --> 00:16:24,959 So clearly Windows is more secure than 452 00:16:24,960 --> 00:16:26,069 Linux. 453 00:16:26,070 --> 00:16:27,949 This is completely flawed because these 454 00:16:27,950 --> 00:16:30,449 severe identifiers don't 455 00:16:30,450 --> 00:16:32,489 even try to be complete. 456 00:16:32,490 --> 00:16:34,619 And if you don't believe me, 457 00:16:34,620 --> 00:16:36,659 there's a talk from the guy who invented 458 00:16:36,660 --> 00:16:38,189 the CVA I.D. 459 00:16:38,190 --> 00:16:39,779 where he thinks you shouldn't do these 460 00:16:39,780 --> 00:16:41,379 kinds of statistics. 461 00:16:41,380 --> 00:16:42,380 Um. 462 00:16:44,180 --> 00:16:45,180 So 463 00:16:46,250 --> 00:16:48,469 my feeling is that security 464 00:16:48,470 --> 00:16:50,569 is largely not bias based on scientific 465 00:16:50,570 --> 00:16:53,059 evidence, and it's a bit of something 466 00:16:53,060 --> 00:16:54,799 that bothers me because I work in I.T. 467 00:16:54,800 --> 00:16:57,559 security and I'm a very scientifically 468 00:16:57,560 --> 00:16:59,719 minded person. So when someone tells 469 00:16:59,720 --> 00:17:01,789 me, hey, this is healthy, then 470 00:17:01,790 --> 00:17:03,679 I say, do you have some studies to show 471 00:17:03,680 --> 00:17:05,539 me? And if you don't have to study it, 472 00:17:05,540 --> 00:17:07,068 then I don't believe it. 473 00:17:07,069 --> 00:17:08,749 And at the same time, I'm working in a 474 00:17:08,750 --> 00:17:10,489 field where if I ask this question, the 475 00:17:10,490 --> 00:17:12,799 answer is just the evidence is not there 476 00:17:12,800 --> 00:17:13,800 very often. 477 00:17:14,930 --> 00:17:17,088 Um, and now you 478 00:17:17,089 --> 00:17:18,979 might say, OK, but aren't there plenty of 479 00:17:18,980 --> 00:17:21,169 scientific papers and conferences on I.T. 480 00:17:21,170 --> 00:17:22,170 security? 481 00:17:23,089 --> 00:17:25,249 And, um, here's 482 00:17:25,250 --> 00:17:28,279 a list of some of the most cited papers 483 00:17:28,280 --> 00:17:29,989 and a quick remark on that. 484 00:17:29,990 --> 00:17:32,299 Like counting the citations of papers 485 00:17:32,300 --> 00:17:34,519 itself is a very controversial 486 00:17:34,520 --> 00:17:35,520 thing, 487 00:17:36,650 --> 00:17:38,749 but I cannot go into that. 488 00:17:38,750 --> 00:17:40,699 But there's a whole debate about whether 489 00:17:40,700 --> 00:17:42,229 you should use something like an impact 490 00:17:42,230 --> 00:17:43,880 factor or whether that's a bad idea. 491 00:17:45,200 --> 00:17:47,659 But at least I think it tells us, which 492 00:17:47,660 --> 00:17:49,729 are the scientific papers that other 493 00:17:49,730 --> 00:17:51,049 scientists care about. 494 00:17:52,220 --> 00:17:54,589 And this is a list from Google Scholar, 495 00:17:54,590 --> 00:17:57,470 from papers from 2012 and 2013. 496 00:17:58,910 --> 00:18:01,309 So the first one here says Candidate 497 00:18:01,310 --> 00:18:03,259 indistinguishably obfuscation and 498 00:18:03,260 --> 00:18:05,359 functional encryption for autocrat's. 499 00:18:06,830 --> 00:18:09,799 Now, I could ask if we have an 500 00:18:09,800 --> 00:18:12,289 average user who is using the Internet, 501 00:18:12,290 --> 00:18:14,479 writing emails, using a Web browser, 502 00:18:14,480 --> 00:18:16,219 using Facebook, whatever. 503 00:18:16,220 --> 00:18:18,409 Um, how does this matter for 504 00:18:18,410 --> 00:18:19,639 him? 505 00:18:19,640 --> 00:18:21,409 If you have an answer for that, I would 506 00:18:21,410 --> 00:18:23,300 really like to hear you talk to me later. 507 00:18:25,550 --> 00:18:27,169 And I think you could ask a similar 508 00:18:27,170 --> 00:18:28,849 question for all of these papers. 509 00:18:30,230 --> 00:18:32,479 Um, I had to go to 510 00:18:32,480 --> 00:18:34,609 No.11 where I found something 511 00:18:34,610 --> 00:18:36,739 that sounded like it was actually about 512 00:18:36,740 --> 00:18:37,969 real software. 513 00:18:37,970 --> 00:18:40,879 That was a paper about Android malware. 514 00:18:40,880 --> 00:18:43,099 And also at number twenty, 515 00:18:43,100 --> 00:18:44,959 I found another paper that was about real 516 00:18:44,960 --> 00:18:47,689 software, which was the lucky 13 paper. 517 00:18:47,690 --> 00:18:49,909 And that one made me kind 518 00:18:49,910 --> 00:18:51,769 of question myself, because this is the 519 00:18:51,770 --> 00:18:54,469 kind of paper that I usually care about 520 00:18:54,470 --> 00:18:56,179 because I do a lot of crypto stuff. 521 00:18:56,180 --> 00:18:58,219 And this is a crypto attack. 522 00:18:58,220 --> 00:19:00,409 Um, it's a timing attack. 523 00:19:00,410 --> 00:19:02,419 And it's really hard to pull that attack 524 00:19:02,420 --> 00:19:03,619 off. 525 00:19:03,620 --> 00:19:05,869 Um, and 526 00:19:05,870 --> 00:19:08,329 it's so hard that I am almost certain 527 00:19:08,330 --> 00:19:10,639 that this attack has never been used 528 00:19:10,640 --> 00:19:13,519 in the wild to attack a real user. 529 00:19:13,520 --> 00:19:15,829 But these are the kinds of papers 530 00:19:15,830 --> 00:19:18,169 we find interesting because we say, 531 00:19:18,170 --> 00:19:20,419 oh, they were able to pull off this 532 00:19:20,420 --> 00:19:21,799 interesting attack. 533 00:19:21,800 --> 00:19:23,809 That's great. That questions all the way 534 00:19:23,810 --> 00:19:25,969 how we did encryption 535 00:19:25,970 --> 00:19:28,369 and it had a pretty big impact. 536 00:19:28,370 --> 00:19:30,019 Yeah. I'm also proud that I found a 537 00:19:30,020 --> 00:19:31,429 little mistake in that paper. 538 00:19:31,430 --> 00:19:34,009 Actually, it's not very important, 539 00:19:34,010 --> 00:19:35,389 but so. 540 00:19:35,390 --> 00:19:37,359 Yeah, so I 541 00:19:38,780 --> 00:19:40,909 yeah. But in this whole list with 542 00:19:40,910 --> 00:19:43,099 26 papers that were the most cited 543 00:19:43,100 --> 00:19:45,259 papers, there was not a single paper that 544 00:19:45,260 --> 00:19:47,629 was doing anything with real users. 545 00:19:47,630 --> 00:19:49,489 What they were trying to see what's 546 00:19:49,490 --> 00:19:51,679 happening when real users act with 547 00:19:51,680 --> 00:19:53,119 the Internet, do something about 548 00:19:53,120 --> 00:19:54,229 security. 549 00:19:54,230 --> 00:19:56,059 So it seems the users not really 550 00:19:56,060 --> 00:19:58,429 something a security researcher cares 551 00:19:58,430 --> 00:19:59,430 a lot about. 552 00:20:00,680 --> 00:20:02,929 Um, so my feeling is 553 00:20:02,930 --> 00:20:04,759 most academic research in I.T. 554 00:20:04,760 --> 00:20:07,549 security is comparable to basic research. 555 00:20:07,550 --> 00:20:09,469 When we talk about, I don't know, human 556 00:20:09,470 --> 00:20:11,899 morphic encryption are indistinguishable 557 00:20:11,900 --> 00:20:12,949 obfuscation. 558 00:20:12,950 --> 00:20:14,869 And these are crypto theories that may 559 00:20:14,870 --> 00:20:16,939 lead to some interesting products for the 560 00:20:16,940 --> 00:20:17,940 future. 561 00:20:18,680 --> 00:20:19,609 And that's fine. 562 00:20:19,610 --> 00:20:22,189 I mean, basic research is totally fine, 563 00:20:22,190 --> 00:20:24,139 but I feel we completely missing the 564 00:20:24,140 --> 00:20:25,579 applied research. 565 00:20:25,580 --> 00:20:27,679 And if we do like the more if we 566 00:20:27,680 --> 00:20:29,599 look at the more practical research, I 567 00:20:29,600 --> 00:20:32,669 feel it tends to go into interesting 568 00:20:32,670 --> 00:20:35,089 problems, but not the most important 569 00:20:35,090 --> 00:20:37,219 problems, which is also 570 00:20:37,220 --> 00:20:39,349 kind of fine. But I feel there's 571 00:20:39,350 --> 00:20:40,939 a whole big area we are missing here. 572 00:20:44,870 --> 00:20:47,209 So what would we do 573 00:20:47,210 --> 00:20:48,589 if we would say we want to do a 574 00:20:48,590 --> 00:20:50,869 randomized control trial on a security 575 00:20:50,870 --> 00:20:53,059 software because the OK, get a 576 00:20:53,060 --> 00:20:55,429 large group of users and randomly 577 00:20:55,430 --> 00:20:56,419 split them in groups? 578 00:20:56,420 --> 00:20:58,279 We have some groups that use some 579 00:20:58,280 --> 00:21:00,709 different I.T. security products. 580 00:21:00,710 --> 00:21:02,719 We could say one group uses some 581 00:21:02,720 --> 00:21:04,789 alternative treatment, which could be 582 00:21:04,790 --> 00:21:07,009 something like applying regular 583 00:21:07,010 --> 00:21:09,169 updates and doing application wide 584 00:21:09,170 --> 00:21:10,909 listing, which is generally considered 585 00:21:10,910 --> 00:21:13,279 the most viable alternative to antivirus 586 00:21:13,280 --> 00:21:14,569 software. 587 00:21:14,570 --> 00:21:16,699 Um, and we can't say one group 588 00:21:16,700 --> 00:21:18,889 gets a training where we say, OK, 589 00:21:18,890 --> 00:21:21,199 don't click on these email attachments. 590 00:21:21,200 --> 00:21:23,449 Um, I have to say here, I don't 591 00:21:23,450 --> 00:21:25,819 think training users is a very, 592 00:21:25,820 --> 00:21:27,979 very good strategy, but I think we should 593 00:21:27,980 --> 00:21:29,719 test that anyway. 594 00:21:29,720 --> 00:21:31,939 And then we could have a placebo group 595 00:21:31,940 --> 00:21:33,649 where we say just do the same thing you 596 00:21:33,650 --> 00:21:34,650 did before. 597 00:21:35,660 --> 00:21:36,660 Um, 598 00:21:38,090 --> 00:21:40,129 and then we try to measure security 599 00:21:40,130 --> 00:21:42,499 incidents, which maybe 600 00:21:42,500 --> 00:21:44,689 be. To even decide when a security 601 00:21:44,690 --> 00:21:45,690 incident happened, 602 00:21:47,270 --> 00:21:49,459 then we could also try to measure 603 00:21:49,460 --> 00:21:51,409 what side effects does this have to 604 00:21:51,410 --> 00:21:53,549 things crash? Do things get slower 605 00:21:53,550 --> 00:21:55,169 or what does it cost? 606 00:21:55,170 --> 00:21:57,919 We have some downturns and 607 00:21:57,920 --> 00:22:00,079 then after some time, we compare the 608 00:22:00,080 --> 00:22:01,080 result. 609 00:22:01,870 --> 00:22:03,949 Now, I have discussed 610 00:22:03,950 --> 00:22:05,629 this with a number of people before I did 611 00:22:05,630 --> 00:22:07,909 this talk. And the first reaction 612 00:22:07,910 --> 00:22:10,129 that usually comes as some form of this 613 00:22:10,130 --> 00:22:12,469 is really harsh and there's this problem 614 00:22:12,470 --> 00:22:14,779 and that problem and that problem. 615 00:22:14,780 --> 00:22:17,689 And I totally agree. 616 00:22:17,690 --> 00:22:19,489 This is really hard. 617 00:22:19,490 --> 00:22:20,509 Science is hard. 618 00:22:20,510 --> 00:22:21,979 That's just how it is. 619 00:22:21,980 --> 00:22:23,470 But it doesn't mean we shouldn't do it. 620 00:22:30,510 --> 00:22:33,029 So some problems that would show up 621 00:22:33,030 --> 00:22:35,249 for you could ask, what about the ethics 622 00:22:35,250 --> 00:22:36,929 of such a trial? Because you would say 623 00:22:36,930 --> 00:22:38,729 you give some security products to some 624 00:22:38,730 --> 00:22:40,259 people and not to others. 625 00:22:40,260 --> 00:22:41,700 So do you put them at risk? 626 00:22:42,900 --> 00:22:44,669 But if you think about it, that's a very 627 00:22:44,670 --> 00:22:46,919 comparable situation to medicine. 628 00:22:46,920 --> 00:22:49,299 If you test a medical track and 629 00:22:49,300 --> 00:22:51,209 then you give the drug to some people and 630 00:22:51,210 --> 00:22:52,949 you don't give it to other people. 631 00:22:52,950 --> 00:22:55,019 So it could be that this drug helps some 632 00:22:55,020 --> 00:22:56,549 people and doesn't help the others. 633 00:22:56,550 --> 00:22:58,499 But it could also be that this drug has a 634 00:22:58,500 --> 00:23:00,749 risk and that people 635 00:23:00,750 --> 00:23:02,939 suffer from taking that drug. 636 00:23:02,940 --> 00:23:04,829 But we generally have the idea in 637 00:23:04,830 --> 00:23:07,079 medicine that if we don't know whether 638 00:23:07,080 --> 00:23:09,149 drug helps, then testing it 639 00:23:09,150 --> 00:23:11,369 is the ethical thing because it will help 640 00:23:11,370 --> 00:23:13,769 many more people in the future. 641 00:23:13,770 --> 00:23:14,819 Um, yeah. 642 00:23:14,820 --> 00:23:16,919 Then we may wonder how do we 643 00:23:16,920 --> 00:23:19,499 reliably measure what's an incident? 644 00:23:19,500 --> 00:23:21,359 Because we may have situations where 645 00:23:22,680 --> 00:23:24,809 it's not even clear what was 646 00:23:24,810 --> 00:23:27,089 a heck or you have been hacked 647 00:23:27,090 --> 00:23:28,709 and you don't know about it. 648 00:23:28,710 --> 00:23:30,929 And you probably get 649 00:23:30,930 --> 00:23:33,029 very different results whether you 650 00:23:33,030 --> 00:23:35,129 have someone who is just affected 651 00:23:35,130 --> 00:23:37,319 by the normal everyday 652 00:23:37,320 --> 00:23:39,749 Internet of thing or someone 653 00:23:39,750 --> 00:23:42,209 who is targeted, someone 654 00:23:42,210 --> 00:23:44,189 who is targeted by a professional 655 00:23:44,190 --> 00:23:46,439 attacker and many 656 00:23:46,440 --> 00:23:49,289 more. I'm not saying it's easy, but 657 00:23:49,290 --> 00:23:51,110 there are many problems to be solved. 658 00:23:53,770 --> 00:23:56,169 Um, and, uh, 659 00:23:56,170 --> 00:23:58,839 I think the the security applications 660 00:23:58,840 --> 00:24:00,279 and antivirus software are just an 661 00:24:00,280 --> 00:24:01,779 example here, I think there are many 662 00:24:01,780 --> 00:24:03,819 things that we could test for such tests 663 00:24:03,820 --> 00:24:05,799 like there are debates about the safety 664 00:24:05,800 --> 00:24:07,909 of programing languages is 665 00:24:07,910 --> 00:24:09,879 rather better than C++. 666 00:24:09,880 --> 00:24:11,739 I think so. But I would like to see 667 00:24:11,740 --> 00:24:14,109 studies on it or application 668 00:24:14,110 --> 00:24:16,629 security like this browser, a 669 00:24:16,630 --> 00:24:18,879 more secure than browser B could 670 00:24:18,880 --> 00:24:19,929 be tested. 671 00:24:19,930 --> 00:24:20,930 Um. 672 00:24:24,210 --> 00:24:26,099 And finally, I want to bring up an 673 00:24:26,100 --> 00:24:28,349 example, which I think in some sense 674 00:24:28,350 --> 00:24:30,419 both a good and a bad example. 675 00:24:30,420 --> 00:24:32,639 So this was a tweet from the 676 00:24:32,640 --> 00:24:34,799 FTC, the Federal Trade 677 00:24:34,800 --> 00:24:35,940 Commission in the US, 678 00:24:37,320 --> 00:24:39,179 where they say, encourage your loved ones 679 00:24:39,180 --> 00:24:41,459 to change passwords often and some 680 00:24:41,460 --> 00:24:42,569 other things. 681 00:24:42,570 --> 00:24:43,979 So, um. 682 00:24:46,750 --> 00:24:48,939 Oh, um, 683 00:24:48,940 --> 00:24:51,399 yeah, so at some point they found out 684 00:24:51,400 --> 00:24:53,319 they actually had no scientific evidence 685 00:24:53,320 --> 00:24:54,819 for this recommendation to change 686 00:24:54,820 --> 00:24:56,919 passwords often, and they tried to find 687 00:24:56,920 --> 00:24:58,539 out why are we recommending this? 688 00:24:58,540 --> 00:24:59,919 And then they said, OK, we were 689 00:24:59,920 --> 00:25:01,479 recommending this because we are doing it 690 00:25:01,480 --> 00:25:04,019 ourselves. So it must be good. 691 00:25:04,020 --> 00:25:06,730 Um, um. 692 00:25:08,680 --> 00:25:10,779 And then they basically recommended 693 00:25:10,780 --> 00:25:12,250 the opposite. They said, OK, 694 00:25:13,300 --> 00:25:15,609 we we have reconsidered this. 695 00:25:15,610 --> 00:25:16,929 We looked at the evidence. 696 00:25:16,930 --> 00:25:19,119 We have some studies that say that 697 00:25:19,120 --> 00:25:21,039 mandatory password changes are not a 698 00:25:21,040 --> 00:25:23,379 probably not a good idea and maybe 699 00:25:23,380 --> 00:25:25,059 you should not change your passwords on a 700 00:25:25,060 --> 00:25:26,859 regular basis. 701 00:25:26,860 --> 00:25:29,259 So, um, 702 00:25:29,260 --> 00:25:30,939 I looked at the studies that they were 703 00:25:30,940 --> 00:25:33,189 citing for that, and I was not 704 00:25:33,190 --> 00:25:34,179 completely convinced. 705 00:25:34,180 --> 00:25:36,279 So I felt the quality 706 00:25:36,280 --> 00:25:38,199 of these studies was not very high. 707 00:25:38,200 --> 00:25:40,959 So all of them were based on 708 00:25:40,960 --> 00:25:43,089 on observational data that they 709 00:25:43,090 --> 00:25:45,039 didn't make any intervention where they 710 00:25:45,040 --> 00:25:46,059 put people in groups. 711 00:25:46,060 --> 00:25:48,729 But they they did things like they had 712 00:25:48,730 --> 00:25:51,009 password data from a company 713 00:25:51,010 --> 00:25:53,049 where at some point they had a password 714 00:25:53,050 --> 00:25:54,999 changing policy and at other points not. 715 00:25:55,000 --> 00:25:57,489 And, um, then 716 00:25:57,490 --> 00:25:59,529 there was one which was trying to make a 717 00:25:59,530 --> 00:26:01,659 theoretical model of password breaks 718 00:26:01,660 --> 00:26:03,219 and password changes and how much that 719 00:26:03,220 --> 00:26:04,539 matters. 720 00:26:04,540 --> 00:26:06,729 But the basis of all of these studies was 721 00:26:06,730 --> 00:26:08,439 observational data. 722 00:26:08,440 --> 00:26:10,689 And then also some of these studies tried 723 00:26:10,690 --> 00:26:12,669 to measure things like password quality 724 00:26:12,670 --> 00:26:14,169 by the entropy. 725 00:26:14,170 --> 00:26:16,149 And if you think about that, that's not 726 00:26:16,150 --> 00:26:17,649 really what we care about. 727 00:26:17,650 --> 00:26:19,779 What we care about is whether our data 728 00:26:19,780 --> 00:26:21,519 gets hacked. 729 00:26:21,520 --> 00:26:23,169 We don't care about the entropy of our 730 00:26:23,170 --> 00:26:24,729 passwords. Maybe the entropy of our 731 00:26:24,730 --> 00:26:26,769 passwords is an indicator that we have a 732 00:26:26,770 --> 00:26:28,119 good password. 733 00:26:28,120 --> 00:26:30,009 But it's only one factor. 734 00:26:30,010 --> 00:26:32,109 And there are other things like if we 735 00:26:32,110 --> 00:26:33,709 use a password, that's also bad. 736 00:26:33,710 --> 00:26:35,889 So maybe people use a strong password, 737 00:26:35,890 --> 00:26:37,449 but use it for many different services 738 00:26:37,450 --> 00:26:38,649 and it's also better. 739 00:26:38,650 --> 00:26:40,869 And in medicine, there's 740 00:26:40,870 --> 00:26:42,879 a term for that. And that's a surrogate 741 00:26:42,880 --> 00:26:44,769 endpoint, which is when you're measuring 742 00:26:44,770 --> 00:26:47,019 something that's not the thing you really 743 00:26:47,020 --> 00:26:49,059 care about, but maybe an indicator of 744 00:26:49,060 --> 00:26:50,709 what you care about. 745 00:26:50,710 --> 00:26:52,359 And that's generally considered a lower 746 00:26:52,360 --> 00:26:53,710 quality of evidence. 747 00:26:55,800 --> 00:26:57,989 Um, so the good thing 748 00:26:57,990 --> 00:26:59,879 here is the FTC found out that they 749 00:26:59,880 --> 00:27:01,709 didn't have scientific evidence for their 750 00:27:01,710 --> 00:27:03,359 recommendations and they said, OK, we 751 00:27:03,360 --> 00:27:05,639 have to look at the scientific evidence, 752 00:27:05,640 --> 00:27:06,989 but they're not. So good thing is, I 753 00:27:06,990 --> 00:27:09,299 think the quality of the evidence was not 754 00:27:09,300 --> 00:27:10,300 so good. 755 00:27:11,100 --> 00:27:13,289 So the real conclusion would be maybe 756 00:27:13,290 --> 00:27:14,699 we just don't know what we should do. 757 00:27:14,700 --> 00:27:16,349 Some proper studies on that. 758 00:27:18,390 --> 00:27:20,669 Um, so finally, 759 00:27:20,670 --> 00:27:23,099 I have some, uh, things 760 00:27:23,100 --> 00:27:25,259 for I think like I think 761 00:27:25,260 --> 00:27:26,819 this is the right approach, but I think 762 00:27:26,820 --> 00:27:28,889 it has some limits that should be 763 00:27:28,890 --> 00:27:30,509 considered. 764 00:27:30,510 --> 00:27:32,699 Um, there are 765 00:27:32,700 --> 00:27:34,889 things that we want to protect ourselves 766 00:27:34,890 --> 00:27:37,109 against a threat that we cannot 767 00:27:37,110 --> 00:27:38,909 really measure because they may be just 768 00:27:38,910 --> 00:27:40,259 future threats. 769 00:27:40,260 --> 00:27:42,299 For example, currently we have a debate 770 00:27:42,300 --> 00:27:44,429 about post quantum cryptography. 771 00:27:44,430 --> 00:27:46,649 Do we need to protect ourselves 772 00:27:46,650 --> 00:27:48,329 against quantum computers? 773 00:27:48,330 --> 00:27:50,579 There are no quantum computers today, 774 00:27:50,580 --> 00:27:52,649 so we cannot measure any attacks with 775 00:27:52,650 --> 00:27:53,849 quantum computers. 776 00:27:53,850 --> 00:27:55,679 But we still may want to prepare for 777 00:27:55,680 --> 00:27:56,639 that. 778 00:27:56,640 --> 00:27:58,829 And there are things where we have attack 779 00:27:58,830 --> 00:28:00,719 scenarios that are very obscure where we 780 00:28:00,720 --> 00:28:02,849 say, OK, what if a nation state 781 00:28:02,850 --> 00:28:05,279 is compromising a Debian developer 782 00:28:05,280 --> 00:28:07,049 and he gives me a different package than 783 00:28:07,050 --> 00:28:09,299 the one he is telling me, 784 00:28:09,300 --> 00:28:11,099 which is something that the reproducible, 785 00:28:11,100 --> 00:28:12,869 built community is trying to tackle, 786 00:28:14,790 --> 00:28:17,009 which I'm 787 00:28:17,010 --> 00:28:18,719 not sure if such an attack ever happened, 788 00:28:18,720 --> 00:28:20,279 but it may still be something you want to 789 00:28:20,280 --> 00:28:22,019 think about against protecting so that 790 00:28:22,020 --> 00:28:23,939 there are definitely situations where you 791 00:28:23,940 --> 00:28:25,619 cannot do this approach with a controlled 792 00:28:25,620 --> 00:28:27,009 study. 793 00:28:27,010 --> 00:28:29,009 Um, and one more thing. 794 00:28:29,010 --> 00:28:30,329 There are sometimes claims that are 795 00:28:30,330 --> 00:28:32,489 simply against that violate basic 796 00:28:32,490 --> 00:28:33,809 scientific principles. 797 00:28:33,810 --> 00:28:35,549 For example, if a vendor promises for 798 00:28:35,550 --> 00:28:37,739 protection from malware, that's just 799 00:28:37,740 --> 00:28:39,999 a lie. It's simply a lie because 800 00:28:40,000 --> 00:28:42,089 that's impossible because of 801 00:28:42,090 --> 00:28:43,889 the so-called halting problem, which is a 802 00:28:43,890 --> 00:28:46,859 very basic theorem of computer science. 803 00:28:46,860 --> 00:28:48,809 And there's a related debate in medicine 804 00:28:48,810 --> 00:28:50,609 where some people argue we shouldn't even 805 00:28:50,610 --> 00:28:52,949 study something like homeopathy because 806 00:28:52,950 --> 00:28:54,629 it simply cannot be true based on the 807 00:28:54,630 --> 00:28:56,429 laws of physics. 808 00:28:56,430 --> 00:28:58,499 So, yeah, that was my last 809 00:28:58,500 --> 00:28:59,489 slide. 810 00:28:59,490 --> 00:29:02,009 Um, so I think today 811 00:29:02,010 --> 00:29:04,079 think security is often very often 812 00:29:04,080 --> 00:29:06,659 not based on scientific evidence. 813 00:29:06,660 --> 00:29:08,729 We rely on experience, we rely on 814 00:29:08,730 --> 00:29:10,319 experts are even worse. 815 00:29:10,320 --> 00:29:12,329 We may rely on marketing. 816 00:29:12,330 --> 00:29:14,039 We should have evidence based IP 817 00:29:14,040 --> 00:29:16,049 security. But right now we don't have the 818 00:29:16,050 --> 00:29:17,909 science to do that. 819 00:29:17,910 --> 00:29:18,910 Yeah, I think. 820 00:29:28,150 --> 00:29:30,279 I said we likely don't have time 821 00:29:30,280 --> 00:29:31,809 for question, but if someone is very 822 00:29:31,810 --> 00:29:33,819 quick and runs up to the microphone, then 823 00:29:33,820 --> 00:29:34,839 we can take one. 824 00:29:34,840 --> 00:29:36,369 Yes, microphone three. 825 00:29:36,370 --> 00:29:38,469 And if you do, you hear 826 00:29:38,470 --> 00:29:39,549 me? Yeah. 827 00:29:39,550 --> 00:29:41,379 If you go to medical studies, it's 828 00:29:41,380 --> 00:29:42,709 actually the highest quality that you do 829 00:29:42,710 --> 00:29:44,739 double randomized controlled studies, 830 00:29:44,740 --> 00:29:47,139 which means neither the experimental 831 00:29:47,140 --> 00:29:48,939 nor the participant knows whether it 832 00:29:48,940 --> 00:29:52,179 takes placebo or the actual medication. 833 00:29:52,180 --> 00:29:53,799 You think that is something that you can 834 00:29:53,800 --> 00:29:55,299 actually implement because people act 835 00:29:55,300 --> 00:29:56,739 differently if they know they have the 836 00:29:56,740 --> 00:29:59,529 placebo or if they have the drug 837 00:29:59,530 --> 00:30:00,529 greatly interferes. 838 00:30:00,530 --> 00:30:02,639 The randomized control, that's 839 00:30:02,640 --> 00:30:04,749 fine. So so BLANNING studies, 840 00:30:04,750 --> 00:30:06,249 if you can do that, depends on your 841 00:30:06,250 --> 00:30:08,259 situation because there are situations 842 00:30:08,260 --> 00:30:10,539 where you cannot blind if it's something 843 00:30:10,540 --> 00:30:12,609 that the user has to actively do, but if 844 00:30:12,610 --> 00:30:14,289 possible, you are blinding us better. 845 00:30:15,850 --> 00:30:17,979 OK, sorry. Unfortunately we don't have 846 00:30:17,980 --> 00:30:20,259 any more time for questions, but 847 00:30:20,260 --> 00:30:22,449 as I said, Hannah will be prepared 848 00:30:22,450 --> 00:30:24,639 and ready to answer some questions right 849 00:30:24,640 --> 00:30:25,640 here. Thanks.