Predicting Out Of Memory Kill events with Machine Learning (Ep. 203)

Sometimes applications crash. Some other times applications crash because memory is exhausted. Such issues exist because of bugs in the code, or heavy memory usage for reasons that were not expected during design and implementation. Can we use machine learning to predict and eventually detect out of memory kills from the operating system?

Apparently, the Netflix app many of us use on a daily basis leverage ML and time series analysis to prevent OOM-kills.

Enjoy the show!

Our Sponsors

Explore the Complex World of Regulations. Compliance can be overwhelming. Multiple frameworks. Overlapping requirements. Let Arctic Wolf be your guide. Check it out at https://arcticwolf.com/datascience

 

Amethix works to create and maximize the impact of the world’s leading corporations and startups, so they can create a better future for everyone they serve. We provide solutions in AI/ML, Fintech, Healthcare/RWE, and Predictive maintenance.

 

Transcript

1 00:00:04,150 --> 00:00:09,034 And here we are again with the season four of the Data Science at Home podcast.

2 00:00:09,142 --> 00:00:19,170 This time we have something for you if you want to help us shape the data science leaders of the future, we have created the the Data Science at Home's Ambassador program.

3 00:00:19,340 --> 00:00:28,378 Ambassadors are volunteers who are passionate about data science and want to give back to our growing community of data science professionals and enthusiasts.

4 00:00:28,534 --> 00:00:37,558 You will be instrumental in helping us achieve our goal of raising awareness about the critical role of data science in cutting edge technologies.

5 00:00:37,714 --> 00:00:45,740 If you want to learn more about this program, visit the Ambassadors page on our website@datascienceathome.com.

6 00:00:46,430 --> 00:00:49,234 Welcome back to another episode of Data Science at Home podcast.

7 00:00:49,282 --> 00:00:55,426 I'm Francesco Podcasting from the Regular Office of Amethyx Technologies, based in Belgium.

8 00:00:55,618 --> 00:01:02,914 In this episode, I want to speak about a machine learning problem that has been formulated at Netflix.

9 00:01:03,022 --> 00:01:22,038 And for the record, Netflix is not sponsoring this episode, though I still believe that this problem is a very well known problem, a very common one across factors, which is how to predict out of memory kill in an application and formulate this problem as a machine learning problem.

10 00:01:22,184 --> 00:01:39,142 So this is something that, as I said, is very interesting, not just because of Netflix, but because it allows me to explain a few points that, as I said, are kind of invariance across sectors.

11 00:01:39,226 --> 00:01:56,218 Regardless of your application, is a video streaming application or any other communication type of application, or a fintech application, or energy, or whatever, this memory kill, out of memory kill still occurs.

12 00:01:56,314 --> 00:02:05,622 And what is an out of memory kill? Well, it's essentially the extreme event in which the machine doesn't have any more memory left.

13 00:02:05,756 --> 00:02:16,678 And so usually the operating system can start eventually swapping, which means using the SSD or the hard drive as a source of memory.

14 00:02:16,834 --> 00:02:19,100 But that, of course, will slow down a lot.

15 00:02:19,430 --> 00:02:45,210 And eventually when there is a bug or a memory leak, or if there are other applications running on the same machine, of course there is some kind of limiting factor that essentially kills the application, something that occurs from the operating system most of the time that kills the application in order to prevent the application from monopolizing the entire machine, the hardware of the machine.

16 00:02:45,710 --> 00:02:48,500 And so this is a very important problem.

17 00:02:49,070 --> 00:03:03,306 Also, it is important to have an episode about this because there are some strategies that I've used at Netflix that are pretty much in line with what I believe machine learning should be about.

18 00:03:03,368 --> 00:03:25,062 And usually people would go for the fancy solution there like this extremely accurate predictors or machine learning models, but you should have a massive number of parameters and that try to figure out whatever is happening on that machine that is running that application.

19 00:03:25,256 --> 00:03:29,466 While the solution at Netflix is pretty straightforward, it's pretty simple.

20 00:03:29,588 --> 00:03:33,654 And so one would say then why making an episode after this? Well.

21 00:03:33,692 --> 00:03:45,730 Because I think that we need more sobriety when it comes to machine learning and I believe we still need to spend a lot of time thinking about what data to collect.

22 00:03:45,910 --> 00:03:59,730 Reasoning about what is the problem at hand and what is the data that can actually tickle the particular machine learning model and then of course move to the actual prediction that is the actual model.

23 00:03:59,900 --> 00:04:15,910 That most of the time it doesn't need to be one of these super fancy things that you see on the news around chatbots or autonomous gaming agent or drivers and so on and so forth.

24 00:04:16,030 --> 00:04:28,518 So there are essentially two data sets that the people at Netflix focus on which are consistently different, dramatically different in fact.

25 00:04:28,604 --> 00:04:45,570 These are data about device characteristics and capabilities and of course data that are collected at Runtime and that give you a picture of what's going on in the memory of the device, right? So that's the so called runtime memory data and out of memory kills.

26 00:04:45,950 --> 00:05:03,562 So the first type of data is I would consider it very static because it considers for example, the device type ID, the version of the software development kit that application is running, cache capacities, buffer capacities and so on and so forth.

27 00:05:03,646 --> 00:05:11,190 So it's something that most of the time doesn't change across sessions and so that's why it's considered static.

28 00:05:12,050 --> 00:05:18,430 In contrast, the other type of data, the Runtime memory data, as the name says it's runtime.

29 00:05:18,490 --> 00:05:24,190 So it varies across the life of the session it's collected at Runtime.

30 00:05:24,250 --> 00:05:25,938 So it's very dynamic data.

31 00:05:26,084 --> 00:05:36,298 And example of these records are for example, profile, movie details, playback information, current memory usage, et cetera, et cetera.

32 00:05:36,334 --> 00:05:56,086 So this is the data that actually moves and moves in the sense that it changes depending on how the user is actually using the Netflix application, what movie or what profile description, what movie detail has been loaded for that particular movie and so on and so forth.

33 00:05:56,218 --> 00:06:15,094 So one thing that of course the first difficulty of the first challenge that the people at Netflix had to deal with was how would you combine these two things, very static and usually small tables versus very dynamic and usually large tables or views.

34 00:06:15,142 --> 00:06:36,702 Well, there is some sort of join on key that is performed by the people at Netflix in order to put together these different data resolutions, right, which is data of the same phenomenon but from different sources and with different carrying very different signals in there.

35 00:06:36,896 --> 00:06:48,620 So the device capabilities is captured usually by the static data and of course the other data, the Runtime memory and out of memory kill data.

36 00:06:48,950 --> 00:07:04,162 These are also, as I said, the data that will describe pretty accurately how is the user using that particular application on that particular hardware.

37 00:07:04,306 --> 00:07:17,566 Now of course, when it comes to data and deer, there is nothing new that people at Netflix have introduced dealing with missing data for example, or incorporating knowledge of devices.

38 00:07:17,698 --> 00:07:26,062 It's all stuff that it's part of the so called data cleaning and data collection strategy, right? Or data preparation.

39 00:07:26,146 --> 00:07:40,782 That is, whatever you're going to do in order to make that data or a combination of these data sources, let's say, compatible with the way your machine learning model will understand or will read that data.

40 00:07:40,916 --> 00:07:58,638 So if you think of a big data platform, the first step, the first challenge you have to deal, you have to deal with is how can I, first of all, collect the right amount of information, the right data, but also how to transform this data for my particular big data platform.

41 00:07:58,784 --> 00:08:12,798 And that's something that, again, nothing new, nothing fancy, just basics, what we have been used to, what we are used to seeing now for the last decade or more, that's exactly what they do.

42 00:08:12,944 --> 00:08:15,222 And now let me tell you something important.

43 00:08:15,416 --> 00:08:17,278 Cybercriminals are evolving.

44 00:08:17,374 --> 00:08:22,446 Their techniques and tactics are more advanced, intricate and dangerous than ever before.

45 00:08:22,628 --> 00:08:30,630 Industries and governments around the world are fighting back on dealing new regulations meant to better protect data against this rising threat.

46 00:08:30,950 --> 00:08:39,262 Today, the world of cybersecurity compliance is a complex one, and understanding the requirements your organization must adhere to can be a daunting task.

47 00:08:39,406 --> 00:08:42,178 But not when the pack has your best architect.

48 00:08:42,214 --> 00:08:53,840 Wolf, the leader in security operations, is on a mission to end cyber risk by giving organizations the protection, information and confidence they need to protect their people, technology and data.

49 00:08:54,170 --> 00:09:02,734 The new interactive compliance portal helps you discover the regulations in your region and industry and start the journey towards achieving and maintaining compliance.

50 00:09:02,902 --> 00:09:07,542 Visit Arcticwolves.com DataScience to take your first step.

51 00:09:07,676 --> 00:09:11,490 That's arcticwolf.com DataScience.

52 00:09:12,050 --> 00:09:18,378 I think that the most important part, though, I think are actually equally important.

53 00:09:18,464 --> 00:09:26,854 But the way they treat runtime memory data and out of memory kill data is by using sliding windows.

54 00:09:26,962 --> 00:09:38,718 So that's something that is really worth mentioning, because the way you would frame this problem is something is happening at some point in time and I have to kind of predict that event.

55 00:09:38,864 --> 00:09:49,326 That is usually an outlier in the sense that these events are quite rare, fortunately, because Netflix would not be as usable as we believe it is.

56 00:09:49,448 --> 00:10:04,110 So you would like to predict these weird events by looking at a historical view or an historical amount of records that you have before this particular event, which is the kill of the application.

57 00:10:04,220 --> 00:10:12,870 So the concept of the sliding window, the sliding window approach is something that comes as the most natural thing anyone would do.

58 00:10:13,040 --> 00:10:18,366 And that's exactly what the researchers and Netflix have done.

59 00:10:18,488 --> 00:10:25,494 So unexpectedly, in my opinion, they treated this problem as a time series, which is exactly what it is.

60 00:10:25,652 --> 00:10:26,190 Now.

61 00:10:26,300 --> 00:10:26,754 They.

62 00:10:26,852 --> 00:10:27,330 Of course.

63 00:10:27,380 --> 00:10:31,426 Use this sliding window with a different horizon.

64 00:10:31,558 --> 00:10:32,190 Five minutes.

65 00:10:32,240 --> 00:10:32,838 Four minutes.

66 00:10:32,924 --> 00:10:33,702 Two minutes.

67 00:10:33,836 --> 00:10:36,366 As close as possible to the event.

68 00:10:36,548 --> 00:10:38,886 Because maybe there are some.

69 00:10:39,008 --> 00:10:39,762 Let's say.

70 00:10:39,896 --> 00:10:45,678 Other dynamics that can raise when you are very close to the event or when you are very far from it.

71 00:10:45,704 --> 00:10:50,166 Like five minutes far from the out of memory kill.

72 00:10:50,348 --> 00:10:51,858 Might have some other.

73 00:10:51,944 --> 00:10:52,410 Let's say.

74 00:10:52,460 --> 00:10:55,986 Diagrams or shapes in the data.

75 00:10:56,168 --> 00:11:11,310 So for example, you might have a certain number of allocations that keep growing and growing, but eventually they grow with a certain curve or a certain rate that you can measure when you are five to ten minutes far from the out of memory kill.

76 00:11:11,420 --> 00:11:16,566 When you are two minutes far from the out of memory kill, probably this trend will change.

77 00:11:16,688 --> 00:11:30,800 And so probably what you would expect is that the memory is already half or more saturated and therefore, for example, the operating system starts swapping or other things are happening that you are going to measure in this.

78 00:11:31,550 --> 00:11:39,730 And that would give you a much better picture of what's going on in the, let's say, closest neighborhood of that event, the time window.

79 00:11:39,790 --> 00:11:51,042 The sliding window and time window approach is definitely worth mentioning because this is something that you can apply if you think pretty much anywhere right now.

80 00:11:51,116 --> 00:11:52,050 What they did.

81 00:11:52,160 --> 00:12:04,146 In addition to having a time window, a sliding window, they also assign different levels to memory readings that are closer to the out of memory kill.

82 00:12:04,208 --> 00:12:10,062 And usually these levels are higher and higher as we get closer and closer to the out of memory kill.

83 00:12:10,136 --> 00:12:15,402 So this means that, for example, we would have, for a five minute window, we would have a level one.

84 00:12:15,596 --> 00:12:22,230 Five minute means five minutes far from the out of memory kill, four minutes would be a level two.

85 00:12:22,280 --> 00:12:37,234 Three minutes it's much closer would be a level three, two minutes would be a level four, which means like kind of the severity of the event as we get closer and closer to the actual event when the application is actually killed.

86 00:12:37,342 --> 00:12:51,474 So by looking at this approach, nothing new there, even, I would say not even a seasoned data scientist would have understood that using a sliding window is the way to go.

87 00:12:51,632 --> 00:12:55,482 I'm not saying that Netflix engineers are not seasoned enough.

88 00:12:55,556 --> 00:13:04,350 Actually they do a great job every day to keep giving us video streaming platforms that actually never fail or almost never fail.

89 00:13:04,910 --> 00:13:07,460 So spot on there, guys, good job.

90 00:13:07,850 --> 00:13:27,738 But looking at this sliding window approach, the direct consequence of this is that they can plot, they can do some sort of graphical analysis of the out of memory kills versus the memory usage that can give the reader or the data scientist a very nice picture of what's going on there.

91 00:13:27,824 --> 00:13:39,330 And so you would have, for example, and I would definitely report some of the pictures, some of the diagrams and graphs in the show notes of this episode on the official website datascienceaton.com.

92 00:13:39,500 --> 00:13:48,238 But essentially what you can see there is that there might be premature peaks at, let's say, a lower memory reading.

93 00:13:48,334 --> 00:14:08,958 And usually these are some kind of false positives or anomalies that should not be there, then it's possible to set a threshold where the threshold to start lowering the memory usage because after that threshold something nasty can happen and usually happens according to your data.

94 00:14:09,104 --> 00:14:18,740 And then of course there is another graph about the Gaussian distribution or in fact no sharp peak at all.

95 00:14:19,250 --> 00:14:21,898 That is like kills or out of memory.

96 00:14:21,934 --> 00:14:33,754 Kills are more or less distributed in a normalized fashion and then of course there are the genuine peaks that indicate that kills near, let's say, the threshold.

97 00:14:33,802 --> 00:14:38,758 And so usually you would see that after that particular threshold of memory usage.

98 00:14:38,914 --> 00:14:42,142 You see most of the out of memory kills.

99 00:14:42,226 --> 00:14:45,570 Which makes sense because given a particular device.

100 00:14:45,890 --> 00:14:48,298 Which means certain amount of memories.

101 00:14:48,394 --> 00:14:50,338 Certain memory characteristics.

102 00:14:50,494 --> 00:14:53,074 Certain version of the SDK and so on and so forth.

103 00:14:53,182 --> 00:14:53,814 You can say.

104 00:14:53,852 --> 00:14:54,090 Okay.

105 00:14:54,140 --> 00:15:10,510 Well for this device type I have this memory memory usage threshold and after this I see that I have a relatively high number of out of memory kills immediately after this threshold.

106 00:15:10,570 --> 00:15:18,150 And this means that probably that is the threshold you would like to consider as the critical threshold you should never or almost never cross.

107 00:15:18,710 --> 00:15:38,758 So once you have this picture in front of you, you can start thinking of implementing some mechanisms that can monitor the memory usage and of course kind of preemptively dialocate things or keep that memory threshold as low as possible with respect to the critical threshold.

108 00:15:38,794 --> 00:15:53,446 So you can start implementing some logic that prevents the application from being killed by the operating system so that you would in fact reduce the rate of out of memory kills overall.

109 00:15:53,578 --> 00:16:11,410 Now, as always and as also the engineers state in their blog post, in the technical post, they say well, it's much more important for us to predict with a certain amount of false positive rather than false negatives.

110 00:16:11,590 --> 00:16:18,718 False negatives means missing an out of memory kill that actually occurred but got not predicted.

111 00:16:18,874 --> 00:16:40,462 If you are a regular listener of this podcast, that statement should resonate with you because this is exactly what happens, for example in healthcare applications, which means that doctors or algorithms that operate in healthcare would definitely prefer to have a bit more false positives rather than more false negatives.

112 00:16:40,486 --> 00:16:54,800 Because missing that someone is sick means that you are not providing a cure and you're just sending the patient home when he or she is sick, right? That's the false positive, it's the mess.

113 00:16:55,130 --> 00:16:57,618 So that's a false negative, it's the mess.

114 00:16:57,764 --> 00:17:09,486 But having a false positive, what can go wrong with having a false positive? Well, probably you will undergo another test to make sure that the first test is confirmed or not.

115 00:17:09,608 --> 00:17:16,018 So adding a false positive in this case is relatively okay with respect to having a false negative.

116 00:17:16,054 --> 00:17:19,398 And that's exactly what happens to the Netflix application.

117 00:17:19,484 --> 00:17:32,094 Now, I don't want to say that of course Netflix application is as critical as, for example, the application that predicts a cancer or an xray or something on an xray or disorder or disease of some sort.

118 00:17:32,252 --> 00:17:48,090 But what I'm saying is that there are some analogies when it comes to machine learning and artificial intelligence and especially data science, the old school data science, there are several things that kind of are, let's say, invariant across sectors.

119 00:17:48,410 --> 00:17:56,826 And so, you know, two worlds like the media streaming or video streaming and healthcare are of course very different from each other.

120 00:17:56,888 --> 00:18:05,274 But when it comes to machine learning and data science applications, well, there are a lot of analogies there.

121 00:18:05,372 --> 00:18:06,202 And indeed.

122 00:18:06,286 --> 00:18:10,234 In terms of the models that they use at Netflix to predict.

123 00:18:10,342 --> 00:18:24,322 Once they have the sliding window data and essentially they have the ground truth of where this out of memory kill happened and what happened before to the memory of the application or the machine.

124 00:18:24,466 --> 00:18:24,774 Well.

125 00:18:24,812 --> 00:18:30,514 Then the models they use to predict these things is these events is Artificial Neural Networks.

126 00:18:30,622 --> 00:18:31,714 Xg Boost.

127 00:18:31,822 --> 00:18:36,742 Ada Boost or Adaptive Boosting Elastic Net with Softmax and so on and so forth.

128 00:18:36,766 --> 00:18:39,226 So nothing fancy.

129 00:18:39,418 --> 00:18:45,046 As you can see, Xg Boost is probably one of the most used I would have expected even random forest.

130 00:18:45,178 --> 00:18:47,120 Probably they do, they've tried that.

131 00:18:47,810 --> 00:18:58,842 But XGBoost is probably one of the most used models on kaggle competitions for a reason, because it works and it leverages a lot.

132 00:18:58,916 --> 00:19:04,880 The data preparation step, that solves already more than half of the problem.

133 00:19:05,810 --> 00:19:07,270 Thank you so much for listening.

134 00:19:07,330 --> 00:19:11,910 I also invite you, as always, to join the Discord Channel.

135 00:19:12,020 --> 00:19:15,966 You will find a link on the official website datascience@home.com.

136 00:19:16,148 --> 00:19:17,600 Speak with you next time.

137 00:19:18,350 --> 00:19:21,382 You've been listening to Data Science at home podcast.

138 00:19:21,466 --> 00:19:26,050 Be sure to subscribe on itunes, Stitcher, or Pot Bean to get new, fresh episodes.

139 00:19:26,110 --> 00:19:31,066 For more, please follow us on Instagram, Twitter and Facebook or visit our website at datascienceathome.com

 

References

https://netflixtechblog.com/formulating-out-of-memory-kill-prediction-on-the-netflix-app-as-a-machine-learning-problem-989599029109

Episode source