FounderQuest
FounderQuest
Are 3rd-party scripts out to get you? CSP to the rescue!
Full transcript:
Josh: 00:00 A middle of the night disruption for Ben, what is that? Like 8:30?
Starr: 00:05 Oh, somebody called the burn unit! Oh!
Announcer: 00:08 You are in a maze of twisty little passages, all alike. Time to start a fire! Crack open a can of Tab, and settle in for Founder Quest.
Starr: 00:21 It is telling that the only way we can have sick burns on Ben is to accuse him of being too productive.
Josh: 00:27 Yeah.
Ben: 00:28 It's all good.
Starr: 00:29 So you know how turkey makes you sleepy?
Ben: 00:31 Tryptophan.
Starr: 00:32 You can buy pills full of that stuff.
Josh: 00:35 I forgot about Tryptophan supplements.
Ben: 00:38 Wait, do they have pills that have both Tryptophan and Melatonin?
Starr: 00:41 No, but...
Josh: 00:43 Oh that's, that's like a cocktail.
Starr: 00:44 I don't know if I want to do that. I want to wake up in the morning eventually.
Ben: 00:49 You got to have the Tryptophan, plus the Melatonin, plus the NyQuil chaser.
Josh: 00:54 Yeah.
Starr: 00:54 And you just empty all those pill... empty all those capsules into a shot of whiskey. And then you just pound that bad boy.
Josh: 01:01 Yeah!
Ben: 01:03 Throw some Benadryl in there if you haven't... If you're still awake.
Josh: 01:06 So today, I thought it would be fun to talk about like an actual sort of feature, like I don't think we've actually shipped this feature yet. Have we?
Ben: 01:15 We kind of shipped the feature.
Josh: 01:17 We've shipped it a few times in various forms, not-
Ben: 01:22 It is not GA yet, as the big boys say.
Josh: 01:25 Oh, so if you're a VIP, you get the feature flag. Like the "you can totally use this feature".
Ben: 01:30 Exactly. If you're on the list then you get to use this feature.
Starr: 01:32 Oh, yeah.
Josh: 01:32 You did deploy... yeah, you deployed some of it, right? Or is it all of it?
Ben: 01:35 Yes, it's deployed. It's out there.
Josh: 01:35 Oh, okay.
Ben: 01:36 We could launch it today if we really felt like it. But...
Starr: 01:40 Anyway, I thought it would be fun to talk about this feature because it's something we've been discussing for literally years.
Ben: 01:47 Literally.
Starr: 01:48 I think...
Josh: 01:49 Yeah, I think about-
Starr: 01:50 I think Ben has taken a couple shots at it.
Josh: 01:52 I've taken a couple of shots at it, too.
Starr: 01:54 Damn, I feel like I should've taken some shots at it. And this feature is CSP reporting. So could somebody please tell me what the heck that is? Like what, what is CSP reporting?
Josh: 02:06 Well, CSP is content security policy reporting. Content security policy is a feature of modern browsers that allows you to alert, basically send alerts to a URL of your choice when content that you don't, that you didn't authorize is loaded on the page.
Ben: 02:30 So yeah, in addition to reporting, it also blocks that content, right? I mean, that's the primary use case, is to prevent your side from serving something that you didn't intend it to serve. Right? And so the reporting is kind of an extra benefit that you can, you can track-
Josh: 02:44 Yeah.
Ben: 02:44 ... but, you can see this in your browser. You know when you open up the console and those console errors, right? You can see if anything violates a content security policy. You can see that the browser is like, "Nope, didn't load that."
Starr: 02:56 So what, what might be... What is an example of some bad behavior that this is trying to prevent?
Ben: 03:02 So cross-site scripting is huge. So you know, you can inject some JavaScript into a vulnerable page. Let's say you have a content management system that allows you to put some user input in and then it displays it to some other user. And if it-
Starr: 03:16 Or you have user comments or something.
Ben: 03:18 ... yeah, yeah. And so, if you don't escape that stuff properly, you can have... you can inject some scripts into the page that some other user sees, right? And so you could steal their cookies or do all kinds of things, right? And so-
Josh: 03:31 Yeah.
Ben: 03:31 ... content security policy says, "well, I don't actually want to allow in-line scripts to be executed on my pages because I don't do that sort of thing." Right? And so if there is some sort of weakness in your system that allows in-line scripting to be put in there that you didn't put in there, then that policy would say, "Nope, not going to run it."
Starr: 03:49 Oh, wow. That's... I had no idea that it could do things like block in-line scripting. I assumed it was just solely a sort of URL based thing, where it's like any JavaScript that doesn't come from these specific domains, block that and report it but you can do in-line stuff too.
Josh: 04:07 Yeah. So, it can do both on the page and remote, and yeah, I guess like if... an example of something it would block is if a remote script like Google Analytics got hijacked or something, and someone tried to load another remote script, you know? By dropping it into the DOM or something that loaded a JavaScript file from a bad domain, then it would not have that domain white-listed and it would block it.
Ben: 04:39 Yeah. It can also block like you were thinking, Starr, it can block connections to remote JavaScript and remote style sheets and remote forms. So yeah, there are a lot of things that it can do.
Josh: 04:51 Yeah.
Ben: 04:52 And one... another-
Josh: 04:54 It's like any XH... like any cross domain request, right? It'll block anything, I think, or...
Ben: 05:01 Yeah, I don't know. I mean, I know it has script source-
Josh: 05:03 Yeah. Like XHR.
Ben: 05:03 ... and style source directives. So, I don't know if it does anything, but...
Starr: 05:07 So how do you actually add... so this is... so the CSP policy tells the browser to block certain things and possibly report them. So how do you... what actually does that look like? How do you add a CSP to your page? Is it like a meta tag in the document head or what?
Ben: 05:27 Typically, it's going to be a header that your web server would return, right? So Apache or NGINX would be configured to return this header.
Josh: 05:36 You send it with the request. Actually, I believe in the latest version of Rails, it actually has some pre-configured CSP headers that it sends. And I think it probably does that, like I haven't checked, but like in a rack middleware or something, I'm guessing. Where it would just add the headers to the request as it's making it. And I think that you can like configure what you're going to allow.
Ben: 06:00 Yeah. So the... in the new versions of Rails, I think it started in 5.2 actually.
Josh: 06:04 Was it 5.2?
Ben: 06:04 Yeah.
Josh: 06:04 Okay.
Ben: 06:06 You can... there's this new initializer called content security policy, which allows you to configure that header being sent or not. And one of the things that's cool about the way that Rails did it is that one of the things that content security policy allows, specifically for in-line scripts and styles, is this concept of a nonce. So, if you actually do want to have in-line script run, because for some reason you like to work that way, I don't know, you can actually say, "This particular in-line script is safe to run, so run this, but not anything that I don't declare as safe." Right?
Josh: 06:42 Mmhmm. (affirmative)
Ben: 06:43 And so you could use nonces for that and basically it's just like this random string that you generate that says, "Yeah, this particular JavaScript section is safe to run." And what Rails does is it generates that nonce for you, and then there's actually a helper. A view helper you can use to wrap your JavaScript that says, "Yes, wrap this JavaScript inside this nonce and declare it safe." Basically.
Starr: 07:02 Oh, nice. Cause otherwise, that would be a huge pain in the neck, because you'd have to somehow coordinate your headers with sort of the content of your... I guess HTML, if you've got in-line JavaScript in that HTML file.
Ben: 07:16 Yep.
Josh: 07:17 Yeah.
Starr: 07:17 That'd be a huge pain in the neck. And the fact that-
Josh: 07:19 I love that about Rails.
Starr: 07:20 ... the fact that Rails is doing this now, that's... this is all kind of... it's not super new, but it's kind of new-ish. Like people were just now...
Josh: 07:29 Yeah.
Ben: 07:29 It's still pretty new.
Starr: 07:30 ... getting used to this idea.
Josh: 07:31 That's why I think that's the big... that's the reason we never fully... Ben and I have been tracking this for a long time, and we've been tracking it since the early... since you know it first came out, and I think we kind of saw that this is going to catch on at some point, but we don't know when it's going to reach the mainstream, basically. And I think that Rails... Rails adopting it is a pretty good sign that this is, you know, pretty much everyone's gonna know about this at some point now.
Ben: 08:00 Yeah. Yeah, definitely. I think especially with Rails 6.0, people are going to be upgrading to it and seeing this new initializer showing up in their... when they do the upgrade.
Josh: 08:08 Yeah.
Ben: 08:08 I think it'll definitely become more widespread.
Josh: 08:11 A few of our earlier attempts when we were first learning about CSP and so... there's some other browser... similar browser features that go with it. But we were initially thinking that, you know, this would be a cool standalone, kind of, new thing to build... like a product around it or something like that. Or a tool, even. And we kind of jumped back and forth between, "should we put this on honeybadger?" or "should this be like, should we make something specifically for this?"
Starr: 08:40 So what exactly is this? So we've described CSPs a little bit.
Josh: 08:45 Yeah.
Starr: 08:45 But what... where do we come in? What part are we playing?
Josh: 08:48 Yeah. So the report, the reporting feature I mentioned is where we come in. So CSP headers have a feature called report URI, which... it's that URL that I mentioned that you can give it that it will post the reports to when there is a violation.
Josh: 09:07 And so there's two ways that people use that, there's when they're blocking... when they're entirely blocking any requests that are not allowed, they can still report the violations. And they might want to know that just, you know, for security reasons, like to know that these requests are being made, even though they're blocked.
Josh: 09:26 And the other reason... the other way is that there's also a report-only mode, which will basically still allow the request to happen, but it will send the report as if there was a violation. And that can be handy if you're rolling out CSP, but you don't want to like risk breaking your entire site if you like forgot to white-list your assets, for example.
Ben: 09:48 Yeah. When I was doing some testing for this, I had a site that I used as a Guinea pig, and that was exactly my scenario. Like if there's a bunch of JavaScript already and styles being loaded on the site, I didn't want to break it, but I did want to find out what things would trigger a CSP violation.
Josh: 10:06 Yeah.
Ben: 10:06 So I deployed the app with the report-only header, and started tracking the violations, and I was able to go back and update the policy based on the things that I saw, like "Oh yes, yes, I'm going to load Google Analytics, so that's fine, I'll put that in my white list."
Starr: 10:22 So the idea is that people will set their report URI to point at our servers, and then we'll take those reports and do something intelligent with them, right? Because we already have all this infrastructure for reporting application errors, for integrating with people's chat systems, ticketing systems, stuff like that were already in people's lives. So it kind of makes sense for us to sort of handle that for people as well. Especially since it's going to be in Rails, and so this is just something that... we really support Rails very thoroughly, out of the box.
Ben: 10:59 Yeah.
Josh: 11:00 Yeah. I'm really excited about that aspect of it. Cause I assume... I don't think we've looked into it yet to build it, but I'm hoping that we can actually automatically... will automatically be able to, you know, send the CSP reports from Rails to Honeybadger without any, you know, any kind of additional configuration. At least that's how I hope that we can make it work. And then maybe even on the Honeybadger side we could like, you know, tell people what headers they need to send, if there's a violation that they're not catching, you know? And they want to white-list something, we could help them like just give them like a drop-in, you know, tell them how to basically configure it.
Starr: 11:36 So, the idea is that our gem- our Ruby gem currently goes in and attaches itself to Rails and picks up all this error data automatically. So, you know, hopefully it'll be able to pick up the CSP data automatically too.
Josh: 11:51 Yeah, I think right now we're just... We're just building the infrastructure to report... to monitor and report this data, is as where Ben's at, I think. But yeah, I think that...
Ben: 12:00 Yeah, at the moment, since we're still in Beta, what I've asked- what I've told the people who are testing it, they can do is just, "here's the report URI setting that you can use to report your violations to our stack."
Josh: 12:14 Yeah.
Ben: 12:14 Yeah. But I think adding that to the gem, like we could detect if it's enabled and then we say, "okay, just slide in this report that you run for them automatically." That'd be pretty cool.
Josh: 12:22 Here it is. Yeah. Yeah. I think one of the reasons- I think one of the reasons we didn't go down the let's create a new like CSP product route is that the data volume can be fairly large for these types of monitoring services, because if you imagine potentially every- every user who visits your website could be sending multiple requests per page visit. If there's a lot of violations to whatever your reporting services, is that right, Ben?
Josh: 12:57 Yeah, so just imagine the volume of reports that you could potentially be like hammering your API with if you're doing this with thousands or tens of thousands of customers to a single service. So we thought like, you know, do we- we already have one ops-heavy startup, do we really want like to have an entirely separate one? And the answer was no, let's just roll it into our existing one and handle the traffic anyway.
Ben: 13:23 Yeah. Another consideration was that the use case for this is a lot of people want to know that this violation happened, right? Once you have the enforcement happening, it's actually an error, right? It could be someone trying to hack into your site when you see a violation pop up after you've done all the white-listing work. Right? So the idea there is we already have all this infrastructure for people to be alerted for errors, so why would we ask our customers to set that up again? Right? You know, you wanted to get- check this other kind of thing that looks like an error and feels like an error separately from your errors. Well...
Josh: 13:57 Mmhmm. (affirmative)
Ben: 13:58 Yeah. But the one- the reason why it wasn't a slam dunk and we thought about a long time whether to keep it separate, was the other issue- the other side of it is these payloads are very small. They don't have back traces. They don't really have any, you know, actionable information that we're used to having with a typical error. Like, you know, there's not context, there's not, you know, a lot of that kind of stuff. So we're like, "well, it doesn't necessarily fit like a glove into our current thing."
Ben: 14:25 So we had to basically create a shim where we take the information that we do get from the browser, which is pretty minimal, and kind of shape that into what we're expecting for notification. So we can't just make up a back-trace, right? But we can put in things like, "oh well the user agent header can turn into the the user agent environment variable that we usually get from a server psych" and things like that.
Josh: 14:49 I really like how you're reusing our fingerprinting system too. So that- because our fingerprinting system is how we actually de-duplicate or de-dupe errors- error reports. So if- if you have like a bunch of errors that are reported from your application that are like the same- from the same location in your app, you don't want to get 50,000 error reports, you want to get one error report.
Josh: 15:12 And so we use fingerprinting to kind of analyze the error data that's coming in and determine if it's the same error or if it's different. And we're kind of doing the same thing, basically reuse- repurposing that system to de-dupe the CSP reports, right?
Ben: 15:24 Yeah. Yeah, and it's, it's still not perfect. Like they're still- looking into my sample project and I can see that there are a bunch of violations that look very similar, but they're somewhat different, right? So, I don't know that I've totally nailed that yet and that's one reason why I haven't released it for GA, but, but yeah, there's definitely a lot of volume like you mentioned and if he didn't de-dupe that, it would just be noise.
Josh: 15:48 Yeah. Well, I'm sure as we get more data to work with, those fingerprints will evolve just like they did for errors too, because-
Ben: 15:54 Yeah.
Josh: 15:55 ... as I recall, we had this same situation where as you get more data, you see more edge cases and you can build that into the algorithm, basically.
Ben: 16:02 Yeah.
Starr: 16:03 So question. How are we dealing with the fact that the service is going to have so much traffic? Like it's conceivably gonna have a lot more traffic than our error reporting or inbound error reporting system. What- like how are we managing that?
Ben: 16:23 Well, I mean it might have significantly more traffic. It's hard to say yet, because I mean, if you look at our JavaScript client traffic, we get a lot of traffic from JavaScript, right?
Starr: 16:33 Oh, yeah, that's true.
Josh: 16:34 Yeah. Which, by the way, is a significantly small portion of our actual installs. Like most- a lot of people- we have a lot of server side customers and relatively fewer buy inside. But the client side is like a large portion of our...
Ben: 16:49 Right. Yeah.
Josh: 16:51 Everyone, don't go out and install the client side reporting at once, please.
Ben: 16:55 Yeah.
Starr: 16:56 So you're saying we should ditch JavaScript and just go back to straight Rails.
Josh: 17:00 Oh, can we please?
Ben: 17:02 Let's start a no JS movement, right?
Josh: 17:04 Yeah.
Ben: 17:05 So yeah, we might- we might be buried in traffic because of- yeah, like Josh was saying that-
Josh: 17:11 It depends who uses it, right? Like, or how fast it gets adopted.
Ben: 17:14 So you know, we have our- our normal auto scaling stuff in place and so we can handle, you know, increases in traffic pretty gracefully. But I'm also looking at maybe using lambda for this. So, back in November of 2018, AWS announced that you could, in fact, connect an application load balancer and route that traffic directly to a lambda, rather than having to use API Gateway, which saves you buckets and buckets of money.
Josh: 17:43 Oh, that's cool.
Ben: 17:44 Right. If you are using a lot of API requests. And so we might be able to do that. I know we- like Josh, way, way, way long time ago looked at using a lambda for some other high volume stuff and like the API Gateway costs were just ridiculous.
Josh: 18:00 Well, that service is a lot like this service, actually. That was our log plex service, which takes basically like a Heroku log drain and scans the logs for error-like events, like on Heroku's side, and then it's the same situation where it's not- it's not an application exception, but we kind of massage it a little bit and convert it into one if it's- if it meets our criteria and then forwards it onto our API. But yeah, the like API Gateway was going to be crazy expensive based on the amount of traffic that we would receive.
Starr: 18:30 So, the Heroku logs' basically like we get people- we send people error alerts if like Heroku times out.
Josh: 18:36 Yeah, time's out. There's, there's a whole class, like a class of errors. There's like, I don't know, 20 or so, I think, different error codes that you can get with Heroku, that's like before it reaches your application, like your applications execution.
Ben: 18:50 Like a memory- memory use, like if you run out of memory.
Josh: 18:53 Yeah. Memory, yeah.
Ben: 18:55 Yeah, so that was a nonstarter. We tried that back- back a few years ago, but now that you can use an application load balancer instead and just send that traffic directly to a lambda function that that might be viable. And the, the benefit there is that, you know, Lambda could scale to infinity, right? At least that's what the marketing materials say. So, we don't have to worry about spinning up instances in an auto-scaling group like we do today. So that's one less concern. So, that could be interesting. So I'm playing around with that.
Josh: 19:26 If that is a successful experiment, I could see us moving a lot of stuff into there actually. Like, I mean we can move log plex into there, back in, you know, back into Lambda, we could potentially move our entire collector someday.
Ben: 19:38 Yeah, we could.
Josh: 19:40 You know, that's pretty exciting.
Ben: 19:42 Yeah. It gets us to the goal- closer to the goal that Starr has of no infrastructure, right? Just put it all on Amazon's plate and let them do it all. Yeah?
Starr: 19:50 Yeah, I should probably explain that Lambda is Amazon's serverless play where instead of administering a server, you just give them some code and they run it whenever a certain event happens that you define. So you know, whenever somebody goes to our URL with some payload data, the load balancer would then run, you know, tell Lambda to run this bit of code, send the data to it. And you know, bada bing bada boom.
Josh: 20:15 Yeah, yeah. It's- some people refer to that as functions as a service. F-A-A-S.
Ben: 20:21 Yeah. So we'll see. I mean, so right now we're just, it's deployed on our current infrastructure since we're, you know, not expecting a whole lot of volume from our test users. And we might stick with that because it's been working well for all these years and we just keep on working at. But you know, Lambda is interesting. So I'm playing with it.
Starr: 20:39 Even though Lambda sort of bills by invocation, it's still possibly doable in terms of price?
Ben: 20:46 Yeah. So, there was a great presentation at ServerlessConf San Francisco about your utilization of your, let's say- let's say you're comparing costs for, like in our scenario, running EC2 instances all the time versus doing that same kind of workload in Lambda. And, so the concern is if I have a per-invocation cost, then that's going to be way more expensive. But in reality, like usually many loads in EC2 are, like, 20 to 30% CPU utilization, right? So you're paying for a lot of idleness, in that case. And so the theory is that if you're not paying for that idle, if you're paying, you know, per invocation, then it can actually work out. So we're just gonna test it and see what it looks like and maybe it'll work and maybe it won't.
Josh: 21:33 Yeah, that'll be, that'll be good to know though. Cause this is probably like one of the- of all the things I can think of, like BIS, like CSP and log plex are the two things that I could see like, I mean, that's going to be the highest number of invocations that we would ever, ever use.
Ben: 21:48 Right.
Josh: 21:48 I, well, I guess I may have just screwed us.
Ben: 21:52 Until the next thing, that's the highest.
Josh: 21:55 Yeah. Well, we're gonna put our- like we're going to bring back metrics and put metrics into Lambda or something.
Starr: 22:02 I guess one thing that would be neat is if there were way to- in a nice serverless automated way to just kind of like batch the requests. Cause I don't- like for CSP, it doesn't really matter if it's a second layer. It would be nice if they could sort of batch requests into groups of a hundred and then you only have...
Josh: 22:21 This is why we pay Starr the big bucks, because he's- he's saving- this is how he saves us money. He's like- he's like, "what big bucks?"
Starr: 22:30 Stay tuned at the end of this episode for my phone number.
Starr: 22:35 No, no.
Ben: 22:37 Yeah. So, I mean, we've been working on this, like we- like I mentioned for years, literally. It was funny because I didn't really realize that until I went back to our customer support and you know, we keep track of all those things, and everyone who requested CSP along the years, I went back and told them, "hey, now you can come and test it."
Ben: 22:54 And I just happened to notice that the oldest request was three years ago. So I was like, "yeah!"
Starr: 22:58 Oh my goodness.
Ben: 22:59 Three years in the making. This was a well- a well- a good vintage feature.
Starr: 23:04 Where are we on this? Like is there a- do you have sort of plans to release this general availability anytime soon or is it we're just gonna wait for a while and play it by ear?
Ben: 23:14 Yeah, I mean I don't, I don't think I'll take us long time to figure out whether it's working for these- for these tests, the assessors, so you know, a week or two.
Starr: 23:22 Unless it's not working.
Josh: 23:23 I thought you were going to say we're going to do it live today.
Ben: 23:25 Oh, like do it right now, like while we're recording the podcast.
Starr: 23:26 On the podcast.
Josh: 23:27 Oh, we should do that. You know? We should ship it on a podcast though, on a Friday.
Ben: 23:32 On a Friday. Well, I mean, that's kind of cheating, cause it's already shipped.
Josh: 23:36 Right, I know. Well, I-
Josh: 23:39 We'll cut that out, Ben. Shut up.
Starr: 23:41 We could- we could just tell people that we're shipping it.
Ben: 23:46 Speaking of, though-
Starr: 23:47 Yeah, exactly.
Ben: 23:48 ... that is a strategy that we've used time and time again, just launching darkly? And I love that strategy. You know? Putting it out there in production, seeing how it works before you tell the masses about it and it works out great for us. Like if we can find out any edge cases like what you were talking about earlier and maybe some bad mistakes that we could avoid.
Ben: 24:08 There've been times where we've put something out there and get some real data in there, and it was like, "Oh! That was a bad idea." You know? Launching darkly really helps with that.
Josh: 24:16 Yeah. If only we could do that with changes to the notice pipeline or the fingerprinting thing I mentioned, like deploying anything that touches that, like the finger- you know, the fingerprinting or anything that could instantly change how many notifications people get on any given second-to-second basis is terrifying.
Ben: 24:37 Yeah. Yeah.
Josh: 24:38 But I don't really see a way around that, you know? I mean, I guess maybe there's something, but... yeah. We- we got to live with a little bit of uncertainty.
Ben: 24:48 Yeah.
Starr: 24:49 Well, this is cool. This is, you know, I've learned some stuff about CSP that I didn't know before and yeah, I'm really looking forward to seeing this feature in production and everything.
Josh: 24:59 Yeah, me too.
Ben: 25:00 Yeah, that's pretty cool. Like the test site that I was using it on. It's actually pretty neat to see the violations come in and be like, "Oh yeah, I do want to have that particular JavaScript loaded" or "let me take a closer look at that and see what it's doing" because it's- yeah, like Josh mentioned the- the eval thing. A third party script can inject some- some script into your page and maybe you don't want that.
Josh: 25:24 Mmhmm. (affirmative)
Ben: 25:24 So yeah, it's pretty- it's pretty cool to see it in action and to say, "yeah, I feel better now that that site has that in place."
Starr: 25:30 Yeah. And even though like this stuff seems like kind of big company security audit type thing, this is just a reality that we're living in now that the web is becoming more and more sort of regulated. There's more and more issues of compliance that people have to deal with, and you know, part of that is security best practices. And so, you know, here comes along CSP, so as much as it sounds not very- very fun to just set that up and like look at all these alerts and stuff, that's just- that's just the way things are going I think.
Ben: 26:06 Yup.
Josh: 26:07 Yeah. Well, that's why it's nice. Frameworks like Rails that are kind of making it turnkey and services like Honeybadger that can plug into that also as a turnkey monitoring solution.
Starr: 26:19 Yeah.
Ben: 26:19 Yeah.
Josh: 26:19 It makes it a lot easier.
Ben: 26:22 We love our customers and so we made their lives easier, by providing them with these awesome tools.
Starr: 26:30 It's a shame we don't have video cause we could flash a 1-800 number.
Josh: 26:32 Yeah. Except who would answer it?
Starr: 26:36 Ben.
Josh: 26:37 But the answer is Ben.
Ben: 26:38 I wasn't going to say anything, but yes. I think we all know the answer to that question.
Josh: 26:43 Yeah.
Starr: 26:45 All right. Well, are we... are we good to go on this one? Do you think we should put in-
Josh: 26:49 I think so.
Ben: 26:50 That's a wrap.
Josh: 26:51 Ship it.
Ben: 26:52 Ship it.
Starr: 26:52 Okay. See you later, guys. Have a good one.
Starr: 26:58 Well, that wraps up our show for this week. Those boys really sound like they know what they're talking about, huh? Want to help us out? Head on over to your podcast service and give us a review. Any questions or comments? We're on Twitter @FounderQuest. Thanks!
Announcer: 27:12 FounderQuest is a weekly podcast by the founders of Honeybadger. Zero instrumentation. 360 degree coverage of errors. Outages and service degradations for your web apps. If you have web app, you need it. Available at Honeybadger.io.
Announcer: 27:26 Want more from the Founders? Go to founderquestpodcast.com. That's one word. You can access our huge back catalog, or sign up for our newsletter to get exclusive, VIP content. FounderQuest is available on iTunes, Spotify, and other purveyors of fine podcasts. We'll see you next week.
