In the second part of our conversation, Mr. Soto walks us through what MDR looks like when theory becomes reality: from web shells and overnight investigations to forensic reports that help customers understand exactly what happened.
See part one for an explanation of what ESET MDR is like as a service.
See part one for an explanation of what ESET MDR is like as a service.
BT: We're now kind of getting to where something actually happens. I'm very curious to understand what you see while leading a large security team. What's the craziest thing you've seen happen within a customer's environment?
MS: [laugh] The craziest thing that I've seen happen? Let's see… in one particular instance, I believe, the customer had a misconfigured server, which allowed an attacker to drop some web shells into their environment, essentially trying to establish remote access. Working with one of our analysts through the night, they were able to identify it, get everything situated, and then get them [server/client] back online.
Multiple aspects made this interesting. One is just the fact that we were able to work with the customer in the middle of the night, get multiple members of their team in place, and then get them the solution rather quickly. Two, we were also able to go in and do digital forensics within the customer's environment, providing them with a full analysis report of everything that had happened on that specific device to give them a clear picture of the timeline of events. So, that was probably one of the more interesting ones, because the customer actually got to see and directly benefit from the multiple activities the service covers.
When you say working through the night, is this multiple analysts from various time zones, or is it a dedicated team that's just staying up until the job’s done?
In this specific case, we did have a dedicated analyst that was working with them overnight, so one specific analyst who was on duty at that time, but it was kind of like an ongoing investigation.
There was, essentially, a handoff at a certain point, when that analyst needed to finish up their day. So, as the next analyst was coming on duty, he was able to synchronize with them, hand off the information, and then they were able to continue the customer comms. Now, it did involve communication through our ESET Services sub platform, going back and forth, sending messages, as well as multiple calls with the customer and their team. And along the way, we also had other members of our own team jump on said calls as well, continually working with the customer until the case was fully resolved.
In terms of the end result, you mentioned a forensic report. Could you explain or summarize what the customer got as a result of this response process?
Sure. The customer got an analysis report that had essentially a list of artefacts that were found on the affected server, a timeline of events as seen by our back-end team that's doing the digital forensics, giving them the full picture of everything that had happened, ensuring that the correct remediation actions were taken. Also, a list of additional recommendations that customers can do to harden their environment, just to kind of wrap everything up in a nice package so that they can see the incident from a higher-level view.
A couple of people have likened ESET MDR to an iceberg. You've got the visible bit above the water (the security analysts), then you've got the service package itself, but underneath that, there’s a lot of organizational knowledge, like threat intelligence, telemetry, and the customer service team, making sure there's consistent communication. Could you just walk us through what that means?
From my point of view, I would say, the tip of the iceberg might be something like the detections the customer is able to see inside ESET PROTECT or ESET Inspect. But at the back end, like you're mentioning, we do have all the telemetry data. It's things like that, plus knowledge from the analysts, being able to see what's typical behavior and events across multiple environments, that maybe one particular customer might not have the benefit of seeing.
Our understanding of what looks like typical behavior in a client environment makes it easier for us to spot anomalies that might be linked to an incident. And then, on the other side, we also have the researchers, who have access to the telemetry data on the back end, and things can happen where they're seeing something that is very concerning, so they reach out to our team and say: “Hey, we see one of your customers showing signs of an APT group inside of their network. It'd be beneficial for you to make them aware and then help them out.”
This has happened a few times, resulting in us reaching out to a customer and then helping them mitigate a threat before it got too severe. They actually turned out to increase their level of services with us in the end because of this.
That's over and above the regular threat intel reports that go out with the detections built into them, I’m guessing. More proactive, at least. Is that the case?
Yeah, it is something that they're seeing on the back end, actively saying that we need to bring this to the customer's attention ASAP, because it's something that's actively going on in their environment.
Each customer has its own behaviors. Could you talk about a time when something happened in a customer environment that would have caused an incident elsewhere, but was just a Tuesday morning for their way of working?
Definitely. Different organizations often have administrators with their own ways of managing networks, and those workflows can vary quite a bit from one environment to another. Some things that may look more like an anomaly, or suspicious, that we've seen in one of our customers’ environments might be an admin using, for example, PSsExec and some really odd scripts to try and complete some regular administrative function.
The nice thing about that, though, is that we do have assessment forms filled out by the customers during MDR onboarding, so we get an idea of what types of tools they're going to be using specifically, and then if there’s something that we need confirmed, we'll reach out to them just to make sure, like “Hey, is it you actually doing this, or is this something that we actually need to respond to or take further action?”
AI is also obviously changing the security landscape, including the way attackers prepare, automate, or disguise parts of their activity. From an MDR point of view, is that already making analysts’ work more complicated?
Yes, from an MDR perspective, the threat landscape is constantly evolving. In fact, the use of AI can have an impact on the complexity of an analyst’s day-to-day work in multiple areas.
For example, by using AI, the attackers now have the ability to craft more sophisticated phishing emails. Likewise, they can also accelerate attacks by increasing their skills with the help of AI, allowing these AI-enhanced skills to help with identifying new zero-day exploits.
Also, identifying anomalies of some of the more complex users, such as developers, can be tricky at times. With the use of AI by end users, this can change the standard user to mirror said developer-type activity on their endpoints, making analyst work more difficult.
From the analyst usage perspective, AI provides benefits by helping reduce manual effort and narrowing focus by aggregating related indicators to a single incident. However, each incident still requires careful analysis and validation, as analysts must verify the accuracy and relevance of AI input before taking action.
Have you ever been involved in an incident where you've got a feel for how the attacker was behaving, and they've done something that's particularly impressive? Or are they all a bit mid, a bit average?
I believe, in one of the instances, we've seen some pretty advanced encryption and obfuscation being used to where we had to escalate it up to the tier-three team, and even they had some difficulties trying to get everything fully de-obfuscated and decrypted to identify what the malicious executable was really doing.
Then, sometimes, we see scenarios where the attackers look like they're operating at a very basic skill level. Maybe they already have remote access, and they're trying to execute some commands, but make some typos. And we've seen it before, where the activity will look like a lower-level skilled attacker, and then down the line, things get really advanced, and then start picking up speed really quickly.
Could you give me an example of where you've seen a customer do something clever—or maybe, even something not so clever?
For the most part, I would say, some of the things that we see customers do that they shouldn't be doing is having their ports exposed to the internet. So that's something where they might have Desktop Protocol (RDP) open to the internet, and maybe not even have Multifactor Authentication (MFA) in place. Those are some of the things that we do see on our end and provide recommendations ASAP.
As far as something clever that we're seeing the customers doing, I think the main point is that, as long as they're maintaining communication with us, and doing that rather quickly, it’s the best thing they can do. If we are giving them a set of recommendations, the quicker that they are able to respond, implement, and confirm that they've made those changes, the better, because we only have so much control on our end.
So, if we're giving them a set of recommendations and they're not adhering to those, then it's just going to be something that could potentially turn into a really bad scenario.
Many small and midsize businesses think MDR is out of reach to them, whether because of expense or technical hurdles. What do those that choose MDR give as their reasons, and what is your message to companies considering it?
The SMBs I talk to want to make sure that they have visibility into how things are actually happening in their environment, to keep being protected. There are also several other things that do come up, such as compliance that they need to adhere to, or there could potentially be some other incident that they've encountered in the past that's kind of bringing them to the conversations with us, or it could just be that they are a security-focused IT admin that knows the true benefit and value of having an MDR team on their side.
As for MDR seekers, the best answer I can give them is that when the detections come in, they’re only seeing a piece of the picture. So yes, something may have been detected, and yes, we said we had stopped an attack. But where it actually came from, how it originated and got onto that specific device — that is the window that's still left open that they need to make sure is taken care of. Closed, so that the attacker is not going to try some different technique in order to get into their environment again.
You mentioned detections, which brings me to the people behind MDR: the analysts doing the response, forensics, reporting, and customer communication. What is the human value of an MDR service in terms of the people that it brings to support the customer?
Oh, the human value is definitely very important. It's somebody that the customer is going to have the ability to talk to, work with them, to communicate with them, breaking down technical terms to something that the customer can understand.
It’s also not just one analyst, but a whole team of them. We have multiple tiers of our security services that are, essentially, at the customer's disposal, instead of having to train somebody locally on their team. ESET MDR is not just about having somebody that's ready and available 24/7, but someone they can utilize, and have the option to just call up and talk to… receiving an actual human response about the active incident.
How does it feel to help a customer in that situation where they've got themselves into a bit of a pickle, there's an attack underway, and you've saved the day. What's the feeling you get when you finish that engagement?
For me, personally, I would say, most of the credit goes to our analysts. They're the ones that are on the front lines stopping these threats, getting on the phone with the customer, explaining exactly what happened, and walking them through how the threat was stopped.
Being the team manager, I think it's amazing, because I get to see the customers’ response, how happy they are with the service, how clearly they see its value. But ultimately, it comes down to the analysts, especially the tier-one analysts. Anytime they’re able to step in and mitigate something, it’s an awesome feeling. We always try to recognize and encourage that, because the work they do really matters.
Mitsuyuki, thank you ever so much for this interview. I think I might need to kidnap some of your colleagues at some point… sounds like your team has lots of stories to tell!
Sure, thank you again for inviting me. Happy to talk, anytime.
That, perhaps, is where the value of MDR becomes clearest. Behind every detection, report, and late-night call, there are analysts helping customers make sense of what is happening, close the gaps attackers might try to use again, and get back to business with a little more confidence than before.
For more on how ESET MDR supports customers, see part one of our conversation with Mitsuyuki Soto.
For more on how ESET MDR supports customers, see part one of our conversation with Mitsuyuki Soto.
