Is Windows Defender Up To Par?

How Effective Is Defender?

Short answer: Effective Enough

Long answer: Read on

Testing Methodology:

    I have a bunch of random malware samples on my PC ranging from Mimikatz to ransomware so we have a lot to work with! I'm going to be testing initially with Sophos vs Defender and our first test is to download Windows 10, boot up a VM, and see if we can download the renamed Mimikatz exe. As seen below, this was immediately flagged.

    Not only was the exe flagged, so was the zip file. So let's try renaming everything in the source, removing Mimikatz, Powerkatz, all the names from the authors, everything.

    Now these are well known files with signatures everywhere you look to detect them. How does Defender do with something completely new? For this, I wrote a quick ransomware emulator. It just encrypts a random file I made on execution.

    Here is where Defender and Sophos started to differ. This sample, when converted to an exe, was caught by 16/68 products... That's not great. Among those that caught this were Defender, FireEye, and BitDefender. And for some reason McAfee. This was unexpected. I had high expectations that even common AV would be able to identify an executable encrypting arbitrary files. But Sophos and a number of others fell flat here.

    I thought that maybe this was only doing some sort of static code analysis only and that run time protection on the VM would be able to block this, but no. I was able to encrypt the random txt file. At this point I wanted to make the simulation a little more real in the hopes that other AV's might catch this new behavior more accurately. I changed the path from that random txt name to encrypting all txt files in the folder. This still wasn't caught... This isn't entirely true though. Each time this was run, SmartScreen kicked up an error since the code was missing a cert, so point to Defender there.

    The last piece I want to test is phishing detection. For this, I pulled a random submission any.run that was flagged with phishing. I tested Defender first and in Edge the site was detected! But, in Brave, the site was not flagged as such. It would be nice if this functionality was extended to other browsers but this is a pretty big lift. On the other side, Sophos didn't detect this at all and let me proceed normally.

Conclusions

    In all of this, Defender has scored pretty high marks from me and has evidently improved leaps and bounds from where it was even a few years ago. Are you fine running this? Most likely. It picks up on the simple traits in my exe and accurately flags it as dangerous. I'm going to work to make my test set more obfuscated so that when I revisit this, I can test it more thoroughly. One area that it missed the mark on is legacy OS support. If you're not receiving Windows updates anymore, Defender won't be of use. Of note as well, in Gartners Magic Quadrant, while Defender scored well overall, it missed a few marks if used as an enterprise solution due to cost and capabilities. I'm going to quote them for this: 

There is a large gulf in capability and cost between SKUs providing MDE and those that do not, and some organizations are consequently unable to justify the cost premium of Microsoft Defender for Endpoint. Licensing Microsoft Defender individually outside of security bundles is also not cost-effective for these customers. This led to a lower score than the leading vendors in this Magic Quadrant for Sales Execution/Pricing for 2021.