One of my favorite things to do so far this year was benchmarking security software and testing how it integrates with real environments. This time, the security software of choice and the mode of testing is the next-generation AV proof of concept.
NGAVs are essentially a collection of several different security tools consolidated to provide a better picture of your device and network health.
It's worth mentioning that my notes below represent the offering of the products and the effectiveness of the customer success team in this period of time (early 2019). Your interaction may vary, especially if you see this much later. A lot can change quickly in this business.
Table of Contents
Crowdstrike POC
Here are my notes from the initial introduction and my time playing around with Crowdstrike Falcon:
- 3 main services: SaaS Endpoint Protection, Threat Intelligence, and Cloud Security
- Light installer and low memory usage
- AD integrated and can assign policies by OU
- Requires policies to be switched on to be effective. They do not set up or walkthrough policies
- Training courses and YouTube videos are available
- No references and no naming names
- Policy updates and the sensitivity of the software
- Need to restart to have the machine appear in the console
- The client is called Falcon, and there are different package offerings
- Auto-quarantine executables
- Easy exclusions
- Keep a history of scripts run, registry effects, and web origination information
- Need responder privileges as well as admin to issue cleanup, not readily apparent
- Has a threat group lookup page for known threat actors
- SIEM and Syslog integration
What I Liked
The agent installation was a breeze. The hardest part of the process was removing the previous installation of the antivirus solution we used. You don't have to restart the machine for the installation to finalize; however, if you want the computer to show up in the console, you need to restart it.
The low resource requirements for running the agent are nice. The AV we were using was such a resource hog that it ran multiple different processes. I also liked how you can run the agent (Falcon) silently. It was this way by default (I don't recall if you could disable this).
Managing and assigning policies is a nice touch (once they are turned on). I came from manually creating policies and jumping through hoops to edit and save them. Once we got the protection piece working properly (more on this in the next section), I enjoyed the easy exclusions for false positives and the info on the true positives. The group lookup page for known threat actors was a nice touch for investigating the background of certain malware strains.
Regarding the meat and potatoes, I really enjoyed the history and web origination piece. This was leagues ahead of what I had seen at the time. The historical information was easy to find and laid out nicely.
I didn't get a chance to integrate this into a SIEM, but I did forward some logs to see how easy it would be. There were no disappointments here.
What I Didn’t Like
Even though the policies were easy to set up and assign, they require being switched on (this isn't the default like other AV software). This was not readily apparent in the UI. Our contact didn't do a good job explaining this or helping us set up anything, which is surprising for a customer success engineer. I eventually figured this out after seeing some test machines appear in the main console.
Speaking of our contact, the customer service success team did not do a good job introducing the software or explaining how we could use it. Most of my notes above were from me asking quite a few questions. I'm sure there was more information about the product and service, but getting to the point above took an hour. I'm guessing this person was new and didn't know how to onboard people. This wasn't the worst part, though.
After reviewing several tests and noticing that the software wasn't stopping anything I threw at it - RATs, ransomware, scripts, and other viruses, I shared my concerns with the contact. Surely, I was missing something since it didn't catch anything. They didn't know what the problem was (at least initially) and offered to send me their malware to see that the software did indeed work (LOL, nice try).
It turns out that despite telling this person exactly how I would test it, including replacing the current AV, they didn't bother telling me that the endpoint protection part was not enabled by default and that the switch to turn it on was in its own settings menu. Also, after the product is fully enabled in 2 different spots, the person reviewing and cleaning the threats needs the responder privilege and the admin privilege to issue cleanup commands to cleanup found threats. This is the first time I've seen the admin group not include every privilege. I don't mind taking part of the blame for this, but if you are a hands-on person with a lot on your plate like myself, you can understand my frustration.
After getting everything set up properly, I re-ran the tests. This was a lot of extra work for me, but I like to be thorough and properly test and set up things. I just feel that I shouldn't be the one who cares more about this than the customer service success person trying to sell me the product.
The information and documents seem to be scattered. They have training courses, a YouTube channel, and the threat actors page—all of which are in different locations. Whenever I had a question, I was referred to the materials to look it up. It seems like having a knowledgebase with everything together would be more beneficial.
On that note, things were scattered, and so many clicks were required to get to certain views, reports, and settings. I'm not convinced that all the info they provide is necessary, especially how they lay it out. It's almost like the product was developed and designed by an engineer without a user interface or designer testing. I'm sure more feedback on this will help shape their offering for the better.
Final Thoughts
Despite my "Didn't Like" section being longer than the "Liked" section, I really do like this company and its offerings. It has a decorated history, and its solutions have worked for many. Once I got everything set up properly, things worked well. They were just difficult to get used to and use.
They wouldn't be my pick now, but I'd be happy to revisit them in the future. I'm positive there is much more to this product and service than I got a chance to see. Working with a better customer service success engineer and a UI update would go a long way toward improving this experience.
Update 5/19: Crowdstrike gets an IPO filing. More funding for continued research and operations is always a good thing.
Cylance POC
I had heard a lot of great things about Cylance Protect, so I was excited to give them a try. This POC was a couple of weeks after the Crowdstrike POC. Here are my notes from the initial introduction and my time playing around:
- Artificial Intelligence-Based Advanced Threat Prevention
- They can recommend references from organizations of similar size
- They're mostly able to name names
- Can do a contract. No reason to go out to bid
- Memory usage
- 116.8 MB for the Native Agent
- 4.1 MB for Cylance Protect
- There isn't a Windows Defender requirement, but they can run side by side
- Azure integrated with SSO
- Policies can be assigned by zones
- OUs and more
- Optics is more offensive
- MITRE framework capable
- Automated action
- Query packages and push exes
- Detailed threat data
- Evidence based
- License for these products are separate
- Protect and Optics
- Check their documentation area for admin guides on Protect and Optics (product names)
- They like to put it in monitor mode at first when they set it up and then switch it over as they onboard
- Watch for files and stop new executable
- Auto-quarantine executables
- Protection with execution control
- Script control by file, folder, and group (zones)
- Seamless exclusions
- SIEM and Syslog integration
What I Liked
This customer success engineer was the complete opposite of the last one. I had 33 questions prepared and only had to ask a few. This person was so on the ball that I had trouble keeping up. Not only did they introduce the product and service, but they also helped set everything up and gave a brief tour of the interface. Very nice. The ability to give references went a long way as well. I realize that not every organization would want to be a reference, but it was nice that Cylance had a couple of similarly set up organizations to refer us to.
As far as product offering, it's very similar to Crowdstrike. The resource requirements are also low, and the agent runs in the notification area. It's nice to see it and what the agent is working on (by hovering the mouse over).
Managing and assigning policies are nicely set up through "zones." Zones can be set up through more than just OUs, so I like the flexibility. The Optics piece is really versatile and automates a lot of the boring stuff. Of course, the Protect side is fully capable, as it caught all of my tests and was enabled by default. The engineer did ask if we wanted to disable it so we could run it side-by-side with another AV solution. We, of course, said no.
Regarding the meat and potatoes, the history and script controls are a nice touch. I also like the initial scan and executable control. They seem more flexible on the actual operation side of this newer security process. The interface is simple, and it's easy to find stuff. They don't go too crazy trying to shove a bunch of analytics in your face.
The knowledgebase and documents were centrally located and easy to find. When I asked questions, they answered them and also provided the documents for further reference, which was a nice touch.
I didn't get a chance to integrate this into a SIEM, but I did forward some logs to see how easy the integration would be. No disappointments here as well.
What I Didn’t Like
The initial scan was a full scan, and it's required to move forward. That in itself isn't a bad thing, but doing this full scan was extremely slow. This is expected since full scans are CPU and drive intensive, but I'm surprised this is still a problem in 2019.
The interface looked old and dated. I think they mentioned an update was coming, so this isn't as big of a deal.
That's all I can think of for now. I know this section is really nitpicky but there weren't a whole lot of negatives during this next gen AV proof of concept.
Final Thoughts
Even though Crowdstrike offers very similar products and solutions, Cylance really closed the gap for me. They excelled in everything I had previously had a problem with using Crowdstrike. Playing around with Cylance has been a blast and the customer service success engineer was awesome. After talking with a few people who use Cylance, it's clear why they are where they are. Setup was a breeze and everything worked well.
If I had to make a decision now, I would pick Cylance. The proof of concept was so smooth that it's almost unfair that I engaged this solution right after the sub-par experience with Crowdstrike. Even though that is my experience, I would still recommend organizations looking to make the switch to a newer antivirus solution engage in both of these companies for a next-gen AV proof of concept at a minimum.
Update 3/19: As I publish this article, I see that Blackberry recently acquired Cylance. This is interesting. I always keep an eye out for mergers and acquisitions. This could be because I graduated with an MBA with international honors, or it could be that mergers and acquisitions are commonplace among tech startups, SaaS companies, and security companies. More funding for continued research and operations is always a good thing.
Other Next-Gen AV Proof of Concept Options
If you need to find more than 2 options for an AV solution, here are a couple more options:
- Carbon Black Defense endpoint security with big data analytics.
- Update 8/19: VMware acquires Carbon Black
- Endgame Platform larger scope endpoint protection.
- Update 10/19: Elastic acquires Endgame
The next time I need to do this, I would like to test the additional options above, time permitting of course.
Update: Wow, 2019 was a good year for endpoint protection companies. So many acquisitions.
Update 2: One term that I'm starting to see more and more is EDR - Endpoint Defense and Response. If you're searching for options to do a next Gen AV proof of concept, you may wish to search for next gen antivirus, NGAV, endpoint detection and response, or EDR.
