Troubleshooting Configuration vs. Interop for Endpoint Standard

Introduction

Review this resource to learn troubleshooting best practices for your enterprise. This resource is intended as guidance to understand order of operations, how best to resolve your issue without over exposing your environment.

This guidance is intended for Endpoint Standard customers.

Diagnostics

To get to the root cause of an action that is causing disruption we need to assess the symptoms and determine if it is one of the following:

Configuration: Is there a behavior or reputation reason that Carbon Black policies are blocking my application from operating as expected?
Interoperability: Is there an application-level conflict / incompatibility causing application slowness?

Once the cause is identified we want to respond in a manner that will introduce the least amount of security risk while restoring system functionality.

Configuration

To determine if your issue is caused by configuration you will visit the Alerts section of your console. Sort your view under Policy Status select “Policy Applied”, and Sensor Action select “Deny” and “Terminate”.

This subset of alerts will now highlight all actions that have been taken on your environment on the behalf of the VMware Carbon Black Cloud Sensor. From here you need to identify if this block is due to Reputation or Behavior of the targeted application.

If your target is not in the alerts skip to Interoperability section.

Reputation-Based

Reputation is determined by analysis of pre-existing, newly introduced, or accessed on the network. There are 3 stages in which Carbon Black Cloud will assess and assign a reputation:

Resident: Pre-existing file on a system that is detected by RepMgr.exe as a part of a background scan. There is no attempted run of the file.
Dropped: Newly introduced files to a system.
Ran: Attempt to access, modify or delete a pre-existing file on system.

Does the application in question have an untrusted reputation?

Reputation Definitions

Reputation	Description
Banned	Hashes manually added into Company Banned list.
Known Malware	Known bad by Carbon Black from the cloud or/and local scanner.
Suspect/Heuristic Malware	Suspect malware detected by Carbon Black, but not necessarily malicious.
Adware/PUP Malware	Adware and Potential Unwanted Programs detected by Carbon Black.
Not Listed / Adaptive White	Hash not previously seen and local scanner does not classify as banned.
Unknown	Sensor observes file drop, but does not yet have reputation from the cloud or local scanner.

Behavior-Based

Behavior-based alert(s) will apply to any Trusted application that is being blocked due to abnormal application behavior. Blocking behaviors include but are not limited to the following:

Behavior	Description
Runs or is running	Process operating on the endpoint.
Communicates over the network	Network activity caused by the process
Scrapes memory of another process	Targeted attempt to read memory of processes such as lsass.exe. Could indicate credential scraping.
Executes code from memory	Untargeted attempt to run code from dynamic memory.
Invokes an untrusted process	Untrusted application or process is accessed.
Invokes a command interpreter	Attempt to use a shell / command line tool.
Performs ransomware-like behavior	Access Carbon Black Cloud decoy files, attempt to write to the main boot record, attempt to access Volume Shadow Copy Service (VSS). Terminate is only option because denying ransomware does not prevent further encryption.
Executes a fileless script	Use trusted process for malicious use. Also called non-malware or “living off the land.”
Injects code or modifies memory of another process	Trusted application injects code, or any use of process hollowing.

Interoperability

If your target was not included on the alerts page there is a high likelihood that you have an application interoperability. Interoperability will surface in one of the following manners on the local systems:

Application slowness
Delayed start of an application
An application operation is not performing
Application crashes

Resolution

Once the cause is identified we want to respond in a manner that will introduce the least amount of security risk while restoring system functionality.

Overview of Carbon Black Cloud Policies

Permissions allow applications at path or specific behaviors of applications to run.
Reputation determinations result from signatures, reputation lookups, heuristics, and deterministic models.
Behavioral-based controls enforced according to policy configurations for streaming analytics.
Upload unknown binaries for additional analysis using cloud-based sandbox.

Resolving a Configuration Block

Before resolving the alert the file be sure to consider the following determinations if you are uncertain of the application in question:

Context: Am I observing an Inconsistent vector, hash, certificate, behavior, directory, application type, content type, device type, username, user context this would suggest normal or abnormal activity.
Frequency: How frequently does this application run? Concentrated or dispersed occurrences over time that are consistent or inconsistent with expected information technology enablement practices.
Prevalence: Where else does this exist in my environment? One of only a few devices/types of devices involved that would suggest normal or abnormal prevalence of activity. Volume of new occurrences that develop in a given point of time.

If you have determined that the application in question has been raised due to reputation or behavior, we want to consider resolving with the concept of least privilege. The Carbon Black Cloud sensor resolves and categorizes based in order of priority review the table below.

Priority	Reputation	Description
1	Ignore	Highest priority. Files have full permissions to run without observance. Applies to Allow, Allow & Log, and Bypass rules.
2	Company White	Hashes manually added into Company allowlist.
3	Company Black	Hashes manually added into Company denylist.
4	Trusted White	Known good by Carbon Black from the cloud or/and local scanner.
5	Known Malware	Known bad by Carbon Black from the cloud or/and local scanner.
6	Suspect/Heuristic Malware	Suspect malware detected by Carbon Black, but not necessarily malicious.
7	Adware/PUP Malware	Adware and Potential Unwanted Programs detected by Carbon Black.
8	Local White	Any of the following conditions: pre-existing file, allowlist by IT Tools, denylisted by Certificate.
9	Common White	Any of the following conditions: hash not on any known good or bad list AND file is signed, hash previously analyzed AND not on any known good or bad lists.
10	Not Listed / Adaptive White	Cloud: hash not previously seen and local scanner: no known bad
11	Unknown	Lowest priority. Sensor observes file drop, but does not yet have reputation from the cloud or local scanner.

So you have your application in question follow the below scenarios to determine the best course of action to resolve.

When to use IT Tools

Defined by path of dropper, useful for developers

The IT Tools allowlist functionality is designed to permit processes with an inconsistent hash, with or without subprocesses, and a NOT_LISTED/UNKNOWN reputation. The priority 8 designation prevents malware from making its way into being inadvertently approved through such mechanisms.

When to use Certificates

Trusted Vendors

The Certificate allowlist functionality is designed to permit processes with a consistent or inconsistent hash, but a consistent certificate, and a NOT_LISTED/UNKNOWN reputation. The priority 8 designation prevents malware from leveraging a trusted certificate, otherwise affording it the opportunity to be inadvertently approved through such mechanisms.

When to use Hash

SHA256, best for static applications

The hash-based allowlist functionality is designed to permit applications with a consistent hash and a reputation determination other than known good. The priority 2 designation allows for decisive control over the reputation of an application.

When to use Allow

Resolving application behavior without removing prevention rules.

The permissions functionality allows for authorizing applications with a consistent path, that may otherwise have, or be subject to, a misrepresented reputation and has no other consistent properties as far as it pertains to hash and certificates.

Permitting an application provides priority 1 status which affords absolute release from reputation-based determinations. There two sub-configurations for this that involve allowing without logging and allowing with logging.

Allow: allows the specified behavior in the specified path; None of the specified behavior at path is logged and no data is sent to the CBC backend.
Allow & Log: allows the specified behavior in the specified path; All activity is logged and reported to the CBC backend.

Resolving an Interoperability

When to use Bypass

App Interoperability. Co-installed security solution is a common case.

If you are experiencing application slowness or instability gather information on the following:

Device Name
Date
Time
Application Impacted
Any relevant Log Files

Once the information is collected move your sensor into a bypass state.

Bypass; all policy enforcement on the device deactivated and the sensor will not send data to the cloud; or, sensors momentarily enter Bypass mode during a sensor update

If you application has moved into a stable state once the OS is in bypass then move to make a bypass permissions rule for the application.

Start first by creating an API Bypass (Performs any API Operation), if that is not sufficient in resolving the interoperability then move to full Bypass (Performs any Operation).

PRO TIP: The #1 reason for application interoperability is when permissions rules are not created for other security solutions that may reside on the system. Be sure to put in your AV exclusions if you have not already!

If the application has remained unstable proceed to opening a support case.

Conclusion

Summary

When troubleshooting in your environment, keep it simple.

Policy prevention identified on the alerts page remember the concept of least privilege and only approve for the permission level required. When and where possible use IT Tool and Certificate approvals over Hash and Allow rules.

Sensor interoperability will show when there is application slowness, delayed start of applications, or application crashes. Resolve interoperability with the use of Bypass rules.

Additional Resources

For more information on allowlisting and permissions explore the following resources: