Troubleshooting Configuration vs. Interop

Introduction

Review this resource to learn troubleshooting best practices for your enterprise. This resource is intended as guidance to understand order of operations, how best to resolve your issue without over exposing your environment.

This guidance is intended for Endpoint Standard customers.

Diagnostics

To get to the root cause of an action that is causing disruption we need to assess the symptoms and determine if it is one of the following: 

  • Configuration: Is there a behavior or reputation reason that Carbon Black policies are blocking my application from operating as expected?
  • Interoperability: Is there an application-level conflict / incompatibility causing application slowness?

Once the cause is identified we want to respond in a manner that will introduce the least amount of security risk while restoring system functionality.

Configuration

To determine if your issue is caused by configuration you will visit the Alerts section of your console. Sort your view under Policy Status select “Policy Applied”, and Sensor Action select “Deny” and “Terminate”.

Graphical user interface, application</p>
<p>Description automatically generated

This subset of alerts will now highlight all actions that have been taken on your environment on the behalf of the VMware Carbon Black Cloud Sensor. From here you need to identify if this block is due to Reputation or Behavior of the targeted application.

If your target is not in the alerts skip to Interoperability section.

Reputation-Based

Reputation is determined by analysis of pre-existing, newly introduced, or accessed on the network. There are 3 stages in which Carbon Black Cloud will assess and assign a reputation:

  1. Resident: Pre-existing file on a system that is detected by RepMgr.exe as a part of a background scan. There is no attempted run of the file.
  2. Dropped: Newly introduced files to a system.
  3. Ran: Attempt to access, modify or delete a pre-existing file on system.

Does the application in question have an untrusted reputation?

Reputation Definitions

Reputation

Description

Banned

Hashes manually added into Company Banned list.

Known Malware

Known bad by Carbon Black from the cloud or/and local scanner.

Suspect/Heuristic Malware

Suspect malware detected by Carbon Black, but not necessarily malicious.

Adware/PUP Malware

Adware and Potential Unwanted Programs detected by Carbon Black.

Not Listed / Adaptive White

Hash not previously seen and local scanner does not classify as banned.

Unknown

Sensor observes file drop, but does not yet have reputation from the cloud or local scanner.

 

Behavior-Based

Behavior-based alert(s) will apply to any Trusted application that is being blocked due to abnormal application behavior. Blocking behaviors include but are not limited to the following:

Behavior

Description

Runs or is running

Process operating on the endpoint.

Communicates over the network

 

Network activity caused by the process

Scrapes memory of another process

Targeted attempt to read memory of processes such as lsass.exe. Could indicate credential scraping.

Executes code from memory

 

Untargeted attempt to run code from dynamic memory.

Invokes an untrusted process

 

Untrusted application or process is accessed.

Invokes a command interpreter

 

Attempt to use a shell / command line tool.

Performs ransomware-like behavior

Access Carbon Black Cloud decoy files, attempt to write to the master boot record, attempt to access Volume Shadow Copy Service (VSS). Terminate is only option because denying ransomware does not prevent further encryption.

Executes a fileless script

Use trusted process for malicious use. Also called non-malware or “living off the land.”

Injects code or modifies memory of another process

 

 

Trusted application injects code, or any use of process hollowing.

Interoperability

If your target was not included on the alerts page there is a high likelihood that you have an application interoperability. Interoperability will surface in one of the following manners on the local systems:

  • Application slowness
  • Delayed start of an application
  • An application operation is not performing
  • Application crashes

Resolution

Once the cause is identified we want to respond in a manner that will introduce the least amount of security risk while restoring system functionality.

Overview of Carbon Black Cloud Policies

  1. Permissions allow applications at path or specific behaviors of applications to run.
  2. Reputation determinations result from signatures, reputation lookups, heuristics, and deterministic models.
  3. Behavioral-based controls enforced according to policy configurations for streaming analytics.
  4. Upload unknown binaries for additional analysis using cloud-based sandbox.

Graphical user interface, text, application</p>
<p>Description automatically generated

 

 

 

 

 

Resolving a Configuration Block

Before resolving the alert the file be sure to consider the following determinations if you are uncertain of the application in question:

  1. Context: Am I observing an Inconsistent vector, hash, certificate, behavior, directory, application type, content type, device type, username, user context this would suggest normal or abnormal activity.
  2. Frequency: How frequently does this application run? Concentrated or dispersed occurrences over time that are consistent or inconsistent with expected information technology enablement practices.
  3. Prevalence: Where else does this exist in my environment? One of only a few devices/types of devices involved that would suggest normal or abnormal prevalence of activity. Volume of new occurrences that develop in a given point of time.

If you have determined that the application in question has been raised due to reputation or behavior, we want to consider resolving with the concept of least privilege. The Carbon Black Cloud sensor resolves and categorizes based in order of priority review the table below.

Priority

Reputation

Description

1

Ignore

Highest priority. Files have full permissions to run without observance. Applies to Allow, Allow & Log, and Bypass rules.

2

Company White

Hashes manually added into Company Whitelist.

3

Company Black

Hashes manually added into Company Blacklist.

4

Trusted White

Known good by Carbon Black from the cloud or/and local scanner.

5

Known Malware

Known bad by Carbon Black from the cloud or/and local scanner.

6

Suspect/Heuristic Malware

Suspect malware detected by Carbon Black, but not necessarily malicious.

7

Adware/PUP Malware

Adware and Potential Unwanted Programs detected by Carbon Black.

8

Local White

Any of the following conditions: pre-existing file, whitelisted by IT Tools, whitelisted by Certificate.

9

Common White

Any of the following conditions: hash not on any known good or bad list AND file is signed, hash previously analyzed AND not on any known good or bad lists.

10

Not Listed / Adaptive White

Cloud: hash not previously seen and local scanner: no known bad

11

Unknown

Lowest priority. Sensor observes file drop, but does not yet have reputation from the cloud or local scanner.

So you have your application in question follow the below scenarios to determine the best course of action to resolve.

When to use IT Tools

Defined by path of dropper, useful for developers

The IT Tools whitelist functionality is designed to permit processes with an inconsistent hash, with or without subprocesses, and a NOT_LISTED/UNKNOWN reputation. The priority 8 designation prevents malware from making its way into being inadvertently approved through such mechanisms.

A picture containing shape</p>
<p>Description automatically generated

When to use Certificates

Trusted Vendors

The Certificate whitelist functionality is designed to permit processes with a consistent or inconsistent hash, but a consistent certificate, and a NOT_LISTED/UNKNOWN reputation. The priority 8 designation prevents malware from leveraging a trusted certificate, otherwise affording it the opportunity to be inadvertently approved through such mechanisms.

Diagram</p>
<p>Description automatically generated

When to use Hash

SHA256, best for static applications

The hash-based whitelist functionality is designed to permit applications with a consistent hash and a reputation determination other than known good. The priority 2 designation allows for decisive control over the reputation of an application.

When to use Allow

Resolving application behavior without removing prevention rules.

The permissions functionality allows for authorizing applications with a consistent path, that may otherwise have, or be subject to, a misrepresented reputation and has no other consistent properties as far as it pertains to hash and certificates.

Permitting an application provides priority 1 status which affords absolute release from reputation-based determinations. There two sub-configurations for this that involve allowing without logging and allowing with logging.

  • Allow: allows the specified behavior in the specified path; None of the specified behavior at path is logged and no data is sent to the CBC backend.
  • Allow & Log: allows the specified behavior in the specified path; All activity is logged and reported to the CBC backend.

 

Diagram</p>
<p>Description automatically generated

Resolving an Interoperability

When to use Bypass

App Interoperability. Co-installed security solution is a common case.

If you are experiencing application slowness or instability gather information on the following:

  1. Device Name
  2. Date
  3. Time
  4. Application Impacted
  5. Any relevant Log Files

Once the information is collected move your sensor into a bypass state.

Bypass; all policy enforcement on the device is disabled and the sensor will not send data to the cloud; or, sensors momentarily enter Bypass mode during a sensor update

Graphical user interface, application</p>
<p>Description automatically generated

If you application has moved into a stable state once the OS is in bypass then move to make a bypass permissions rule for the application.

Start first by creating an API Bypass (Performs any API Operation), if that is not sufficient in resolving the interoperability then move to full Bypass (Performs any Operation).

Graphical user interface, text, application</p>
<p>Description automatically generated

PRO TIP: The #1 reason for application interoperability is when permissions rules are not created for other security solutions that may reside on the system. Be sure to put in your AV exclusions if you have not already!

If the application has remained unstable proceed to opening a support case.

Diagram</p>
<p>Description automatically generated

Conclusion

Summary

When troubleshooting in your environment, keep it simple.

Policy prevention identified on the alerts page remember the concept of least privilege and only approve for the permission level required. When and where possible use IT Tool and Certificate approvals over Hash and Allow rules.

Sensor interoperability will show when there is application slowness, delayed start of applications, or application crashes. Resolve interoperability with the use of Bypass rules.

Additional Resources

For more information on whitelisting and permissions explore the following resources:

 

 

 

Filter Tags

VMware Carbon Black Cloud Endpoint Standard Intermediate Advanced