Splunk Halt at Processing While Upload Data

      By: David Allen  |  Senior Splunk Consultant

Similar many things in life, having the right tools to fix a problem is what separates the novice from the expert. The novice has virtually no tools in their toolbox, and the tools that they have are large and clunky. Whereas the expert has many tools which accept been refined over many years of feel fixing many problems. Without the right tools for the job, many tasks simply could not exist accomplished — or at least would get much more difficult and time-consuming. Understanding what tools are available to set up and troubleshoot common Splunk Enterprise problems and how to use those tools — at a macro level — is the purpose of this blog.

The offset tool demand y'all will equally you brainstorm your Splunk troubleshooting journey is some basic knowledge on how to troubleshoot and how to narrow downwardly all the possibilities, like peeling off layers of an onion until somewhen yous find the root cause of the problem. Some problems we come across over and over and have learned to do a couple of simple checks to solve the problem. How many times have you run a search and said to yourself, "Where are all my fields?" We quickly learned to check the search modem and make sure if you want fields to be in Verbose style or to a bottom extent Smart manner.

Search Job Inspector

The first tool yous will need for troubleshooting basic searching problems is the Splunk Search Job Inspector. If you suspect that your search is not working properly, then using the Search Task Inspector may shed some light on the consequence. The Search Job Inspector is a tool that lets yous take a closer look at what your search is doing and run into where the Splunk software is spending well-nigh of its time.

Yous tin can access the Job Inspector by clicking the dropdown to the left of the search mode.

From the dropdown, select "Audit Job." Y'all will see another screen containing Execution Costs details and Search Task Properties details. Besides note the corporeality of time it took to consummate the search, which may exist a symptom of a problem you are not fifty-fifty aware of yet!

Execution Costs

The Execution Costs section lists information about the components of the search and how much impact each component has on the overall performance of the search.

• – The component durations in seconds.
• – How many times each component was invoked while the search ran.
• – The input and output consequence counts for each component.

With the data in the Execution Costs section, yous can troubleshoot the efficiency of your search. You can narrow down which processing components are impacting the search performance.

Search Chore Properties

The Search Task Properties section contains a listing of many fields with additional search parameters. The most important fields for troubleshooting are the post-obit:

• eventCount: The number of events returned past the search.
  eventFieldCount: The number of fields establish in the search results.
  eventIsTruncated: Indicates that events take not been stored and not available for search.
  isDone: Indicates if the search has completed.
  isFailed: Indicates if in that location was a fatal mistake executing the search.
  isFinalized: Indicates if the search was finalized (stopped earlier completion).
  runDuration: Time in seconds that the search took to complete.

You as well accept admission to the search.log link on the far right which opens the search.log where you can search for errors and warnings which may give a clue as to the search event y'all are experiencing.

But for more complex SPL (Search Programming Linguistic communication) problems, ask yourself some basic questions:

• – When did this commencement happening?
• – What is the exact thing I am searching for and what is the time range?
• – Accept I added a lookup file or event type incorrectly?

And then remove lines of SPL and add them back one line at a fourth dimension and discover the line where the problem shows up. Then unravel the problematic line piece by slice until you find the problem.

The above example works fine for basic search issues, just for Enterprise problems, you lot are going to demand some more powerful tools in your toolbox.

BTOOL

Splunk uses configuration files for almost all the settings within it. These .conf files are scattered all over the directory structure in many similarly-named .conf files. Splunk combines these similarly named files lexicographically and by a defined precedence club. To complicate things farther, Splunk needs to exist reset to reload the latest changes to the .conf files. If you think that Splunk is not using the configurations that it should, then BTOOL is your tool of choice.

BTOOL is a Splunk CLI command which shows what the actual settings are on the disk, or most likely SSD these days – not what is in retention and maybe not what Splunk is currently running, so beware of this subtle fact. To exist sure you are seeing the config settings Splunk is really running, you will need to restart Splunk. To access this command be sure to be at the Splunk home directory: $SPLUNK_HOME/bin.

BTOOL comes with Splunk Enterprise software, so no additional purchase, download, or installation is needed.

Beneath is the typical BTOOL command syntax (Note: All CLI commands in this document are typical for *nada OS):

./splunk btool list [options]

Here are some examples of the most common BTOOL commands:

To brandish all the merged settings of the various inputs.conf files, run this command:

./splunk btool inputs listing –-debug

Or you may want to see all input configurations independent in the search app:

./splunk btool –app=search inputs listing

Or you may want to see all props configurations set in the search app, and in what context they are prepare:

./splunk btool props listing –app=search --debug

Lastly, you may want to notice an input stanza for which you know name:

./splunk btool inputs list | grep splunktcp

Splunk Log Files

Under the hood, Splunk is running a lot of processes, from ingesting data to searching information and a lot more. All of these processes, and many of the steps in between, generate information that the Splunk software records into log files. Analyzing these log files tin give clues to assist solve your Splunk problem. The most mutual logs used for troubleshooting in Splunk are the internal logs located in: $SPLUNK_HOME/var/log. This path is monitored by default, and the contents are sent to various indexes based upon the type of log file. The nearly common internal indexes are _introspection, _internal, and _audit.

The _introspection index collects data about the touch on of the Splunk software on the host organisation.
It specifically collects the OS resource usage for Splunk processes, which tin can be cleaved down by Splunk process and viewed by host-level, dynamic CPU utilization and paging information. This alphabetize also contains disk input-output usage statistics. This can be very useful in diagnosing Splunk performance issues.

For example, use this search to find the median CPU usage of the main Splunkd process for i host over the last hour:

index=_introspection component=PerProcess host= data.procedure=splunkd
data.args="-p * starting time" OR data.args="service") earliest=-1h
| timechart median(data.pct_cpu) equally cpu_usage(%)


_internal: This index includes Splunk Enterprise internal logs. This index can be used to check the period of information through the various pipeline processes, data nearly license usage, the search scheduler, various Splunk crash logs, diverse search information, and more.

For instance, to search for the size of the search artifacts, use this search:

alphabetize=_internal sourcetype=splunkd_access method=Become jobs
|stats sum(bytes) by uri

The _audit indexes contain information about user activities such as failed or successful user log ins, modified settings, updated lookup files. Running searches, adequacy checks, and configuration changes generate audit events.

For example, to audit user access use this search:

index="_audit" activity=log* action="login attempt"

One of the most common log files used for troubleshooting is the splunkd.log, which uses source type Splunkd and is indexed to the _internal index. The Splunkd source type is further broken downwardly by component, so you tin can further refine your search by its subcategory. Using the source type Splunkd, you can use a search like this to cheque for whatsoever data quality warnings or errors:

alphabetize=_internal sourcetype=splunkd (component=AggregatorMiningProcessor OR
component=LineBreakingProcessor OR component=DateParserVerbose)
(log_level=WARN OR log_level=ERROR)

Or to cheque for potential index problems, yous tin can use a search like this one:

index=_internal sourcetype=splunkd host=idx*.yourcompany.splunkcloud.com
component=metrics grouping=per_*_regex_cpu

Splunk generates many internal log files, and searching those internal logs is a expert mode to find or isolate many mutual Splunk Enterprise problems.

Health Report Manager – New in Splunk Version eight.0.0

The Health Study Manager is a high-level overview of your Enterprise and lets you view the status of various Splunk Enterprise components. Individual components study their health status every 30 seconds and results are displayed through a tree structure that provides a continuous, real-time view of the health of your deployment.

The Health Report Managing director tin be accessed hither:

Once selected, the dropdown appears equally shown below and displays the health of the diverse components. Check this if y'all suspect there may be an issue with your Enterprise or if the indicator is not light-green.

How the Splunk Health Report Works

The health report records the wellness condition of Splunk features in a tree structure, where leaf nodes stand for particular features, and intermediary nodes categorize the various features. Feature wellness status is color-coded in four states as follows:

• – Dark-green: The feature is functioning properly.
• – Yellow: The feature is experiencing a problem.
• – Ruby-red: The feature has a severe issue and is negatively impacting the functionality of your deployment.
• – Grey: Health study is disabled for the feature.

Let's run through a scenario of how we could find the problem if the Searches Skipped indicator is red.

1. First, select the Searches Skipped feature to view diagnostic information almost the current health status of the feature.
2. Review the data under Root Cause. In this example, the percentage of high priority searches skipped is 44% over the terminal 24 hours, which exceeds the ruby threshold of x% and causes the feature'southward health status to change to red.
3. Review the Last l Related Letters. These log entries include warning messages showing that some scheduled searches cannot be executed. For example: 09-15-2020 16:11:00.324 +0000 WARN SavedSplunker - cannot execute scheduled searches that live at the system level (need an app context).

One explanation for this type of alert bulletin is the possibility that the number of loftier-priority searches running exceeds the maximum concurrent search limit, which can cause searches to be skipped.

After you review the root crusade and log file information, which suggest that maximum search concurrency limits acquired the Searches Skipped feature's condition change, you tin apply the Cloud Monitoring Console to check search scheduler action and confirm if the suspected cause is right.

DIAG

A diag file provides a snapshot of the configurations and logs from the Splunk software along with select data about the platform example. The diag drove process gathers information such as server specifications, operating organization (OS) version, file system information, internal logs, configuration files, and current network connections. No customer data is included in the diag file.

In your troubleshooting quest using the CLI from the $SPLUNK_HOME/bin folder, run the following control for each instance that y'all are troubleshooting:

./splunk diag

If you do contact Splunk regarding an issue, they will frequently asking a diag file for their assay. You lot tin can even index the diag output file and "Splunk it" and create your own troubleshooting dashboards!

To generate and upload a diag, the CLI syntax is:

./splunk diag --upload

This control interactively prompts for values such as a Splunk username and password, choice of open up cases for that user, and a clarification of the upload.

Y'all can also remove sure components form the diag written report with a command as follows:

./splunk diag -collect=

Likewise, components tin be removed with this control:

./splunk diag -disable=

Using Telnet to Exam Open up Ports

When information technology comes to checking if a network port is opened or closed on a remote figurer, in that location'due south no easier fashion than to apply Telnet. Ports that are left open for no reason are a security risk that can exist exploited past malicious programs and viruses. At the same time, if a legitimate software communicates through a certain port, having that port closed will make the program throw errors and malfunction. Telnet allows the user to test private ports and see whether they are open or not.

On a *naught OS box, you can run the Telnet command through the concluding as shown beneath:

telnet [domainname or ip] [port], e.g.>telnet 192.168.1.ane 443

When a computer port is open a blank screen volition show upward, pregnant that the connection has been successful. An unsuccessful connection will be accompanied past an error message.

Many times when you endeavour to use Telnet, you may find that your own network is blocking your connexion. Information technology's quite common for users to be running a firewall, which blocks connection to outbound ports. A basic way to exam whether your firewall is interrupting your Telnet is to disable your firewall and run a Telnet test.

Network Toolkit

This is an app on Splunkbase. The Network Toolkit app provides a series of tools for troubleshooting networks. It includes tools for evaluating net connexion bandwidth, performing pings, traceroutes, DNS lookups, whois tape checks, and waking sleeping computers (via wake-on-lan).

Additional Online Resource for Splunk Troubleshooting

The Splunk How-To YouTube Channel
Splunk Online Documentation
Online Troubleshooting Transmission
Splunk Answers
Submitting a case to Splunk

Want to learn more about data model accelerations? Contact u.s. today!

foxworthdindoutiors.blogspot.com

Source: https://www.tekstream.com/blog-troubleshooting-in-splunk/

Share this post