Practice Free SPLK-2002 Splunk Enterprise Certified Architect Exam Questions Answers With Explanation

As a best practice, the internal licensing logs should be stored on the license server. The license server is a Splunk instance that manages the distribution and enforcement of licenses in a Splunk deployment. The license server generates internal licensing logs that contain information about the license usage, violations, warnings, and pools. The internal licensing logs should be stored on the license server itself, because they are relevant to the license server’s role and function. Storing the internal licensing logs on the license server also simplifies the license monitoring and troubleshooting process. The internal licensing logs should not be stored on the indexing layer, the deployment layer, or the search head layer, because they are not related to the roles and functions of these layers. Storing the internal licensing logs on these layers would also increase the network traffic and disk space consumption

The TailingProcessor channel in the splunkd.log file would help troubleshoot this issue, because it contains information about the files that Splunk monitors and indexes, such as the file path, size, modification time, and CRC checksum. It also logs any errors or warnings that occur during the file monitoring process, such as permission issues, file rotation, or file truncation. The TailingProcessor channel can help identify if Splunk is reading the new data from the monitor input file or not, and what might be causing the problem. Option B is the correct answer. Option A is incorrect because the ModularInputs channel logs information about the modular inputs that Splunk uses to collect data from external sources, such as scripts, APIs, or custom applications. It does not log information about the monitor input file. Option C is incorrect because the ChunkedLBProcessor channel logs information about the load balancing process that Splunk uses to distribute data among multiple indexers. It does not log information about the monitor input file. Option D is incorrect because the ArchiveProcessor channel logs information about the archive process that Splunk uses to move data from the hot/warm buckets to the cold/frozen buckets. It does not log information about the monitor input file12

1: https://docs.splunk.com/Documentation/Splunk/9.1.2/Troubleshooting/WhatSplunklogsaboutitself#splunkd.log 2: https://docs.splunk.com/Documentation/Splunk/9.1.2/Troubleshooting/Didyouloseyourfishbucket#Check_the_splunkd.log_file

According to the Splunk Enterprise Search Head Clustering (SHC) Deployer documentation, the “local_only” push mode is the correct option when deploying built-in apps. This mode ensures that the deployer only pushes configurations from the local directory of built-in Splunk apps (such as search, learned, or launcher) without overwriting or merging their default app configurations.

In an SHC environment, the deployer is responsible for distributing configuration bundles to all search head members. Each push can be executed in different modes depending on how the admin wants to handle the app directories:

full: Overwrites both default and local folders of all apps in the bundle.

merge_to_default: Merges configurations into the default folder (used primarily for custom apps).

local_only: Pushes only local configurations, preserving default settings of built-in apps (the safest method for core Splunk apps).

default only: Pushes only default folder configurations (rarely used and not ideal for built-in app updates).

Using the “local_only” mode ensures that default Splunk system apps are not modified, preventing corruption or overwriting of base configurations that are critical for Splunk operation. It is explicitly recommended for pushing Splunk-provided (built-in) apps like search, launcher, and user-prefs from the deployer to all SHC members.

References (Splunk Enterprise Documentation):

• Managing Configuration Bundles with the Deployer (Search Head Clustering)

• Deployer Push Modes and Their Use Cases

• Splunk Enterprise Admin Manual – SHC Deployment Management

• Best Practices for Maintaining Built-in Splunk Apps in SHC Environments

According to the Splunk Search Job Inspector documentation, the search.log file for a completed search job can be accessed through Splunk Web by navigating to the job’s detailed inspection view.

To access it:

Open the completed search in Splunk Web.

Click the Job menu (top right of the search interface).

Select Inspect Job.

In the Job Inspector window, click the search.log link.

This log provides in-depth diagnostic details about how the search was parsed, distributed, and executed across the search head and indexers. It contains valuable performance metrics, command execution order, event sampling information, and any error or warning messages encountered during search processing.

The search.log file is generated for every search job—scheduled, ad-hoc, or background—and is stored in the job’s dispatch directory ($SPLUNK_HOME/var/run/splunk/dispatch//).

Other listed options are incorrect:

Option A queries the _internal index, which does not store per-search logs.

Option B is used to view search configurations, not logs.

Option C is not a valid Splunk Web navigation option.

Thus, the only correct and Splunk-documented method is via Job ? Inspect Job ? search.log.

References (Splunk Enterprise Documentation):

• Search Job Inspector Overview and Usage

• Analyzing Search Performance Using search.log

• Search Job Management and Dispatch Directory Structure

• Splunk Enterprise Admin Manual – Troubleshooting Searches

 Configurations from the deployer are merged into the SPLUNK_HOME/etc/apps/APP_HOME/local directory on the search head cluster member. The deployer distributes apps and other configurations to the search head cluster members in the form of a configuration bundle. The configuration bundle contains the contents of the SPLUNK_HOME/etc/shcluster/apps directory on the deployer. When a search head cluster member receives the configuration bundle, it merges the contents of the bundle into its own SPLUNK_HOME/etc/apps directory. The configurations in the local directory take precedence over the configurations in the default directory. The SPLUNK_HOME/etc/system/local directory is used for system-level configurations, not app-level configurations. The SPLUNK_HOME/etc/apps/search/default directory is used for the default configurations of the search app, not the configurations from the deployer.

The deployer distributes apps and non-search related and manual configuration file changes to the search head cluster members. The deployer does not bootstrap a clean Splunk install for a search head cluster, as this is done by the captain. The deployer also does not distribute runtime knowledge object changes made by users across the search head cluster, as this is done by the replication factor. For more information, see Use the deployer to distribute apps and configuration updates in the Splunk documentation.

In search head clustering, there are two methods to transfer captaincy to a different member. One method is to use the Search Head Clustering settings menu from Splunk Web on any member. This method allows the user to select a specific member to become the new captain, or to let Splunk choose the best candidate. The other method is to run the splunk transfer shcluster-captain command from the member that the user wants to become the new captain. This method requires the user to know the name of the target member and to have access to the CLI of that member. Using the Monitoring Console is not a method to transfer captaincy, because the Monitoring Console does not have the option to change the captain. Running the splunk transfer shcluster-captain command from the current captain is not a method to transfer captaincy, because this command will fail with an error message

According to the Splunk Enterprise User Authentication and Authorization Guide, effective user management during deployment planning involves identifying how users will authenticate (native, LDAP, or SAML) and defining what roles and capabilities they will need to perform their tasks.

However, counting or analyzing the number of users who appear in Splunk log events (Option C) is not part of user management planning. This metric relates to audit and monitoring, not access provisioning or role assignment.

A proper user management plan should address:

Authentication method selection (native, LDAP, or SAML).

User mapping and provisioning workflows from existing identity stores.

Role-based access control (RBAC) — assigning users appropriate permissions via Splunk roles and capabilities.

Administrative governance — ensuring access policies align with compliance requirements.

Determining the number of users visible in log events provides no operational value when planning Splunk authentication or authorization architecture. Therefore, this task can be safely disregarded during initial planning.

References (Splunk Enterprise Documentation):

• User Authentication and Authorization in Splunk Enterprise

• Configuring LDAP and SAML Authentication

• Managing Users, Roles, and Capabilities

• Splunk Deployment Planning Manual – Security and Access Control Planning

The splunk rebuild command is used for thawing the archive bucket. Thawing is the process of restoring frozen data back to Splunk for searching. Frozen data is data that has been archived or deleted from Splunk after reaching the end of its retention period. To thaw a bucket, the user needs to copy the bucket from the archive location to the thaweddb directory under SPLUNK_HOME/var/lib/splunk and run the splunk rebuild command to rebuild the .tsidx files for the bucket. The splunk collect command is used for collecting diagnostic data from a Splunk instance. The splunk convert command is used for converting configuration files from one format to another. The splunk dbinspect command is used for inspecting the status and properties of the buckets in an index.

According to the Splunk documentation1, multi-site clustering is an indexer cluster that spans multiple physical sites, such as data centers. Each site has its own set of peer nodes and search heads. Each site also obeys site-specific replication and search factor rules. The use cases that are made possible by multi-site clustering are:

Greatly reduce WAN traffic by preferentially searching assigned site (search affinity). This means that if you configure each site so that it has both a search head and a full set of searchable data, the search head on each site will limit its searches to local peer nodes. This eliminates any need, under normal conditions, for search heads to access data on other sites, greatly reducing network traffic between sites2.

Seamlessly route searches to a redundant site in case of a site failure. This means that by storing copies of your data at multiple locations, you maintain access to the data if a disaster strikes at one location. Multisite clusters provide site failover capability. If a site goes down, indexing and searching can continue on the remaining sites, without interruption or loss of data2.

The other options are false because:

Use blockchain technology to audit search activity from geographically dispersed data centers. This is not a use case of multi-site clustering, as Splunk does not use blockchain technology to audit search activity. Splunk uses its own internal logs and metrics to monitor and audit search activity3.

Enable a forwarder to send data to multiple indexers. This is not a use case of multi-site clustering, as forwarders can send data to multiple indexers regardless of whether they are in a single-site or multi-site cluster. This is a basic feature of forwarders that allows load balancing and high availability of data ingestion4.

According to the Splunk documentation1, when migrating an indexer cluster from single-site to multi-site, you must remove the existing single-site attributes from the server.conf file of each peer node. These attributes include replication_factor, search_factor, and cluster_label. You must also restart each peer node after removing the attributes. The other options are false because:

Multi-site policies will apply only to the data created after migration, unless you configure the manager node to convert legacy buckets to multi-site1.

All peer nodes do not need to run the same version of Splunk, as long as they are compatible with the manager node2.

Single-site buckets can be converted to multi-site buckets by changing the constrain_singlesite_buckets setting in the manager node’s server.conf file to "false"1.

 The index-time props.conf attributes that impact indexing performance are LINE_BREAKER and SHOULD_LINEMERGE. These attributes determine how Splunk breaks the incoming data into events and whether it merges multiple events into one. These operations can affect the indexing speed and the disk space consumption. The REPORT attribute does not impact indexing performance, as it is used to apply transforms at search time. The ANNOTATE_PUNCT attribute does not impact indexing performance, as it is used to add punctuation metadata to events at search time. For more information, see [About props.conf and transforms.conf] in the Splunk documentation.

 In a four site indexer cluster, the configuration that stores two searchable copies at the origin site, one searchable copy at site2, and a total of four searchable copies is site_search_factor = origin:2, site2:1, total:4. This configuration tells the cluster to maintain two copies of searchable data at the site where the data originates, one copy of searchable data at site2, and a total of four copies of searchable data across all sites. The site_search_factor determines how many copies of searchable data are maintained by the cluster for each site. The site_replication_factor determines how many copies of raw data are maintained by the cluster for each site. For more information, see Configure multisite indexer clusters with server.conf in the Splunk documentation.

A search head cluster is a group of Splunk Enterprise search heads that share configurations, job scheduling, and search artifacts1. The deployer is a Splunk Enterprise instance that distributes apps and other configurations to the cluster members1. The local folder of each Splunk app contains the custom configurations that override the default settings2. The default folder of each Splunk app contains the default configurations that are provided by the app2.

By default, when the deployer pushes an app to the search head cluster, it merges the local folder of the app into the default folder and deploys the merged folder to the search heads3. This means that the custom configurations in the local folder will take precedence over the default settings in the default folder. However, this also means that the local folder of the app on the search heads will be empty, unless the app is modified through the search head UI3.

Option B is the correct answer because it reflects the default behavior of the deployer when pushing apps to the search head cluster. Option A is incorrect because the local folder is not copied to the local folder on the search heads, but merged into the default folder. Option C is incorrect because all the .conf files in the local folder are deployed to the search heads, not only certain ones. Option D is incorrect because the local folder is not ignored, but merged into the default folder.

To activate replication for an index in an indexer cluster, the repFactor attribute must be configured in indexes.conf on all peer nodes. This attribute specifies the replication factor for the index, which determines how many copies of raw data are maintained by the cluster. Setting the repFactor attribute to auto will enable replication for the index. The replicate attribute in indexes.conf is not a valid Splunk attribute. The repFactor attribute in outputs.conf and the replicate attribute in deploymentclient.conf are not related to replication for an index in an indexer cluster. For more information, see Configure indexes for indexer clusters in the Splunk documentation.

Setting site=site0 on all Search Head Cluster members disables search site affinity. Search site affinity is a feature that allows search heads to preferentially search the peer nodes that are in the same site as the search head, to reduce network latency and bandwidth consumption. By setting site=site0, which is a special value that indicates no site, the search heads will search all peer nodes regardless of their site. Setting site=site0 does not set all members to dynamic captaincy, enable multisite search artifact replication, or enable automatic search site affinity discovery. Dynamic captaincy is a feature that allows any member to become the captain, and it is enabled by default. Multisite search artifact replication is a feature that allows search artifacts to be replicated across sites, and it is enabled by setting site_replication_factor to a value greater than 1. Automatic search site affinity discovery is a feature that allows search heads to automatically determine their site based on the network latency to the peer nodes, and it is enabled by setting site=auto

According to the Splunk Search Head Clustering Troubleshooting Guide, when a Search Head Cluster (SHC) member is reverted from a backup or experiences configuration drift (e.g., an outdated Raft state), it can fail to rejoin the cluster due to inconsistent Raft metadata. The Raft database stores the SHC’s internal consensus and replication state, including knowledge object synchronization, captain election history, and peer membership information.

If this Raft metadata becomes corrupted or outdated (as in the scenario where a node is restored from backup), the recommended and Splunk-supported remediation is to clean the Raft metadata using:

splunk clean raft

This command resets the node’s local Raft state so it can re-synchronize with the current SHC captain and rejoin the cluster cleanly.

The steps generally are:

Stop the affected SHC member.

Run splunk clean raft on that node.

Restart Splunk.

Verify that it successfully rejoins the SHC.

Deleting configuration stanzas or forcing re-addition (Options B and C) can lead to further inconsistency or data loss. Reviewing logs (Option A) helps diagnose issues but does not resolve Raft corruption.

References (Splunk Enterprise Documentation):

• Troubleshooting Raft Metadata Corruption in Search Head Clusters

• splunk clean raft Command Reference

• Search Head Clustering: Recovering from Backup and Membership Failures

• Splunk Enterprise Admin Manual – Raft Consensus and SHC Maintenance

The following statements describe a search head cluster captain:

Is the job scheduler for the entire search head cluster. The captain is responsible for scheduling and dispatching the searches that run on the search head cluster, as well as coordinating the search results from the search peers. The captain also ensures that the scheduled searches are balanced across the search head cluster members and that the search concurrency limits are enforced.

Replicates the search head cluster’s knowledge bundle to the search peers. The captain is responsible for creating and distributing the knowledge bundle to the search peers, which contains the knowledge objects that are required for the searches. The captain also ensures that the knowledge bundle is consistent and up-to-date across the search head cluster and the search peers. The following statements do not describe a search head cluster captain:

Manages alert action suppressions (throttling). Alert action suppressions are the settings that prevent an alert from triggering too frequently or too many times. These settings are managed by the search head that runs the alert, not by the captain. The captain does not have any special role in managing alert action suppressions.

Synchronizes the member list with the KV store primary. The member list is the list of search head cluster members that are active and available. The KV store primary is the search head cluster member that is responsible for replicating the KV store data to the other members. These roles are not related to the captain, and the captain does not synchronize them. The member list and the KV store primary are determined by the RAFT consensus algorithm, which is independent of the captain election. For more information, see [About the captain and the captain election] and [About KV store and search head clusters] in the Splunk documentation.

Search is not locked out when a customer has installed a 500GB Enterprise license and a 300GB, no enforcement license on the same license master. The no enforcement license allows the customer to exceed the license quota without locking search, but violations are still recorded. The customer can ingest up to 800GB of data per day without violating the license, but if they ingest more than that, they will incur a violation. However, the violation will not lock search, as the no enforcement license overrides the enforcement policy of the Enterprise license. For more information, see [No enforcement licenses] and [License violations] in the Splunk documentation.

When adding or decommissioning a member from a Search Head Cluster (SHC), the proper order of operations is:

Delete Splunk Enterprise, if it exists.

Install and initialize the instance.

Join the SHC.

This order of operations ensures that the member has a clean and consistent Splunk installation before joining the SHC. Deleting Splunk Enterprise removes any existing configurations and data from the instance. Installing and initializing the instance sets up the Splunk software and the required roles and settings for the SHC. Joining the SHC adds the instance to the cluster and synchronizes the configurations and apps with the other members. The other order of operations are not correct, because they either skip a step or perform the steps in the wrong order.

 Search head clustering is a Splunk feature that allows a group of search heads to share configurations, apps, and knowledge objects, and to provide high availability and scalability for searching. Search head clustering has the following characteristics:

A deployer is required. A deployer is a Splunk instance that distributes the configurations and apps to the members of the search head cluster. The deployer is not a member of the cluster, but a separate instance that communicates with the cluster master.

At least three search heads are needed. A search head cluster must have at least three search heads to form a quorum and to ensure high availability. If the cluster has less than three search heads, it cannot function properly and will enter a degraded mode.

The deployer must have sufficient CPU and network resources to process service requests and push configurations. The deployer is responsible for handling the requests from the cluster master and the cluster members, and for pushing the configurations and apps to the cluster members. Therefore, the deployer must have enough CPU and network resources to perform these tasks efficiently and reliably.

Search heads do not need to meet the high-performance reference server requirements, as this is not a mandatory condition for search head clustering. The high-performance reference server requirements are only recommended for optimal performance and scalability of Splunk deployments, but they are not enforced by Splunk.

A single-site indexer cluster is a group of Splunk Enterprise instances that index and replicate data across the cluster1. A bucket is a directory that contains indexed data, along with metadata and other information2. A replication factor is the number of copies of each bucket that the cluster maintains1. A search factor is the number of searchable copies of each bucket that the cluster maintains1. A searchable copy is a copy that contains both the raw data and the index files3. A search head is a Splunk Enterprise instance that coordinates the search activities across the peer nodes1.

Option D is the correct answer because it reflects the definitions of replication factor and search factor. The cluster will ensure that there are at least three copies of each bucket, one on each peer node, to satisfy the replication factor of 3. The cluster will also ensure that there are at least two searchable copies of each bucket, one primary and one searchable, to satisfy the search factor of 2. The primary copy is the one that the search head uses to run searches, and the searchable copy is the one that can be promoted to primary if the original primary copy becomes unavailable3.

Option A is incorrect because it confuses the replication factor and the search factor. The cluster will ensure there are at least three copies of each bucket, not two, to meet the replication factor of 3. The cluster will ensure there are at least two copies of searchable metadata, not three, to meet the search factor of 2.

Option B is incorrect because it uses the wrong terms. The cluster will ensure there are at least, not at most, three copies of each bucket, to meet the replication factor of 3. The cluster will ensure there are at least, not at most, two copies of searchable metadata, to meet the search factor of 2.

Option C is incorrect because it has nothing to do with the replication factor or the search factor. The cluster does not limit the number of search heads that can access the bucket at the same time. The search head can search across multiple clusters, and the cluster can serve multiple search heads1.

1: The basics of indexer cluster architecture - Splunk Documentation 2: About buckets - Splunk Documentation 3: Search factor - Splunk Documentation

The best practice for ingesting syslog data from network devices on port 514 into Splunk is to configure syslog to write logs and use a Splunk forwarder to collect the logs. This practice will ensure that the data is reliably collected and forwarded to Splunk, without losing any data or overloading the Splunk indexer. Configuring syslog to send the data to multiple Splunk indexers will not guarantee data reliability, as syslog is a UDP protocol that does not provide acknowledgment or delivery confirmation. Using a Splunk indexer to collect a network input on port 514 directly will not provide data reliability or load balancing, as the indexer may not be able to handle the incoming data volume or distribute it to other indexers. Using a Splunk forwarder to collect the input on port 514 and forward the data will not provide data reliability, as the forwarder may not be able to receive the data from syslog or buffer it in case of network issues. For more information, see [Get data from TCP and UDP ports] and [Best practices for syslog data] in the Splunk documentation.

The correct topology to ensure a scalable and performant deployment for the customer’s use case is two search head clusters, one for ITSI and one for ES. This configuration provides high availability, load balancing, and isolation for each Splunk app. According to the Splunk documentation1, ITSI and ES should not be installed on the same search head or search head cluster, as they have different requirements and may interfere with each other. Having two separate search head clusters allows each app to have its own dedicated resources and configuration, and avoids potential conflicts and performance issues1. The other options are not recommended, as they either have only one search head or search head cluster, which reduces the availability and scalability of the deployment, or they have both ITSI and ES installed on the same search head or search head cluster, which violates the best practices and may cause problems. Therefore, option B is the correct answer, and options A, C, and D are incorrect.

1: Splunk IT Service Intelligence and Splunk Enterprise Security compatibility

Splunk configuration files are files that contain settings that control various aspects of Splunk behavior, such as data inputs, outputs, indexing, searching, clustering, and so on1. Troubleshooting Splunk configuration files involves identifying and resolving issues that affect the functionality or performance of Splunk due to incorrect or conflicting configuration settings. Some of the tools and methods that can help with troubleshooting Splunk configuration files are:

search.log: This is a file that contains detailed information about the execution of a search, such as the search pipeline, the search commands, the search results, the search errors, and the search performance2. This file can help troubleshoot issues related to search configuration, such as props.conf, transforms.conf, macros.conf, and so on3.

btool output: This is a command-line tool that displays the effective configuration settings for a given Splunk component, such as inputs, outputs, indexes, props, and so on4. This tool can help troubleshoot issues related to configuration precedence, inheritance, and merging, as well as identify the source of a configuration setting5.

diagnostic logs: These are files that contain information about the Splunk system, such as the Splunk version, the operating system, the hardware, the license, the indexes, the apps, the users, the roles, the permissions, the configuration files, the log files, and the metrics6. These files can help troubleshoot issues related to Splunk installation, deployment, performance, and health7.

Option A is the correct answer because crash logs are the least helpful in troubleshooting Splunk configuration files. Crash logs are files that contain information about the Splunk process when it crashes, such as the stack trace, the memory dump, and the environment variables8. These files can help troubleshoot issues related to Splunk stability, reliability, and security, but not necessarily related to Splunk configuration9.

When designing the number and size of indexes, the following considerations should be applied:

Expected daily ingest volumes: This is the amount of data that will be ingested and indexed by the Splunk platform per day. This affects the storage capacity, the indexing performance, and the license usage of the Splunk deployment. The number and size of indexes should be planned according to the expected daily ingest volumes, as well as the peak ingest volumes, to ensure that the Splunk deployment can handle the data load and meet the business requirements12.

Data retention time policies: This is the duration for which the data will be stored and searchable by the Splunk platform. This affects the storage capacity, the data availability, and the data compliance of the Splunk deployment. The number and size of indexes should be planned according to the data retention time policies, as well as the data lifecycle, to ensure that the Splunk deployment can retain the data for the desired period and meet the legal or regulatory obligations13.

Access controls: This is the mechanism for granting or restricting access to the data by the Splunk users or roles. This affects the data security, the data privacy, and the data governance of the Splunk deployment. The number and size of indexes should be planned according to the access controls, as well as the data sensitivity, to ensure that the Splunk deployment can protect the data from unauthorized or inappropriate access and meet the ethical or organizational standards14.

Option D is the correct answer because it reflects the most relevant and important considerations for designing the number and size of indexes. Option A is incorrect because the number of concurrent users is not a direct factor for designing the number and size of indexes, but rather a factor for designing the search head capacity and the search head clustering configuration5. Option B is incorrect because the number of installed apps is not a direct factor for designing the number and size of indexes, but rather a factor for designing the app compatibility and the app performance. Option C is incorrect because it omits the expected daily ingest volumes, which is a crucial factor for designing the number and size of indexes.

The Splunk Enterprise Capacity Planning Manual states that running Splunk in a virtualized environment typically results in a performance reduction of approximately 20% to 45% compared to equivalent deployments on physical hardware.

This degradation is primarily due to the virtualization overhead inherent in hypervisor environments (such as VMware, Hyper-V, or KVM), which can affect:

Disk I/O throughput and latency — the most critical factor for indexers.

CPU scheduling efficiency, particularly for multi-threaded indexing processes.

Network latency between clustered components.

Splunk’s documentation strongly emphasizes that while virtualized environments offer operational flexibility, they cannot match bare-metal performance, especially under heavy indexing loads.

To mitigate performance loss, Splunk recommends:

Reserving dedicated CPU and I/O resources for Splunk VMs.

Avoiding over-commitment of hardware resources.

Using high-performance SSD storage or paravirtualized disk controllers.

These optimizations can narrow the performance gap, but a 20–45% reduction remains a realistic expectation under typical conditions.

References (Splunk Enterprise Documentation):

• Splunk Enterprise Capacity Planning Manual – Virtualization Performance Considerations

• Splunk on Virtual Infrastructure – Best Practices and Performance Tuning

• Indexer and Search Head Hardware Recommendations

• Performance Testing Guidelines for Splunk Deployments

According to the Indexer Clustering Administration Guide, configuration bundles pushed from the Cluster Manager (Master Node) overwrite the contents of the $SPLUNK_HOME/etc/slave-apps/ directory on each search peer (indexer). However, Splunk provides a special persistent location — the _cluster app’s local directory — for files that must survive bundle redeployments.

Specifically, any configuration files placed in:

$SPLUNK_HOME/etc/slave-apps/_cluster/local/

will persist after future bundle pushes because this directory is excluded from the automatic overwrite process.

This is particularly useful for maintaining local overrides or custom configurations that should not be replaced by the Cluster Manager, such as environment-specific inputs, temporary test settings, or monitoring configurations unique to that peer.

Other directories under slave-apps are overwritten each time a configuration bundle is pushed, ensuring consistency across the cluster. Likewise, master-apps exists only on the Cluster Manager and is used for deployment, not persistence.

Thus, the _cluster/local folder is the only safe, Splunk-documented location for configurations that need to survive bundle redeployment.

References (Splunk Enterprise Documentation):

• Indexer Clustering: How Configuration Bundles Work

• Maintaining Local Configurations on Clustered Indexers

• slave-apps and _cluster App Structure and Behavior

• Splunk Enterprise Admin Manual – Cluster Configuration Management Best Practices

 The following statements are true about Splunk indexer clustering:

All peer nodes must run exactly the same Splunk version. This is a requirement for indexer clustering, as different Splunk versions may have different data formats or features that are incompatible with each other. All peer nodes must run the same Splunk version as the master node and the search heads that connect to the cluster.

The search head must run the same or a later Splunk version than the peer nodes. This is a recommendation for indexer clustering, as a newer Splunk version may have new features or bug fixes that improve the search functionality or performance. The search head should not run an older Splunk version than the peer nodes, as this may cause search errors or failures. The following statements are false about Splunk indexer clustering:

The master node must run the same or a later Splunk version than the search heads. This is not a requirement or a recommendation for indexer clustering, as the master node does not participate in the search process. The master node should run the same Splunk version as the peer nodes, as this ensures the cluster compatibility and functionality.

The peer nodes must run the same or a later Splunk version than the master node. This is not a requirement or a recommendation for indexer clustering, as the peer nodes do not coordinate the cluster activities. The peer nodes should run the same Splunk version as the master node, as this ensures the cluster compatibility and functionality. For more information, see [About indexer clusters and index replication] and [Upgrade an indexer cluster] in the Splunk documentation.

The license_usage.log file is generated and maintained on the License Manager node in a Splunk deployment. This log provides detailed statistics about daily license consumption, including data volume indexed per pool, index, sourcetype, source, and host.

In a distributed or clustered environment (with both search head and indexer clusters), the License Manager acts as the central authority that collects license usage information from all indexers and consolidates it into this log. The License Manager receives periodic reports from each license peer (indexer) and records them in:

$SPLUNK_HOME/var/log/splunk/license_usage.log

The log is automatically indexed into the _internal index with sourcetype=splunkd and can be queried using searches such as:

index=_internal source=*license_usage.log* type="RolloverSummary"

Other components like the Cluster Manager, SHC Deployer, or individual indexers do not store the full consolidated license usage data — they only send summarized reports to the License Manager.

Therefore, the License Manager is the definitive and Splunk-documented location for retrieving and analyzing license_usage.log data across a distributed deployment.

References (Splunk Enterprise Documentation):

• Managing Licenses in a Distributed Environment

• license_usage.log Reference and Structure

• Monitoring License Consumption Using the License Manager

• Splunk Enterprise Admin Manual – License Reporting and Troubleshooting

To expand the search head cluster by adding a new member, node2, the first step is to initialize the cluster configuration on node2 using the splunk init shcluster-config command. This command sets the required parameters for the cluster member, such as the management URI, the replication port, and the shared secret key. The management URI must be unique for each cluster member and must match the URI that the deployer uses to communicate with the member. The replication port must be the same for all cluster members and must be different from the management port. The secret key must be the same for all cluster members and must be encrypted using the splunk _encrypt command. The master_uri parameter is optional and specifies the URI of the cluster captain. If not specified, the cluster member will use the captain election process to determine the captain. Option C shows the correct syntax and parameters for the splunk init shcluster-config command. Option A is incorrect because the splunk bootstrap shcluster-config command is used to bring up the first cluster member as the initial captain, not to add a new member. Option B is incorrect because the master_uri parameter is not required and the mgmt_uri parameter is missing. Option D is incorrect because the splunk add shcluster-member command is used to add an existing search head to the cluster, not to initialize a new member12

1: https://docs.splunk.com/Documentation/Splunk/9.1.2/DistSearch/SHCdeploymentoverview#Initialize_cluster_members 2: https://docs.splunk.com/Documentation/Splunk/9.1.2/DistSearch/SHCconfigurationdetails#Configure_the_cluster_members

The index=_internal component=DC* host= search will show all deployment client messages from the universal forwarder. The component field indicates the type of Splunk component that generated the message, and the host field indicates the host name of the machine that sent the message. The index=_audit component=DC* host= search will not return any results, because the deployment client messages are not stored in the _audit index. The index=_internal component=DS* host= search will show the deployment server messages from the deployment server, not the client. The index=_audit component=DS* host= search will also not return any results, for the same reason as above

 A Technical Add-On (TA) is a Splunk app that contains configurations for data collection, parsing, and enrichment. It can also enable event data for a data model, which is useful for creating dashboards and reports. Therefore, before installing a TA, it is important to identify the number of scheduled or real-time searches that will use the data model, and to validate if the TA enables event data for a data model. The number of forwarders that the TA can support is not relevant, as the TA is installed on the indexer or search head, not on the forwarder. The installation location of the TA depends on the type of data and the use case, so it is not a fixed requirement

According to the Splunk documentation1, the “Unknown sid” error message means that the search job associated with the link has expired or been deleted. The sid (search ID) is a unique identifier for each search job, and it is used to retrieve the results of the search. If the sid is not found, the search cannot be displayed. The other options are false because:

The users having insufficient permissions would result in a different error message, such as “You do not have permission to view this page” or "You do not have permission to run this search"1.

An add-on needing to be updated would not affect the validity of the sid, unless the add-on changes the search syntax or the data source in a way that makes the search invalid or inaccessible1.

One or more indexers being down would not cause the “Unknown sid” error, as the sid is stored on the search head, not the indexers. However, it could cause other errors, such as “Unable to distribute to peer” or "Search peer has the following message: not enough disk space"1.

A deployment plan should include business continuity and disaster recovery plans, current logging details and data source inventory, and current and future topology diagrams of the IT environment. These elements are essential for planning, designing, and implementing a Splunk deployment that meets the business and technical requirements. A comprehensive list of stakeholders, either direct or indirect, is not part of the deployment plan, but rather part of the project charter. For more information, see Deployment planning in the Splunk documentation.

The correct answer is A. Change frozenTimePeriodInSecs to a larger value. This is a possible solution to reduce the need to thaw buckets, as it increases the time period before a bucket is frozen and removed from the index1. The frozenTimePeriodInSecs attribute specifies the maximum age, in seconds, of the data that the index can contain1. By setting it to a larger value, the Splunk administrator can keep the data in the index for a longer time, and avoid having to thaw the buckets frequently. The other options are not effective solutions to reduce the need to thaw buckets. Option B, changing maxTotalDataSizeMB to a smaller value, would actually increase the need to thaw buckets, as it decreases the maximum size, in megabytes, of an index2. This means that the index would reach its size limit faster, and more buckets would be frozen and removed. Option C, changing maxHotSpanSecs to a larger value, would not affect the need to thaw buckets, as it only changes the maximum lifetime, in seconds, of a hot bucket3. This means that the hot bucket would stay hot for a longer time, but it would not prevent the bucket from being frozen eventually. Option D, changing coldToFrozenDir to a different location, would not reduce the need to thaw buckets, as it only changes the destination directory for the frozen buckets4. This means that the buckets would still be frozen and removed from the index, but they would be stored in a different location. Therefore, option A is the correct answer, and options B, C, and D are incorrect.

1: Set a retirement and archiving policy 2: Configure index size 3: Bucket rotation and retention 4: Archive indexed data

This is where in the Job Inspector details can be found to help determine where performance is affected, as it shows the time and resources spent by each component of the search, such as commands, subsearches, lookups, and post-processing1. The Execution Costs > Components section can help identify the most expensive or inefficient parts of the search, and suggest ways to optimize or improve the search performance1. The other options are not as useful as the Execution Costs > Components section for finding performance issues. Option A, Search Job Properties > runDuration, shows the total time, in seconds, that the search took to run2. This can indicate the overall performance of the search, but it does not provide any details on the specific components or factors that affected the performance. Option B, Search Job Properties > runtime, shows the time, in seconds, that the search took to run on the search head2. This can indicate the performance of the search head, but it does not account for the time spent on the indexers or the network. Option C, Job Details Dashboard > Total Events Matched, shows the number of events that matched the search criteria3. This can indicate the size and scope of the search, but it does not provide any information on the performance or efficiency of the search. Therefore, option D is the correct answer, and options A, B, and C are incorrect.

1: Execution Costs > Components 2: Search Job Properties 3: Job Details Dashboard

Splunk Enterprise allows administrators to dynamically increase logging verbosity for a specific search by adding a | noop log_debug=* command immediately after the base search. This method provides temporary, search-specific debug logging without requiring global configuration changes or restarts.

The noop (no operation) command passes all results through unchanged but can trigger internal logging actions. When paired with the log_debug=* argument, it instructs Splunk to record detailed debug-level log messages for that specific search execution in search.log and the relevant internal logs.

This approach is officially documented for troubleshooting complex search issues such as:

Unexpected search behavior or slow performance.

Field extraction or command evaluation errors.

Debugging custom search commands or macros.

Using this method is safer and more efficient than modifying server-wide logging configurations (server.conf or limits.conf), which can affect all users and increase log noise. The “Server logging” page in Splunk Web (Option D) adjusts global logging levels, not per-search debugging.

References (Splunk Enterprise Documentation):

• Search Debugging Techniques and the noop Command

• Understanding search.log and Per-Search Logging Control

• Splunk Search Job Inspector and Debugging Workflow

• Troubleshooting SPL Performance and Field Extraction Issues

The master node distributes configuration bundles to peer nodes in the slave-apps directory under $SPLUNK_HOME/etc. The configuration bundle method is the only supported method for managing common configurations and app deployment across the set of peers. It ensures that all peers use the same versions of these files1. Bundles typically contain a subset of files (configuration files and assets) from $SPLUNK_HOME/etc/system, $SPLUNK_HOME/etc/apps, and $SPLUNK_HOME/etc/users2. The process of distributing knowledge bundles means that peers by default receive nearly the entire contents of the search head’s apps3.

The splunk validate cluster-bundle --check-restart command is the officially documented Splunk Enterprise method to determine if a configuration bundle push will trigger a rolling restart within an indexer cluster.

When configuration changes are made on the Cluster Manager (Master Node)—for example, updates to indexes.conf, props.conf, or transforms.conf—Splunk administrators must validate the bundle before pushing it to all peer nodes. Using this command allows the Cluster Manager to simulate the deployment and verify whether the configuration modifications necessitate a restart across peer indexers to take effect.

The --check-restart flag specifically reports whether:

The configuration changes are minor (no restart required).

The changes affect components that require a full or rolling restart (e.g., changes to indexing paths, volume definitions, or replication factors).

Running this validation prior to an actual splunk apply cluster-bundle command prevents service disruption during production operations.

Other commands such as splunk show cluster-bundle-status display deployment status but not restart requirements, and splunk apply cluster-bundle executes the actual deployment, not validation.

References (Splunk Enterprise Documentation):

• Indexer Clustering: Deploy Configuration Bundles with Validation

• splunk validate cluster-bundle Command Reference

• Managing Indexer Clusters – Rolling Restarts and Bundle Deployment Best Practices

• Splunk Enterprise Admin Manual – Cluster Manager Maintenance Commands

 In a distributed environment, knowledge object bundles are replicated from the search head to the SPLUNK_HOME/var/run/searchpeers directory on the search peer(s). A knowledge object bundle is a compressed file that contains the knowledge objects, such as fields, lookups, macros, and tags, that are required for a search. A search peer is a Splunk instance that provides data to a search head in a distributed search. A search head is a Splunk instance that coordinates and executes a search across multiple search peers. When a search head initiates a search, it creates a knowledge object bundle and replicates it to the search peers that are involved in the search. The search peers store the knowledge object bundle in the SPLUNK_HOME/var/run/searchpeers directory, which is a temporary directory that is cleared when the Splunk service restarts. The search peers use the knowledge object bundle to apply the knowledge objects to the data and return the results to the search head. The SPLUNK_HOME/var/lib/searchpeers, SPLUNK_HOME/var/log/searchpeers, and SPLUNK_HOME/var/spool/searchpeers directories are not the locations where the knowledge object bundles are replicated, because they do not exist in the Splunk file system

Comprehensive and Detailed Explanation (From Splunk Enterprise Documentation)Splunk’s Search Head Clustering documentation explains that the cluster uses a majority-based election system. A captain is elected only when a node sees more than half of the cluster. In a two-site design where site1 has the majority of members, Splunk states that the majority site continues normal operation during a network partition. The minority site (site2) is not allowed to elect a captain and should not promote itself.

Splunk specifically warns administrators not to enable static captain on a minority site during a network split. Doing so creates two independent clusters, leading to configuration divergence and severe data-consistency issues. The documentation emphasizes that static captain should only be used for a complete loss of majority, not for a site partition.

Because Site1 maintains majority, it remains the active cluster and site2 does not perform any actions. Splunk states that minority-site members should simply wait until network communication is restored.

Thus the correct answer is B: No action is required.

According to Splunk’s Search Scheduler and Job Management documentation, the scheduler.log file, located within the _internal index, records the execution of scheduled and saved searches. This log provides a detailed record of when each search is triggered, how long it runs, and its success or failure status.

Each time a scheduled search runs (for example, alerts, reports, or summary index searches), an entry is written to scheduler.log with fields such as:

sid (search job ID)

app (application context)

savedsearch_name (name of the saved search)

user (owner)

status (success, skipped, or failed)

run_time and result_count

By searching the _internal index for sourcetype=scheduler (or directly viewing scheduler.log), administrators can confirm whether a specific saved search executed as expected and diagnose skipped or delayed runs due to resource contention or concurrency limits.

Other internal logs serve different purposes:

metrics.log records performance metrics.

kvstore.log tracks KV Store operations.

btool.log does not exist — btool outputs configuration data to the console, not a log file.

Hence, scheduler.log is the definitive and Splunk-documented source for validating scheduled search activity.

References (Splunk Enterprise Documentation):

• Saved Searches and Alerts – Scheduler Operation Details

• scheduler.log Reference – Monitoring Scheduled Search Execution

• Monitoring Console: Search Scheduler Health Dashboard

• Troubleshooting Skipped or Delayed Scheduled Searches

When using the props.conf LINE_BREAKER attribute to delimit multi-line events, the SHOULD_LINEMERGE attribute should be set to false. This tells Splunk not to merge events that have been broken by the LINE_BREAKER. Setting the SHOULD_LINEMERGE attribute to true, auto, or none will cause Splunk to ignore the LINE_BREAKER and merge events based on other criteria. For more information, see Configure event line breaking in the Splunk documentation.

According to the Splunk documentation1, one of the benefits of search head clustering is the automatic replication of user knowledge objects, such as dashboards, reports, alerts, and tags. This ensures that all cluster members have the same set of knowledge objects and can serve the same search results to the users. The other options are false because:

Allowing indexers to maintain multiple searchable copies of all data is a benefit of indexer clustering, not search head clustering2.

Input settings are not synchronized between search heads, as search head clusters do not collect data from inputs. Data collection is done by forwarders or independent search heads3.

Fewer network ports are not required to be opened between search heads, as search head clusters use several ports for communication and replication among the members4.

The Splunk deployer sends apps to the search head cluster members by default to the path etc/shcluster//default. The deployer is a Splunk component that distributes apps and configurations to members of a search head cluster.

Splunk's documentation recommends placing the configuration bundle in the $SPLUNK_HOME/etc/shcluster/apps directory on the deployer, which then gets distributed to the search head cluster members. However, it should be noted that within each app's directory, configurations can be under default or local subdirectories, with local taking precedence over default for configurations. The reference to etc/shcluster//default is not a standard directory structure and might be a misunderstanding. The correct path where the deployer pushes configuration bundles is $SPLUNK_HOME/etc/shcluster/apps

According to Splunk Enterprise Search Performance and Deployment Optimization guidelines, the phone home interval (configured for deployment clients communicating with a Deployment Server) has no impact on search performance.

The phone home mechanism controls how often deployment clients check in with the Deployment Server for configuration updates or new app bundles. This process occurs independently of the search subsystem and does not consume indexer or search head resources that affect query speed, indexing throughput, or search concurrency.

In contrast:

Increasing the number of indexers (Option B) improves search performance by distributing indexing and search workloads across more nodes.

Workload Management (Option C) allows admins to prioritize compute and memory resources for critical searches, optimizing performance under load.

Increasing search heads (Option D) can enhance concurrency and user responsiveness by distributing search scheduling and ad-hoc query workloads.

Therefore, adjusting the phone home interval is strictly an administrative operation and has no measurable effect on Splunk search or indexing performance.

References (Splunk Enterprise Documentation):

• Deployment Server: Managing Phone Home Intervals

• Search Performance Optimization and Resource Management

• Distributed Search Architecture and Scaling Best Practices

• Workload Management Overview – Resource Allocation in Search Operations

A monitored log file is changing on the forwarder, but Splunk searches are not finding any new data that has been added. This could be caused by two possible reasons: B. An admin has removed the Splunk fishbucket on the forwarder. C. The last 256 bytes of the monitored file are not changing. Option B is correct because the Splunk fishbucket is a directory that stores information about the files that have been monitored by Splunk, such as the file name, size, modification time, and CRC checksum. If an admin removes the fishbucket, Splunk will lose track of the files that have been previously indexed and will not index any new data from those files. Option C is correct because Splunk uses the CRC checksum of the last 256 bytes of a monitored file to determine if the file has changed since the last time it was read. If the last 256 bytes of the file are not changing, Splunk will assume that the file is unchanged and will not index any new data from it. Option A is incorrect because running the splunk clean eventdata -index  command on the indexer will delete all the data from the specified index, but it will not affect the forwarder’s ability to send new data to the indexer. Option D is incorrect because Splunk does not use the first 256 bytes of a monitored file to determine if the file has changed12

1: https://docs.splunk.com/Documentation/Splunk/9.1.2/Data/Monitorfilesanddirectories 2: https://docs.splunk.com/Documentation/Splunk/9.1.2/Troubleshooting/Didyouloseyourfishbucket

A multisite indexer cluster is a type of indexer cluster that spans multiple geographic locations or sites. A multisite indexer cluster requires only one cluster manager, also known as the master node, for the entire cluster. The cluster manager is responsible for coordinating the replication and search activities among the peer nodes across all sites. The cluster manager can reside in any site, but it must be accessible by all peer nodes and search heads in the cluster. Option C is the correct answer. Option A is incorrect because having two cluster managers for the entire cluster would introduce redundancy and complexity. Option B is incorrect because having one cluster manager for each site would create separate clusters, not a multisite cluster. Option D is incorrect because having two cluster managers for each site would be unnecessary and inefficient12

1: https://docs.splunk.com/Documentation/Splunk/9.1.2/Indexer/Multisiteoverview 2: https://docs.splunk.com/Documentation/Splunk/9.1.2/Indexer/Clustermanageroverview

The following may explain the problem of why a colleague cannot see the src_ip field in their search results: The field was extracted as a private knowledge object, and the colleague did not explicitly use the field in the search and the search was set to Fast Mode. A knowledge object is a Splunk entity that applies some knowledge or intelligence to the data, such as a field extraction, a lookup, or a macro. A knowledge object can have different permissions, such as private, app, or global. A private knowledge object is only visible to the user who created it, and it cannot be shared with other users. A field extraction is a type of knowledge object that extracts fields from the raw data at index time or search time. If a field extraction is created as a private knowledge object, then only the user who created it can see the extracted field in their search results. A search mode is a setting that determines how Splunk processes and displays the search results, such as Fast, Smart, or Verbose. Fast mode is the fastest and most efficient search mode, but it also limits the number of fields and events that are displayed. Fast mode only shows the default fields, such as _time, host, source, sourcetype, and _raw, and any fields that are explicitly used in the search. If a field is not used in the search and it is not a default field, then it will not be shown in Fast mode. The events are tagged as communicate, but are missing the network tag, and the Typing Queue, which does regular expression replacements, is blocked, are not valid explanations for the problem. Tags are labels that can be applied to fields or field values to make them easier to search. Tags do not affect the visibility of fields, unless they are used as filters in the search. The Typing Queue is a component of the Splunk data pipeline that performs regular expression replacements on the data, such as replacing IP addresses with host names. The Typing Queue does not affect the field extraction process, unless it is configured to do so

 The Splunk Universal Forwarder is the only Splunk Enterprise offering that has its own license. The Splunk Universal Forwarder license allows the forwarder to send data to any Splunk Enterprise or Splunk Cloud instance without consuming any license quota. The Splunk Heavy Forwarder does not have its own license, but rather consumes the license quota of the Splunk Enterprise or Splunk Cloud instance that it sends data to. The Splunk Cloud Forwarder and the Splunk Forwarder Management are not separate Splunk Enterprise offerings, but rather features of the Splunk Cloud service. For more information, see [About forwarder licensing] in the Splunk documentation.

Two indexers clustered is the recommended deployment for a customer who plans to ingest 600 GB of data per day into Splunk, has six concurrent users, and wants high data availability and high search performance. This deployment will provide enough indexing capacity and search concurrency for the customer’s needs, while also ensuring data replication and searchability across the cluster. The customer can also save on the hardware cost by using only two indexers. Two indexers not in a cluster will not provide high data availability, as there is no data replication or failover. Three indexers not in a cluster will provide more indexing capacity and search concurrency, but also more hardware cost and no data availability. The customer’s data retention period, number of long searches, or volume of saved/scheduled searches are not relevant for determining the number of indexers. For more information, see [Reference hardware] and [About indexer clusters and index replication] in the Splunk documentation.

The splunk offline --enforce-counts command will permanently decommission a peer node operating in an indexer cluster. This command will remove the peer node from the cluster and delete its data. This command should be used when the peer node is no longer needed or is being replaced by another node. The splunk stop -f command will stop the Splunk service on the peer node, but it will not decommission it from the cluster. The splunk offline -f command will take the peer node offline, but it will not delete its data or enforce the replication and search factors. The splunk decommission --enforce-counts command is not a valid Splunk command. For more information, see Remove a peer node from an indexer cluster in the Splunk documentation.

Comprehensive and Detailed Explanation (From Splunk Enterprise Documentation)Splunk Enterprise documentation clarifies that none of the listed configurations are prohibited in production. Splunk allows the Cluster Manager to be colocated with the Monitoring Console in small deployments because both are management-plane functions and do not handle ingestion or search traffic. The documentation also states that the Search Head Cluster Deployer is not a runtime component and has minimal performance requirements, so it may be colocated with the Monitoring Console or Licensing Master when hardware resources permit.

Splunk also supports virtual machines for both search heads and indexers, provided they are deployed with dedicated CPU, storage throughput, and predictable performance. Splunk’s official hardware guidance specifies that while bare metal often yields higher performance, virtualized deployments are fully supported in production as long as sizing principles are met.

Because Splunk explicitly supports all four configurations under proper sizing and best-practice guidelines, there is no correct selection for “unsupported.” The question is outdated relative to current Splunk Enterprise recommendations.

According to the Splunk documentation1, increasing the current deployment client phone home interval can minimize performance issues by reducing the frequency of communication between the clients and the deployment server. This can also reduce the network traffic and the load on the deployment server. The other options are false because:

Modifying deploymentclient.conf to change from a Pull to Push mechanism is not possible, as Splunk does not support a Push mechanism for deployment server2.

Reducing the number of apps in the Manager Node repository will not affect the performance of the deployment server, as the apps are only downloaded when there is a change in the configuration or a new app is added3.

Decreasing the current deployment client phone home interval will increase the performance issues, as it will increase the frequency of communication between the clients and the deployment server, resulting in more network traffic and load on the deployment server1.