This GigaOm Research Reprint Expires: Jan 29, 2023

GigaOm Radar for Application Performance Monitoring (APM)v1.02

1. Summary

Application performance monitoring (APM) is an IT tool that seeks to understand how applications react when in use, and diagnose problems to improve performance and serve business needs. Historically, developers would write bits of code into their applications that would signal what was executing and use other data to determine application performance. Those days have long passed. Within the complicated world of modern distributed applications, microservices, cloud computing, and hybrid environments, heightened business and customer expectations are the norm.

The increase in operating functions and capability places an additional load on developers. They must handle not just the application code complexity but also the use of underlying infrastructure. Applications may run in on-premises infrastructure, public cloud environments, private clouds, SaaS applications, and hybrids of these models.

With the additional emphasis on DevOps, developers are now a part of the operations landscape and must have the tools to participate. APM tools now must handle a much more complex environment with both operations personnel and developers, and within low code/no-code environments, encompassing citizen programmers.

Standards within the APM world are minimal; however, OpenTelemetry, an open-source project, holds the promise of a common method of transferring data. OpenTelemetry is currently a Cloud Native Computing Foundation (CNCF) project with the goal of providing a set of vendor-agnostic libraries and APIs, such as traces, metrics, and logs, for collecting and sending data.

This report explores vendors in the application performance monitoring marketplace.

How to Read this Report

This GigaOm report is one of a series of documents that helps IT organizations assess competing solutions in the context of well-defined features and criteria. For a fuller understanding consider reviewing the following reports:

Key Criteria report: A detailed market sector analysis that assesses the impact that key product features and criteria have on top-line solution characteristics—such as scalability, performance, and TCO—that drive purchase decisions.

GigaOm Radar report: A forward-looking analysis that plots the relative value and progression of vendor solutions along multiple axes based on strategy and execution. The Radar report includes a breakdown of each vendor’s offering in the sector.

Solution Profile: An in-depth vendor analysis that builds on the framework developed in the Key Criteria and Radar reports to assess a company’s engagement within a technology sector. This analysis includes forward-looking guidance around both strategy and product.

2. Market Categories and Deployment Types

For a better understanding of the market and vendor positioning (Table 1), we assess the kind of organization an APM solution targets and how it is deployed. APM solutions must fit the size and capabilities of the organization. This report defines the following market segments:

  • Small-to-medium-sized business (SMB): This category includes solutions that address the needs of small to medium-sized businesses without the requirements of IT expertise within multiple domains. Only SaaS solutions are appropriate for this segment of users.
  • Mid-market or business unit: In this category, we assess solutions on their ability to meet the needs of organizations ranging from medium-sized businesses to business units within a larger corporation. Ease of use and deployment are more important than extensive management functionality, data mobility, and feature set.
  • Large enterprise: We assessed offerings on their ability to support large and business-critical projects. Optimal solutions in this category will have a strong focus on flexibility, performance, data services, and features to improve security and data protection. Scalability is another big differentiator, as is the ability to deploy the same service in different environments. Managed service providers (MSPs) may be a target for some vendors, but are not addressed in this report.

In addition, we recognize three deployment models for solutions in this report: software as a service (SaaS), on-premises, and hybrid or multi-cloud solutions.

  • SaaS: These solutions are available only in the cloud. Often designed, deployed, and managed by the service provider, they are available only from that specific provider. The big advantage of this type of solution is the integration with other services offered by the cloud service provider (functions, for example) and its simplicity.
  • On-premises: The solution can be deployed entirely on-premises to meet requirements for security and privacy not available within SaaS solutions.
  • Hybrid and multi-cloud: These solutions are meant to be installed both on-premises and in the cloud, allowing them to build hybrid or multi-cloud storage infrastructures. Integration with a single cloud provider could be limited compared to the other option and more complex to deploy and manage. On the other hand, it is a more flexible approach and the user usually has more control over the entire stack with respect to resource allocation and tuning. These solutions can be deployed in the form of virtual appliances, like a traditional Network-Attached Storage (NAS) filer but in the cloud, or a software component that can be installed on a Linux VM (i.e., a file system).

Table 1. Vendor Positioning

market Segment

Deployment Models

SMB Mid-Market or Business Unit Large Enterprise SaaS On-Premises Hybrid
BMC
Broadcom
Cisco AppDynamics
Datadog
Dynatrace
IBM Instana
ManageEngine
Micro Focus
Microsoft
Netreo/Stackify
New Relic
Splunk
3 Exceptional: Outstanding focus and execution
2 Capable: Good but with room for improvement
2 Limited: Lacking in execution and use cases
2 Not applicable or absent

3. Key Criteria Comparison

Building on the findings from the GigaOm report, “Key Criteria for Evaluating Application Performance Monitoring (APM),” Table 2 summarizes how each vendor included in this research performs in the areas that we consider differentiating and critical in this sector. The objective is to give the reader a snapshot of the technical capabilities of different solutions and define the perimeter of the market landscape.

Table 2. Key Criteria Comparison

Key Criteria

Ease of Use Business Transactions Real User and Synthetic Monitoring Application Component Monitoring Analytics and Reporting Security Integration
BMC 2 2 2 2 2 2 2
Broadcom 2 2 2 2 2 1 2
Cisco AppDynamics 2 2 2 2 2 2 2
Datadog 2 2 2 2 2 2 2
Dynatrace 2 2 2 2 2 2 2
IBM Instana 1 2 1 2 2 1 2
ManageEngine 1 1 2 2 1 2 1
Micro Focus 1 2 2 2 2 2 1
Microsoft 1 1 1 2 2 2 1
Netreo/Stackify 2 2 2 2 2 2 1
New Relic 2 2 2 2 2 2 1
Splunk 2 2 2 2 2 2 2
3 Exceptional: Outstanding focus and execution
2 Capable: Good but with room for improvement
2 Limited: Lacking in execution and use cases
2 Not applicable or absent

Table 3. Evaluation Metrics Comparison

Evaluation Metrics

Flexibility Ease of Implementation Usability and Manageability Scalability Total Cost of Implementation
BMC 3 2 2 3 2
Broadcom 3 3 3 3 2
Cisco AppDynamics 3 3 3 3 2
Datadog 3 3 3 3 2
Dynatrace 3 3 3 3 2
IBM Instana 3 3 3 3 3
ManageEngine 2 3 3 3 2
Micro Focus 3 2 2 3 2
Microsoft 2 2 3 3 2
Netreo/Stackify 3 2 3 3 2
New Relic 3 3 3 3 2
Splunk 3 3 3 3 1
3 Exceptional: Outstanding focus and execution
2 Capable: Good but with room for improvement
2 Limited: Lacking in execution and use cases
2 Not applicable or absent

By combining the information provided in the tables above, the reader can develop a clear understanding of the technical solutions available in the market.

4. GigaOm Radar

This report synthesizes the analysis of key criteria and their impact on evaluation metrics to inform the GigaOm Radar graphic in Figure 1. The resulting chart is a forward-looking perspective on all the vendors in this report, based on their products’ technical capabilities and feature sets.

The GigaOm Radar plots vendor solutions across a series of concentric rings, with those set closer to the center judged to be of higher overall value. The chart characterizes each vendor on two axes–Maturity versus Innovation and Feature Play versus Platform Play–while providing an arrow that projects each solution’s evolution over the coming 12 to 18 months.

Figure 1. GigaOm Radar for Application Performance Monitoring

As shown in the Radar chart in Figure 1 above, most APM vendors are positioned on the platform side and contain other tools in the operation management arena. Platforms may be particular in their boundaries (e.g., Microsoft .NET), while others provide more functionality within their ecosystems (e.g., Micro Focus, ManageEngine, BMC, and Broadcom).

Dynatrace still dominates market leadership; however, Splunk and Datadog are now presenting more robust platforms for digital experience monitoring. Splunk advanced by adding a software ecosystem, while Datadog continues to grow more powerful year after year. New Relic has fallen behind, not due to loss of functionality, but because of the improvements made by competitors.

Inside the GigaOm Radar

The GigaOm Radar weighs each vendor’s execution, roadmap, and ability to innovate to plot solutions along two axes, each set as opposing pairs. On the Y axis, Maturity recognizes solution stability, strength of ecosystem, and a conservative stance, while Innovation highlights technical innovation and a more aggressive approach. On the X axis, Feature Play connotes a narrow focus on niche or cutting-edge functionality, while Platform Play displays a broader platform focus and commitment to a comprehensive feature set.

The closer to center a solution sits, the better its execution and value, with top performers occupying the inner Leaders circle. The centermost circle is almost always empty, reserved for highly mature and consolidated markets that lack space for further innovation.

The GigaOm Radar offers a forward-looking assessment, plotting the current and projected position of each solution over a 12- to 18-month window. Arrows indicate travel based on strategy and pace of innovation, with vendors designated as Forward Movers, Fast Movers, or Outperformers based on their rate of progression.

Note that the Radar excludes vendor market share as a metric. The focus is on forward-looking analysis that emphasizes the value of innovation and differentiation over incumbent market position.

5. Vendor Insights

BMC APM

Flexibility is exceptional when using BMC products. The ecosystem can ingest data from other sources, which has little impact on APM tools. Companies using BMC as a primary IT operations vendor will have a seamless experience with the APM solution.

Implementation of APM is easier for those already using BMC products, and ease of use is similar to that of other BMC offerings. The BMC universe is scalable up to any size enterprise, but is not a fit for SMBs.

The deployment model is simple for SaaS uses, and the on-premises-only deployment would prove to be relatively easy for a BMC shop. The combination of cloud and on-premises deployment presents more complications, for which support from BMC is readily available.

BMC handles synthetic transactions within “walled gardens” (high security infrastructure without direct access to the internet). This is particularly important for companies that rely on Internet of Things (IoT) devices for business operations. Synthetic transactions to/from websites and other cloud infrastructure work as expected.

BMC has improved its ability to integrate with other non-BMC tools using Intelligent Integration. The acquisition of StreamWeaver allows the connection of real-time data to other IT tools not provided by BMC.

BMC does not contribute to or use OpenTelemetry.

BMC is increasing the use of AI/ML in all of its products. The APM product will have additional capabilities in AI/ML in future releases.

Strengths: BMC APM is an easy selection for an IT enterprise already using BMC products. It integrates seamlessly with all BMC products, creating a solution with a lower learning curve because of the strength of BMC as a solution provider.

Challenges: BMC would not be an easy adoption for a non-BMC centric IT team. The limitations in synthetic monitoring may restrict the use of BMC APM in environments where IoT devices are key to the business. BMC has yet to embrace OpenTelemetry.

Broadcom DX APM

Broadcom’s application performance monitoring tool, Broadcom DX APM, is a capable APM and part of the universe of Broadcom solutions. As a platform, Broadcom encompasses mainframes, storage, enterprise software, and more.

DX APM can be deployed on-premises or as a SaaS solution. DX APM may not be a good fit for smaller shops or those without the expertise to build or support a modern backend. The SaaS offering can work for smaller companies where all software is run in a single cloud environment.

Despite being a part of a much larger technology environment, DX is a flexible solution that can be implemented by most larger IT organizations. The on-premises solution may prove more difficult to deploy, and may stress an SMB shop.

The flexibility of DX APM stands out from other enterprise platforms, as does its ease of implementation. As expected from a platform application, it scales well.

The synthetic monitoring of IoT devices within high security zones requires the deployment of an on-premises monitoring station. Synthetic monitoring of websites and cloud applications is accomplished via Broadcom provided public monitoring stations.

Security integration supports standard SSO methods, and using an agent can remotely gather data or even run OS level scripts. However, once past the initial learning of the system, it does appear to be on par with other vendors in ease of use.

In terms of emerging technologies, DX APM provides some support for rule-based ML within DX APM for triage and pattern analysis. Add-ons such as Business Payload Analyzer and Broadcoms’s AiOps solution make use of AI/ML. DX APM supplies data to these products. Broadcom is not a direct contributor to OpenTelemetry, however, they do make use of components of OpenTelemetry, such as Open Tracing.

Strengths: DX APM is a flexible application performance tool. The strong platform connections to the rest of Broadcom’s tools make it a good fit for enterprises already invested in Broadcom. The solution is surprisingly flexible for a platform solution.

Challenges: It is easier to use Broadcom platform tools throughout the enterprise than to integrate tools from other vendors. This can be a challenge in large enterprises where organic growth has provided multiple vendor solutions. The AI/ML solution is not directly integrated with DX APM. The Broadcom AIOps solution must be added to gain the functionality within the Broadcom suite.

Cisco AppDynamics

Cisco AppDynamics has been a player in the application performance monitoring arena for many years and thus meets all of the table stakes for an APM solution. Flexibility, ease of implementation, usability, ease of management, and scalability have long been strengths for AppDynamics. The cost of implementing the solution is on par with other vendors in the application performance monitoring space, though it is difficult to compare implementation costs from vendor to vendor.

Cisco recently acquired replex GmbH to add to the AppDynamics platform. The impact of this announcement has yet to be evaluated, although replex does have expertise in cloud management.

AppDynamics is a SaaS and on-premises solution with Controllers on-site for aggregation of data when needed. There is full support for real user monitoring and synthetic monitoring of both websites and secure IoT devices. Dynamic tracing is supported in Business iQ.

AppDynamics displays supporting infrastructure, application performance monitoring, user experience, business transactions, and impacts in a single interface. The AppDynamics agent tags user interaction to allow tracing of progress as it passes through web servers, microservices, databases, and other infrastructure components. The agent then sends performance metrics for each level to the APM controller whether on-site or within a cloud environment.

AI/ML capabilities are available for the SaaS solution but not the on-premises deployment. The Cognition Engine provides baseline anomaly detection and root cause analysis. It is included in the solution.

AppDynamics supports an OpenTelemetry backend and can ingest OpenTelemetry data. Cisco is a contributor to OpenTelemetry. OpenTelemetry is not yet available for on-premises Controllers.

Strengths: AppDynamics APM displays infrastructure metrics, application performance, and business transactions and impacts in an easy-to-digest manner, while still allowing detail on all aspects of the application. Support for OpenTelemetry is a plus.

Challenges: Added functionality is available via other products such as Business iQ, which can impact costs and implementation complexity. The future of AppDynamics APM is cloudy.

Datadog

The Datadog APM solution provides the required metrics from the infrastructure that it discovers. Dependency mapping is automatic and connects front-end and back-end pieces to give an end-to-end view of the application.

The use of distributed tracing and Datadog’s continuous profiler are linked automatically when both are enabled. The user has the ability to look into the code level and to correlate data in the “Code Hotspots” dashboard. This transparency allows for comparing and viewing code profiles to see performance changes. Users can correlate tracing data with Infrastructure metrics, RUM sessions, logs, network data, code profiles, and synthetic tests on a single screen.

Datadog has a built-in security monitoring solution and a more robust Cloud Security Platform that includes Cloud SIEM, CWS, CSPM, and Applications. The tool supports real-time threat monitoring via threat feeds or known bad IPs. Datadog APM also supports authentication and identity access tools. Full CI/CD integration is a plus.

Datadog supports three types of integrations: agent-based integrations installed with the Datadog Agent, authentication (crawler) -based integrations using credentials to obtain metrics with the API, and library integrations using the Datadog API to enable monitoring of applications based on the native language. This last integration allows users to build custom checks or metrics and send them to Datadog.

Datadog Notebook notes, documentation, and reports can be added to any dashboard. Import and export of Notebooks is supported.

Strengths: Datadog APM is a robust APM offering with tracing and profiling capabilities that allow real-time comparison of profiles to understand performance changes. The inclusion of a security monitoring tool is a plus.

Challenges: Determining initial implementation costs is difficult.

Dynatrace

Dynatrace is a software platform built around application monitoring in any location, including public/private/hybrid-cloud to on-premises. Dynatrace has evolved from a pure APM vendor with Dynatrace AppMon to a digital transformation software vendor for cloud-native and hybrid environments with their OneAgent architecture.

Dynatrace can be deployed either as a SaaS solution or an on-premises deployment. The on-premises deployment is called Dynatrace Managed. Data may reside in the cloud or on-premises depending on security and design requirements.

Both deployment methods use Dynatrace OneAgent to instrument applications, Docker containers, microservices architectures, or cloud-based infrastructure. OneAgent can instrument message service endpoints.

A single OneAgent per host is required to collect monitoring data—hosts may be deployed within Docker containers, microservices architectures, or cloud-based infrastructure.

Dynatrace ActiveGate (a software product) can act as a proxy between Dynatrace OneAgent and the Dynatrace Cluster. ActiveGate enables agentless monitoring of various technologies and can be installed on Windows, Linux, Kubernetes, or OpenShift.

OneAgent technology uses a single agent to collect and unify all operational and business performance metrics for all types of entities in your application environment—servers, applications, services, databases, and more—across each layer of your technology stack (including containers).

Dynatrace Smartscape provides real-time visualization to map the environment and detects dependencies among websites, applications, services, processes, hosts, networks, and cloud infrastructure. Dynatrace Session Replay visually replays the digital experience of users across browsers and devices.

Dynatrace’s patented PurePath Technology captures timings and code-level context for application transactions end to end, across all supported technologies, from cloud to mainframe.

Another patented technology is the Davis AI engine. It analyzes relationships and dependencies within complex IT environments to improve the analysis of data, enabling users to obtain actionable information about the enterprise.

Dynatrace provides monitoring of applications to any level of detail required by the enterprise. Real User Monitoring provides analysis of user interaction with applications down to the source code level. RUM includes browser and mobile app monitoring plus monitoring of the underlying infrastructures.

Dynatrace Synthetic Monitoring facilitates the monitoring, availability, and performance of applications from a user perspective. Dynatrace Synthetic Monitoring supports private locations allowing walled gardens and IoT systems to be monitored.

Strengths: As an application performance monitoring platform, Dynatrace continues to lead the pack. Its flexible deployment options are suitable for enterprises of all sizes.

Challenges: Deployments in complex on-premises and multi-cloud environments can be difficult. Determining the cost of an implementation may be difficult, especially for more diverse configurations.

IBM Instana

IBM acquired Instana in late 2020. It is not yet integrated into the IBM sphere of products; however, it is a capable APM solution. For this report, Instana is considered an innovative product because of its newness and the lack of integration into the IBM ecosystem.

Instana Agents are deployed on a per-host basis. The agent keeps itself up to date, but upgrades can be controlled to fit the requirements of the IT organization.

The APM supports distributed tracing in a number of environments, including Java, .NET, Node.js, and others. In addition to tracing 100% of requests from an application, Instana creates dependency maps of the infrastructure, and works well with DevOps CI/CD pipelines, supporting Jenkins, Harness, and others.

Instana makes the point that Kubernetes monitoring should be included in its APM solution by supporting distributions from Rancher, OpenShift, and VMware Tanzu. They provide a full-stack context to monitor correlations, traces, calls, services infrastrastructure, and orchestration logs.

End-user monitoring supports sessions in browsers, mobile devices, and frameworks. Also included is monitoring of serverless environments and support for all major cloud vendors.

Alerting and dashboarding are part of the solution, with preconfigured, custom, and shareable data visualizations available. Custom metrics and integrations add to the flexibility of Instana, as do role-based access control and authorization/authentication services for single sign-on.

Synthetic monitoring is part of Instana; however, private locations are not available to monitor walled gardens or IoT devices.

The two emerging technologies, AI/ML and OpenTelemetry, are supported. An AIOps module is specifically called out, with OpenTelemetry supported data from logs, network devices, and other system components. The AI/ML is used for analytics with any data that Instana collects.

Strengths: Instana is a newer player in the APM world and its acquisition by IBM will allow IBM’s APM capabilities to be seen eventually as a part of a large platform for digital transformation. The cost of implementation is easy to determine, and is unique among vendors at this time.

Challenges: The merger into IBM presents questions for the future. No support exists for private location within synthetics.

ManageEngine APM

ManageEngine APM is a capable APM that meets the table stakes with two caveats: infrastructure discovery, monitoring, and reporting require a separate SKU; and application availability reporting lacks depth and flexibility.

The server and application management product includes Applications Manager for on-premises. Application Control Plus handles application control and privilege management, Site24x7 monitors websites and cloud servers and includes real user monitoring, Site24x7 StatusIQ provides status and incident communications, and Site24x7 APM Insight is used for application performance monitoring and troubleshooting.

ManageEngine supports a unique augmented analysis tool that builds distributed models of the infrastructure. The models may take days to weeks to allow their predictive analysis tools to produce meaningful results. However, the models can be used for analytics and smart suggestions based on context. Plus, the AI/ML system, Zia, learns continuously once a model has been built. The output can be sent to Analytics Plus for reports and dashboards.

ManageEngine supports ITSM and support systems such as ServiceNow, Jira Software, and Zendesk. The drag-and-drop UI improves the speed of visualizations.

Though ManageEngine does have larger enterprise clients, it may be better suited for SMBs and mid-market or business units. It has global clients including business units of international conglomerates.

Strengths: Strong support for SMBs, mid-markets, and business units. A unique approach to predictive analysis with Zia.

Challenges: ManageEngine has a number of products and services. Finding the correct blend for an IT enterprise may prove challenging. The Zia engine has great potential but has yet to prove itself in the marketplace.

Micro Focus Application Performance Management (APM) Software

Micro Focus APM is part of the Operations Bridge suite of products and services designed to accomplish full digital experience monitoring. These can include AppPulse Active (synthetic monitoring), Real User Monitoring, Business Process Monitoring, Diagnostic Software, and SiteScope. The APM suite is capable and meets all table stakes.

Real user monitoring and business process monitoring are the strengths of Micro Focus. The solution provides deep dive diagnostics. It also includes service level management and a Diagnostics module.

The solution does have key criteria for which challenges are present. Ease of use is not a strong point, nor is integration with other systems.

The solution can be deployed on-site, but may require professional services. This option is not recommended for SMBs. The on-premises solution fits large enterprises, mid-market companies, and business units with their own IT departments. Deployment of the SaaS solution, using agents in the cloud, may be a match for any enterprise.

Reporting and dashboards are capable; however, nothing stands out as significantly better than other visualization tools. When viewed with Operations Bridge, a holistic view of the enterprise is possible along with APM data.

Considering emerging technologies, Micro Focus has AI/ML within other products, such as OpsB Analytics, but the APM suite does not have AI functionality directly. Micro Focus does not mention, nor are they indicated as a contributor to, OpenTelemetry.

Strengths: Micro Focus has a compelling suite of IT operations products, including ITSM and AIOps offerings. This APM solution slides into place with its other offerings.

Challenges: Micro Focus APM plays best with other Micro Focus offerings. Determining implementation costs in complex environments can be difficult as multiple Micro Focus tools are necessary.

Microsoft Operations Manager

Microsoft Operations Manager is a component of Microsoft System Center. It allows the monitoring of services, devices, and operations from a single console. The Operations console enables checking the health, performance, and availability for all monitored objects in the environment and identifies and resolves problems.

Operations Manager consists of a management server, an operation database for real-time data, and a data warehouse for long-term storage. An agent runs as a service to collect data, compare those data to predefined values, create alerts, and run remediations.

Management packs define the information that the agent collects and returns it to the management server for a specific application or technology, such as BizTalk.

For enterprises based totally on Microsoft .NET, System Insights is a very capable solution. Comparing the Microsoft solution to other vendors within a pure .NET environment may perhaps be unfair to Microsoft. Its product is ideal for use cases found frequently in companies using only Microsoft products.

A succinct description of the APM product is difficult due to the manner in which it is delivered—monitoring becomes an essential part of the application and is tied to the operating system. Monitoring an application is essentially monitoring the system.

Transactions, traces, reporting, and monitoring are tied together in a single console, allowing an operator to drill down into the detail necessary to resolve the problem. Scripts can be created to auto-remediate defined solutions.

Microsoft is a contributor to OpenTelemetry.

Strengths: For a purely Microsoft IT environment, Microsoft APM tools are the obvious choice. The intimate knowledge Microsoft has of the operating system, its SQL database, and development environment is a strength.

Challenges: If an IT organization mixes in an application development environment incompatible with the Microsoft way of doing business, Microsoft would be challenged to support application performance monitoring.

Netreo/Retrace by Netreo (formerly Stackify)

Netreo has acquired Stackify, and all table stakes have been met. The acquisition of Stackify should improve the offering in the future, with the expectation they will reach cohesion in the next six to 12 months.

Netreo includes Retrace for APM and Netreo for infrastructure monitoring. The integration of these products is in progress. The current focus is on alert management from multiple application sources and infrastructure.

The key criteria challenge for Netreo is integration with other systems.

Anomaly detection may prove to be the most differentiating factor for this APM solution. It uses AI in a manner not truly defined publicly to predict when and where a problem might occur.

OpenTelemetry does not appear to be a direction for Netreo at this time, but as the acquisition matures, OpenTelemetry may become a goal.

At present, ease of implementation is low and it’s difficult because each portion of the solution must be installed and then integrated. This issue should resolve itself in future offerings.

Deployment is complicated by the lack of unity. The SaaS version of each product may be easy enough to implement, but integration is more difficult now than it will be in the near future. The on-premises and hybrid models for deployment are more complicated in the current circumstances.

The technology seems best suited for a large organization; however, until a union of the two products surfaces, it may be best to use Netreo in a smaller environment before looking at an enterprise solution.

Strengths: Anomaly detection is the secret sauce for Netreo. Integration with the CI/CD is a strength.

Challenges: The merger of Netreo and Stackify may yield an excellent final product; however, at this time, the lack of solidarity harms the overall proposition.

New Relic APM

New Relic has been a strong player in the APM world. Past iterations were known for their ease of use and the simplicity of implementation. New Relic has evolved into more of a software platform for digital transformation. As a SaaS-only solution, deployment is suitable for any size enterprise. The initial cost of implementation can be low (free); however, scaling to enterprise level can be costly.

New Relic meets all table stakes, and has capability in every key criteria except integration with other systems, which can often be tedious.

In terms of emerging technologies, New Relic has embraced AI, though it is unclear how seamlessly their AI works with APM. The AI tool is a different SKU. Another key criteria New Relic has led on in the past is synthetic monitoring (another SKU). Other vendors have caught up with their deployment of private locations into walled security zones and for IoT devices.

New Relic has embraced OpenTelemetry and supports it for moving data. They are also one of the top ten contributors to OpenTelemetry.

A full APM solution from New Relic requires a number of SKUs, including APM, Infrastructure Monitoring, Browser Monitoring, Mobile Monitoring, and Synthetics Monitoring. All of these tools are included in New Relic ONE, which has a single SKU.

Strengths: New Relic APM is an easy product to use and implement. APM is one part of the entire New Relic ONE digital transformation and observability platform. Anyone looking into the APM solution should also examine the entire package for suitability.

Challenges: A complete APM solution requires multiple SKUs, and integration with other systems may not be simple. Interfacing to ITSM and communications systems are challenges.

Splunk APM

For many years Splunk was known for traversing log files and then pulling alerts, metrics, and other data from the log files. Logs could be anywhere in the environment and Splunk handled them. Splunk has evolved into an APM and digital experience monitoring platform via both acquisition and in-house development. The APM solution meets all table stakes, and is capable in every key criteria.

Splunk APM solution is part of the Splunk Observability platform and features full trace analysis of all transactions, real user monitoring, and synthetic transactions for both website and IoT devices. Content Packs allow the installation of agents where necessary and the integration of other data sources. Additionally, service level monitoring and CI/CD pipelines monitoring are supported.

Dependency mapping and infrastructure discovery are also part of the solution. The ability to satisfy multiple governance directives such as GDPR and SOC2 is a plus, as are the control mechanisms to support PCI and HIPAA environments. The security model ensures only data needed to understand the performance of an application is visible.

Splunk APM is a good fit for any size organization. With deployment support for SaaS, SaaS plus on-premises, and on-premises only deployment, Splunk APM has a great deal of flexibility.

Splunk APM uses AI for analysis and OpenTelemetry for data integration and transport. Splunk is the number one contributor to OpenTelemetry.

As a solution for monitoring the digital experience, Splunk APM delivers a highly scalable environment that integrates well with other Splunk tools. Splunk APM pricing can be compared to other APM vendors with a host and usage models.

The full Splunk platform has introduced a workload pricing model. Workload pricing for the Splunk platform is aimed at flexibility and cost saving, and implies IT spending can be lower for those choosing the entire Splunk ecosystem.

The cost of implementation remains a challenge for large organizations. The yearly spend will depend on how the platform is used, allowing decisions to be made on which workloads are most important to the company.

Strengths: Splunk provides a robust environment for monitoring the digital enterprise regardless of the location and type of infrastructure. Splunk is a very strong player in the OpenTelemtry space, allowing Splunk and others to worry about the meaning of data rather than spending resources moving data.

Challenges: While the cost of Splunk’s APM solution is competitive, the cost of implementing and maintaining the Splunk platform can be high, even with workload pricing. Hybrid solutions may require professional services for design and implementation in complicated environments.

6. Analyst’s Take

Application performance monitoring as a marketplace has several established vendors that continue to add features and capabilities. For instance, Dynatrace is adding features to move closer to an AIOps solution. Others, including Splunk, New Relic, and to a lesser degree, Datadog, have also embraced monitoring the entire IT operations environment from one tool or digital platform. The established IT platform players (BMC, Broadcom, Micro Focus) make adopting their APM tool easy for shops already invested in what they offer.

Microsoft presents a different platform type in that Microsoft controls everything, including development, deployment, and monitoring. Its APM tool is part of the landscape for shops adopting “everything Microsoft” and requires initial investigation. All other APM tools can monitor .NET environments, but do not have the scope or depth Microsoft brings if the entire shop uses Microsoft for everything.

Despite the maturity of APM, some areas present new challenges. It is not simple to compare implementation costs of the various solutions. For SaaS only, there is some common ground in that pricing often centers around the number of hosts; however, that is not always the case. Splunk has introduced workload pricing. How this will change the landscape, if at all, remains to be seen.

Deployment and implementation costs are challenging to determine for all APM tools when deploying hybrid cloud and on-site environments. Deciding which pieces of the solution are on-premises, what is in the cloud, and how to roll out tools without operational impact can tax even the most experienced enterprises. Professional services add significantly to the implementation costs, and are discoverable after careful consultation with a vendor.

Innovations that will become mainstream in the next generation of APM tools include DevOps integration and predictive analysis. IBM Instana is an example of innovation featuring CI/CD integration, while Netreo adds anomaly prediction to the mix in a new way.

Synthetic monitoring in APM is now a staple for watching websites and availability monitoring. However, the use of APM in walled gardens, such as manufacturing, finance, or transportation, is a new requirement for APM and IT operations in general. For example, monitoring the temperature of a device in manufacturing environments is common in industrial control systems; however, temperature sensors themselves are more intelligent than just the control hardware. Their state and function data are added to applications, which was not possible in the past. The ability to see and predict errors from IoT devices and prevent costly shutdowns will increase the impact of APM and the developers who code their usage.

For all APM vendors, the use of AI is increasing, along with the adoption of OpenTelemetry. In any situation in which large amounts of data converge, AI/ML analyzes the data and provides more valuable insights. APM vendors are in different places along this journey. The difficulty becomes determining how the AI functionality from one vendor compares with another. There are no simple answers.

OpenTelemetry provides a consistent method of transporting data to the “intelligent” systems that consume the data. This ability will allow APM vendors to focus more on the inputs and outputs of their systems and less on the transportation of data. Not all APM vendors have embraced OpenTelemetry. The expectation is that this will change as the standard matures. The current APM leaders (Datadog, Dynatrace, New Relic, and Splunk) are significant contributors to OpenTelemetry.

Mergers and acquisitions impacted three vendors (Cisco AppDynamics, IBM Instana, and Netreo/Stackify). The combined company or products have yet to come to fruition, and so questions remain.

Any of the APM vendors investigated can become market leaders in the future.

7. About Ron Williams

Ron Williams

Ron Williams is an astute technology leader with more than 30 years’ experience providing innovative solutions for high-growth organizations. He is a highly analytical and accomplished professional who has directed the design and implementation of solutions across diverse sectors. Ron has a proven history of excellence propelling organizational success by establishing and executing strategic initiatives that optimize performance. He has demonstrated expertise in planning and implementing solutions for enterprises and business applications, developing key architectural components, performing risk analysis, and leading all phases of projects from initialization to completion. He has been recognized for promoting effective governance and positive change that improved operational efficiency, revenues, and cost savings. As an elite communicator and design architect, Ron has transformed strategic ideas into reality through close coordination with engineering teams, stakeholders, and C-level executives.

Ron has worked for the US Department of Defense (Star Wars initiative), NASA, Mary Kay Cosmetics, Texas Instruments, Sprint, TopGolf, and American Airlines, and participated in international consulting in Qatar, Brazil, and the U.K. He has led remote software and infrastructure teams in India, China, and Ghana.

Ron is a pioneer in enterprise architecture who improved response and resolution of enterprise-wide problems by deploying “smart” tools and platforms. In his current role as an analyst, Ron provides innovative technology and strategy solutions in both enterprise and SMB settings. He is currently using his expertise to analyze the IT processes of the future with particular interest in how machine learning and artificial intelligence can improve IT operations.

8. About GigaOm

GigaOm provides technical, operational, and business advice for IT’s strategic digital enterprise and business initiatives. Enterprise business leaders, CIOs, and technology organizations partner with GigaOm for practical, actionable, strategic, and visionary advice for modernizing and transforming their business. GigaOm’s advice empowers enterprises to successfully compete in an increasingly complicated business atmosphere that requires a solid understanding of constantly changing customer demands.

GigaOm works directly with enterprises both inside and outside of the IT organization to apply proven research and methodologies designed to avoid pitfalls and roadblocks while balancing risk and innovation. Research methodologies include but are not limited to adoption and benchmarking surveys, use cases, interviews, ROI/TCO, market landscapes, strategic trends, and technical benchmarks. Our analysts possess 20+ years of experience advising a spectrum of clients from early adopters to mainstream enterprises.

GigaOm’s perspective is that of the unbiased enterprise practitioner. Through this perspective, GigaOm connects with engaged and loyal subscribers on a deep and meaningful level.

9. Copyright

© Knowingly, Inc. 2021 "GigaOm Radar for Application Performance Monitoring (APM)" is a trademark of Knowingly, Inc. For permission to reproduce this report, please contact sales@gigaom.com.