Skip to content

Logs, Metrics and Traces: Are They Enough for Monitoring Microservices?

Val Pliskin
By Val Pliskin
7 min read
API Observability
Logs, Metrics and Traces: Are They Enough for Monitoring Microservices?

Modern software development requires fast action and scalable deployments to catch up with the speed of technology. Nowadays, we develop applications with small teams and deploy them to cloud providers in an agile way. Instead of large monolith applications, smaller services—namely, microservices—are being developed and deployed. 

Because the way we develop software has changed, requirements, methods, architectural paradigms, and performance metrics are also different. Thus, we need to use new methods for monitoring and observability. In this blog post, we will discuss the conventional methods of microservices observability and why they’re challenging.

Monitoring and Observability of Microservices

Microservices are lightweight and modular software services with a limited scope. Microservices architecture, therefore, can be defined as the design, development, and deployment of complex applications as a group of microservices. As mentioned earlier, today’s teams are smaller than before, and they create microservices that communicate with each other via APIs. 

The three main characteristics of microservices are:

  • Loosely coupled: Microservices have minimal dependency on other services.

  • Independent deployment: You can develop, test, and deploy each microservice separately, without creating turbulence.

  • Focused on business value: Each microservice should deliver on business value—and be as small as possible.

That said, although microservices architecture looks like a silver bullet for modern software development, it has its own challenges. These include:

  • Complexity: Microservices create a fragmented and distributed architecture, which means more stuff to manage and coordinate.

  • Cultural shift: When you convert from a monolith to microservices architecture, team members must change their mindset to create more modular and efficient services. 

  • Limited observability: It’s challenging to manage the hundreds of services developed and deployed separately, as you can rapidly lose control due to the domino effect. 

Despite these challenges, microservices are a highly adopted architectural approach. Here are some of the best practices for designing, deploying, and operating them:

  • Containers: Containers are the most popular approach for developing and deploying microservices. Because the container infrastructure separates executables for packaging and runtime, it is the perfect fit for microservices architecture. With popular tools like Kubernetes, it is now reasonably easy to manage containers in a scalable and flexible way.

Containers
  • API gateways: Microservices provide APIs for internal and external communication. API gateways are the services offered by cloud providers or other servicers, such as Kong and nginx, to handle and route requests between microservices. They help make services more lightweight and focus on core business values.

  • Service mesh: Communication between microservices is challenging when you consider security, load balancing, and encryption. A service mesh is a dynamic platform built on top of the networking infrastructure to control how microservices share data. 

Now that we have briefly discussed the characteristics, challenges, and best practices of microservices, let’s dive into monitoring and observability.

What is Microservices Observability? The Traditional Methods

API governance

Monitoring microservices involves creating holistic observability of all cloud-native infrastructures and the applications running on them. In order to ensure such an overview, you need to collect the status data of all your applications, performance data of the infrastructure, requests of the users, and API responses. 

Independent of which tool you are using, your monitoring stack should concentrate on the following critical goals:

  • Observability: Create a comprehensive overview of the stack to help teams find problems and performance issues.

  • Automated discovery and instrumentation: Automatically detect new applications and services and probe them to collect metrics.  

  • Distributed tracing: To find bottlenecks in the microservices stack, track the requests as they travel between services. If there are end-user requests to your microservices, you’ll need to record and evaluate the performance from the real user’s point of view.

To achieve these goals, you need to collect, gather, and analyze measurement data from the microservices and underlying infrastructure. 

Common data sources include:

  • Application data: Data related to your microservices and their business operations

  • Infrastructure/platform data: Data from the cloud services and platforms that your applications depend on  

  • Operational data: Data related to the deployment and management of your microservices, such as upgrade, deployment, and scaling up or down events

When you look at the data types from these sources, there are three essential categories of data to collect: metrics, logs, and traces:

  • Metrics: Metrics are numeric data measurements instrumented over intervals of time. Prometheus is the most popular monitoring system, with flexible query language and an efficient time-series database to collect and work on the metrics.

  • Logs: Event logs are time-stamped records of events that happened in your applications. The ELK Stack is the most commonly used stack in the market. It consists of three open-source products: Elasticsearch, Logstash, and Kibana. The stack helps you store, search, analyze, and visualize log data from various sources.

  • Traces: Traces are a series of casually related events that create a flow of end-to-end requests. Jaeger and Zipkin are the most popular tools for collecting and managing distributed tracing data. Both support distributed tracing standards, such as OpenTelemetry and OpenTracing, and provide rich visualization of traces.

traces

As you can see, the tools and data types we’ve discussed focus on the status of the component itself. This means that you will see the status of each microservice in your stack, but will lack a holistic overview. Thus, monitoring and observability of distributed microservices is more complex than just tracking metrics, logs, and traces. 

The traditional methods discussed here focus on surrounding the production environment and providing insights about the up-and-running applications. On the other hand, cloud-native modern approaches focus on getting insights from development processes, like the build, deployment, and operation stages. In the following section, we will focus on the challenges of modern cloud-native monitoring and discuss the differences in perspective compared to traditional methods.

Challenges of Modern Cloud-Native Monitoring

With modern cloud-native applications, you are developing and managing APIs that are served by the microservices. Therefore, the success of your applications depends on the reliability of the APIs. To measure the success of the APIs, you need to monitor and observe them. However, this is not an easy task, as cloud-native applications with microservices architectures have automated deployment strategies and complex networking setups. In addition, it is common to deploy and manage these applications via Kubernetes and service mesh installations over Kubernetes, such as Istio. In other words, you need to track the state of the APIs that are installed, managed, and communicated via complex systems. 

Here are some of the major challenges you’ll face when creating holistic observability of your microservices:

  • Analysis of API interactions: You need to collect and create an inventory of the APIs you are developing and deploying. It is essential to track the status of all APIs in your stack and ensure they are healthy.

  • Detecting breaking API changes: The microservices in your stack communicate via APIs. You need to ensure they can always understand each other by avoiding breaking API changes. 

  • Automated API tests: The APIs in your system should be testable and automatically checked before deployment. 

API management

These challenges are an essential part of the observability of scalable and reliable microservices. However, they are not covered by traditional monitoring tools, such as Prometheus, the ELK Stack, or Jaeger. That’s because conventional methods are based on probing the services and collecting metrics, logs, or traces. 

On the other hand, modern cloud-native microservices applications require you to monitor what they actually provide: APIs. With modern methods, you will get insights into API behavior, while traditional monitoring only gives you performance metrics and information. In addition, modern methods will help you create overall observability, while traditional ones will probe your applications and show their individual status.

How Seekret Can Help

Seekret solves these challenges by breaking down information silos between microservices and the teams that manage them. We do so by mapping and monitoring your APIs at the schema level, and automatically notifying your teams whenever a schema change occurs (breaking or not). With Seekret’s API observability platform, you can enforce API build policies, analyze interactions between your services and consumers, create templates for more accurate API tests, and ascertain production readiness. 

Book a demo today to start monitoring and observing your microservices with a modern API-first approach!