Introduction

The journey began with a simple question posed to our Senior Software Engineer: "Forget speed concerns. Doyou genuinely believe developing communication in gRPC is better than REST?" The immediate response I didn't want to hear came: "Absolutely yes."

Background

Prior to raising the question, I had been monitoring peculiar behavior in our services during rolling updates and Pod expansions. The majority of our microservices previously communicated via REST calls without any issues. We had migrated some of these integrations to gRPC, primarily to eliminate the overhead of REST. Recently, we observed a series of problems that all pointed in one direction - our gRPC communication.

Kubernetes Load Balancing

Of course, we followed the recommended practices for running gRPC in Kubernetes without a service mesh. On the server side, we used a headless Service object, and on the client side, gRPC employed client-side round-robin load balancing and discovery.

However, things took an interesting turn when we realized that the load balancer built into Kubernetes, at Layer 4, is not designed for load balancing RPCs. Layer 4 load balancers are common due to their simplicity, as they are protocol-agnostic. However, gRPC disrupts the connection-level load balancing provided by Kubernetes.

The reason for this is that gRPC is built on HTTP/2, which is designed to maintain a long-lived TCP connection where all requests can be active simultaneously at any point in time. This reduces the overhead of connection management. However, in this scenario, connection-level load balancing is not very useful since once a connection is established, there is no more load balancing required. All requests are pinned to the original destination Pod until a new DNS discovery (using headless Service) takes place. This doesn't happen until at least one of the existing connections is closed.

Example: Illustrating gRPC Load Balancing

Network I/O activity on a newly created Pod after the gRPC configuration change:

Configuration Tweaks

Two configurations resolved the issue, technically speaking, in one line. As mentioned earlier, we were using client-side load balancing with DNS discovery via a headless Service object. Other options could involve using proxy load balancing or implementing another discovery method that queries the Kubernetes API instead of DNS.

Additionally, the gRPC documentation provides a proposal for server-side connection management, which we also experimented with. Here's a sample Go code snippet for setting the following server parameters that I recommend you try:

      grpc.KeepaliveParams(keepalive.ServerParameters{
        MaxConnectionAge: time.Second  30, // THIS one does the trick
        MaxConnectionAgeGrace: time.Second  10,
      })
    

Real-World Behavior

The new Pod count after the gRPC configuration change:

Rolling Update Issues

With the expansion issues resolved, another problem became more evident. The focus shifted to gRPC code=UNAVAILABLE errors occurring on the client side during rolling updates. Strangely, this was only observed during the rolling update and not during individual Pod expansions.

The number of gRPC errors during the rolling update:

DNS Rediscovery

The rolling update process is straightforward: create a new replica set, create a new Pod, once the Pod is ready, terminate the old Pod from the old replica set, and so on. The start time between each Pod is spaced by 15 seconds.

We know that gRPC DNS rediscovery only initiates when the old connection is dropped or ends with a GOAWAY signal. So every 15 seconds, the clients initiate a new rediscovery, but they are met with stale DNS records. They then keep retrying the rediscovery until they succeed.

DNS TTL Cache

Except it wasn't a DNS issue... DNS TTL caching is prevalent in almost every place. The infrastructure DNS has its own cache. The clients suffer from their default 30-second TTL cache, while Go clients typically do not implement DNS caching. In contrast, Java clients reported hundreds or even thousands of occurrences of this issue.

Of course, we could reduce the time for the TTL cache, but why would this only affect gRPC during rolling updates?

Solution

Fortunately, there is an easy-to-implement workaround. Or rather, the solution is: have the new Pods start with a 30-second delay.

      .spec.minReadySeconds = 30
    

The Kubernetes Deployment spec allows us to set a minimum amount of time a new Pod must be ready before it starts terminating old Pods. After this time, the connections areterminated.

Conclusion

The investigations and experiments conducted in this article have shed light on some of the challenges and potential issues that can arise when utilizing gRPC in a Kubernetes environment. While gRPC provides certain advantages over REST, such as improved performance and efficiency, it is important to consider the specific requirements and constraints of your application when making the choice between the two.

By understanding the intricacies of gRPC communication, connection management, and Kubernetes load balancing, you can effectively deploy and scale your gRPC-based services within Kubernetes.