Envoy: Integrate With Grpc-Web in K8s

Dec 27, 2019

4 mins read

Note Kubernetes Envoy Grpc Grpc web Backend

Envoy is a proxy for modern web app. More importantly, it has a first class support for gRPC.

In this example, I want to extend the helloworld example from the official grpc-web: When you have multiple service, how can one manages the traffic by Envoy.

I will use a cluster which managed by kubernetes, to integrate envoy with grpc-web, we have several things to do:

1. Let’s make two services

service1.proto

syntax = "proto3";

package service1;

service ServiceOneServer {
  rpc SayHello(HelloRequest) returns (HelloReply);
}

message HelloRequest {
  string name = 1;
}

message HelloReply {
  string message = 1;
}

service2.proto

syntax = "proto3";

package service2;

service ServiceTwoServer {
  rpc SayHello(HelloRequest) returns (HelloReply);
}

message HelloRequest {
  string name = 1;
}

message HelloReply {
  string message = 1;
}

2. Generate the javascript code using `protoc` and `protoc-gen-grpc-web`

protoc -I=. service1.proto \
  --js_out=import_style=commonjs:. \
  --grpc-web_out=import_style=commonjs,mode=grpcwebtext:.
protoc -I=. service2.proto \
  --js_out=import_style=commonjs:. \
  --grpc-web_out=import_style=commonjs,mode=grpcwebtext:.

inside the generated service1_grpc_web_pb.js you cand find something like this:

const methodDescriptor_Service1_SayHello = new grpc.web.MethodDescriptor(
  '/service1.ServiceOneServer/SayHello',
  grpc.web.MethodType.UNARY,
  ...

note that the '/service1.ServiceOneServer/SayHello' is the route that generated by grpc-web and we will observe the pattern /{package}.{service}/{api} and use it to configure our envoy proxy.

3. Implementation

I will skip the implementations of the service1 and service2, you can do any languages that support gRPC

4. Configurate Envoy

Now it’s time to write an envoy.yaml file to configure the behavior of Envoy

# for envoy admin
admin:
  access_log_path: /tmp/admin_access.log
  address:
    socket_address: { address: 0.0.0.0, port_value: 9901 }

# for the main proxy
static_resources:
  listeners:
  - name: listener_0
    address:
      # the port_value here specify the port which envoy will listen at
      socket_address: { address: 0.0.0.0, port_value: 8080 }
    filter_chains:
    - filters:
      - name: envoy.http_connection_manager
        config:
          codec_type: auto
          stat_prefix: ingress_http
          route_config:
            name: local_route
            virtual_hosts:
            - name: local_service
              domains: ["*"]
              # add all your service here
              # the prefix in the routes' match is exactly the pattern in the previous generated code
              routes:
              - match: { prefix: "/service1.ServiceOne" }
                route:
                  # this will map to name tag in line 50
                  cluster: service1 
                  max_grpc_timeout: 10s
              - match: { prefix: "/service2.ServiceTwo" }
                route:
                  cluster: service2
                  max_grpc_timeout: 10s
              cors:
                allow_origin:
                - "*"
                allow_methods: GET, PUT, DELETE, POST, OPTIONS
                allow_headers: keep-alive,user-agent,cache-control,content-type,content-transfer-encoding,custom-header-1,x-accept-content-transfer-encoding,x-accept-response-streaming,x-user-agent,x-grpc-web,grpc-timeout
                max_age: "1728000"
                expose_headers: custom-header-1,grpc-status,grpc-message
          http_filters:
          - name: envoy.grpc_web
          - name: envoy.cors
          - name: envoy.router
  clusters:
    # the name here will be matched by the cluster tag in line 31
  - name: service1 
    connect_timeout: 0.25s
    type: strict_dns
    http2_protocol_options: {}
    lb_policy: round_robin
    # the address and port_value here specify where the traffic should be pass to
    # since it is inside k8s, the address can simply be the port name thanks to kubernetes dns
    hosts: [{ socket_address: { address: service1, port_value: 10001 }}]

  - name: service2
    connect_timeout: 0.25s
    type: strict_dns
    http2_protocol_options: {}
    lb_policy: round_robin
    hosts: [{ socket_address: { address: service2, port_value: 10002 }}]

More explicitly, the service1’s yaml loos like this, note that the ports.name in containers is exactly the name used in envoy’s config: hosts: [{ socket_address: { address: service1, port_value: 10001 }}]

apiVersion: v1
kind: Service
metadata:
  labels:
    app: service1
  name: service1
spec:
  ports:
    - port: 10001
      targetPort: 10001
  selector:
    app: service1
  type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: service1
spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
  replicas: 1
  selector:
    matchLabels:
      app: service1
  template:
    metadata:
      labels:
        app: service1
    spec:
      containers:
        - name: service1
          image: somewhere/service1
          imagePullPolicy: Never
          ports:
            - name: service1
              containerPort: 10001
              protocol: TCP

5. Docker

Add a dockerfile, the dockerfile itself is quite simple, all you have to do is to copy the envoy.yaml you just wrote to the envoy image

FROM envoyproxy/envoy:latest
COPY ./configs/envoy.yaml /etc/envoy/envoy.yaml
CMD /usr/local/bin/envoy -c /etc/envoy/envoy.yaml

6. Then deploy the envoy service into your cluster

apiVersion: v1
kind: Service
metadata:
  name: envoy
spec:
  type: NodePort
  selector:
    app: envoy
  ports:
  - name: envoy
    protocol: TCP
    port: 80
    targetPort: 8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: envoy
spec:
  replicas: 1
  selector:
    matchLabels:
      app: envoy
  template:
    metadata:
      labels:
        app: envoy
    spec:
      containers:
      - name: envoy
        image: local/envoy
        imagePullPolicy: Never
        ports:
        - name: envoy
          containerPort: 8080