[date: 2022-05-22 11:57] [visits: 72]

动手系列之开发并部署APISIX插件

HTTP服务

在实际工作中用以完成各种业务场景的后台服务多数为HTTP Server,此处笔者先为集群实现一个HTTP Server用于后续实验所需,这个服务命名为echo,目的是将HTTP Request的信息内容在HTTP Response Body返回。

新建目录echo,添加文件index.js、Dockerfile、k8s.yaml:

$ mkdir echo && cd echo
$ touch index.js Dockerfile k8s.yaml
$ tree .
.
├── Dockerfile
├── index.js
└── k8s.yaml

0 directories, 3 files

index.js

const http = require('http');
const server = http.createServer((req, res) => {
    let {
        method,
        url,
        headers,
    } = req;

    res.setHeader('content-type', 'application/json');
    res.end(JSON.stringify({
        method,
        url,
        headers,
    }), err => {
        console.log(`${Date.now()} ${method} ${url} ${err ? 'fail' : 'success'}`);
    });
});
server.listen('8080');

Dockerfile

FROM node:lts-alpine
WORKDIR /usr/src/app
COPY . .
EXPOSE 8080

CMD node index.js

构建:

$ docker build --pull -t lab.com:release.echo.001 .
Sending build context to Docker daemon  3.072kB
Step 1/5 : FROM node:lts-alpine
lts-alpine: Pulling from library/node
Digest: sha256:1a9a71ea86aad332aa7740316d4111ee1bd4e890df47d3b5eff3e5bded3b3d10
Status: Image is up to date for node:lts-alpine
 ---> e5065cc78074
Step 2/5 : WORKDIR /usr/src/app
 ---> Using cache
 ---> 66657994466c
Step 3/5 : COPY . .
 ---> f75f52b8f915
Step 4/5 : EXPOSE 8080
 ---> Running in cf96d9cf79de
Removing intermediate container cf96d9cf79de
 ---> 6cbfe441d19d
Step 5/5 : CMD node index.js
 ---> Running in 2350d6d6b60e
Removing intermediate container 2350d6d6b60e
 ---> a78bde0cb89b
Successfully built a78bde0cb89b
Successfully tagged lab.com:release.echo.001

k8s.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: echo
  namespace: lab
  labels:
    app: echo
spec:
  replicas: 2
  selector:
    matchLabels:
      app: echo
  template:
    metadata:
      labels:
        app: echo
    spec:
      restartPolicy: Always
      containers:
        - image: "lab.com:release.echo.001"
          imagePullPolicy: IfNotPresent
          name: echo
          ports:
            - containerPort: 8080
              name: http
              protocol: TCP
          readinessProbe:
            failureThreshold: 6
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
            tcpSocket:
              port: 8080

---

apiVersion: v1
kind: Service
metadata:
  name: echo
  namespace: lab
  labels:
    app: echo
spec:
  ports:
    - name: http
      port: 8080
      protocol: TCP
      targetPort: 8080
  selector:
    app: echo

部署服务到k8s集群:

$ kubectl -n lab apply -f k8s.yaml
deployment.apps/echo created
service/echo created

$ kubectl -n lab get svc echo
NAME   TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
echo   ClusterIP   10.98.69.239   <none>        8080/TCP   46s

$ kubectl -n lab get pods |grep echo
echo-5f98ffcf77-6dth4     1/1     Running   0               80s
echo-5f98ffcf77-7w5qr     1/1     Running   0               80s

$ curl http://10.98.69.239:8080
{"method":"GET","url":"/","headers":{"host":"10.98.69.239:8080","user-agent":"curl/7.68.0","accept":"*/*"}}#

自定义插件

实现HTTP Server后接着开发一个插件,插件目标设计为追踪请求链路,如果请求有携带traceId则传递转发到下一个服务否则生成一个traceId,另在response header中返回traceId。

apisix/extra/lua/apisix/plugins/trace-id.lua

在前次部署APISIX到K8S中预留的extra目录中编写如下代码:

local ngx = ngx
local core = require('apisix.core')
local uuid = require('resty.jit-uuid')

local schema = {
    type = 'object',
    properties = {},
    required = {},
}
local _M = {
    version = 1.0,
    name = 'trace-id',
    priority = 0,
    schema = schema,
}

function _M.check_schema(conf)
    return core.schema.check(schema, conf)
end

function _M.rewrite(conf, ctx)
    local id = core.request.header(ctx, 'x-trace-id')
    if not id then
        id = uuid.generate_v4()
    end

    core.request.set_header(ctx, 'x-trace-id', id)
    ctx['x-trace-id'] = id
end

function _M.header_filter(conf, ctx)
    core.response.set_header('x-trace-id', ctx['x-trace-id'])
end

function _M.log(conf, ctx)
    ngx.var.trace_id = ctx['x-trace-id']
end

return _M

apisix/Dockerfile

修改apisix的Dockerfile在构建镜像时将插件copy进去:

FROM apache/apisix:2.13.0-alpine
COPY conf/apisix.yaml /usr/local/apisix/conf/config.yaml
COPY extra /usr/local/apisix/extra
EXPOSE 9080 9090 9091 9443

相比较之前的内容,此处还多expose了control api端口9090

apisix/conf/apisix.yaml

修改apisix配置文件以支持自定义插件:

etcd:
  host:
    - http://etcd-client.lab.svc.cluster.local:2379

apisix:
  allow_admin:
    - 0.0.0.0/0
  admin_key:
    - name: admin
      key: your-secret
      role: admin
  extra_lua_path: /usr/local/apisix/extra/lua/?.lua
  enable_control: true
  control:
    ip: 0.0.0.0
    port: 9090

nginx_config:
  http_server_configuration_snippet: |
    set $trace_id "";
  http:
    access_log_format: '{"remote":"$remote_addr","time":"$time_local","method":"$request_method","uri":"$request_uri","status":$status,"request_time":$request_time,"agent":"$http_user_agent","upstream_status":$upstream_status,"upstream_time":$upstream_response_time,"upstream":"$upstream_addr","trace_id":"$trace_id"}'
    access_log_format_escape: json

plugins:
  - proxy-rewrite
  - trace-id

将nginx访问日志改为json形式输出为后续日志采集进ElasticSearch做铺垫

构建镜像并更新到k8s集群

构建:

$ docker build --pull -t lab.com:release.apisix.001 apisix
Sending build context to Docker daemon   7.68kB
Step 1/4 : FROM apache/apisix:2.13.0-alpine
2.13.0-alpine: Pulling from apache/apisix
Digest: sha256:a39aa5e8d75f188111e9e64c2c8ecd728ad08a3522c0b4bf867531e32459031c
Status: Image is up to date for apache/apisix:2.13.0-alpine
 ---> 890821d363b4
Step 2/4 : COPY conf/apisix.yaml /usr/local/apisix/conf/config.yaml
 ---> Using cache
 ---> 9e4e81400847
Step 3/4 : COPY extra /usr/local/apisix/extra
 ---> Using cache
 ---> 17f9db61cfba
Step 4/4 : EXPOSE 9080 9090 9091 9443
 ---> Using cache
 ---> fc36158522bd
Successfully built fc36158522bd
Successfully tagged lab.com:release.apisix.001

修改k8s资源描述文件apisix.yaml的image为lab.com:release.apisix.001,然后引用:

$ kubctl -n lab apply -f k8s-yaml/apisix.yaml
deployment.apps/apisix configured
configmap/apisix-dashboard-config unchanged
pod/apisix-dashboard configured
service/apisix unchanged
service/apisix-dashboard unchanged

路由

HTTP服务与插件开发完成后,尝试配置第一个路由并启用trace-id插件,但配置路由的方式这里有两个不同的选择,Etcd Vs Stand-alone:

笔者进行了一定的思考后最终从两者中选择了Etcd,原因是两者作为控制平面的控制方式,相比较Etcd更为灵活,Stand-alone则有更多局限性。

确定使用Etcd管理路由规则后,有两种思路:

笔者在实验中使用两者结合,最终目标是完全使用后者。

Windows方式

打开http://admin.lab.com前一篇的内容)使用UI操作操作增加一个路由,但由于admin-dashboard的插件同步需要通过apisix control导出并同步schema.json,笔者最终会向Linux方式靠近故没有做相关适配,因而UI上无法配置插件trace-id

配置并生效路由后,在dashboard中可以查看路由的json描述,如下:

{
  "uri": "/*",
  "name": "echo",
  "methods": [
    "GET",
    "POST",
    "PUT",
    "DELETE",
    "HEAD",
    "OPTIONS"
  ],
  "plugins": {
    "proxy-rewrite": {
      "scheme": "http"
    }
  },
  "upstream": {
    "nodes": [
      {
        "host": "echo.lab.svc.cluster.local",
        "port": 8080,
        "weight": 1
      }
    ],
    "timeout": {
      "connect": 3,
      "send": 3,
      "read": 6
    },
    "type": "roundrobin",
    "hash_on": "vars",
    "scheme": "http",
    "pass_host": "pass",
    "keepalive_pool": {
      "idle_timeout": 60,
      "requests": 1024,
      "size": 16
    }
  },
  "status": 1
}

Linux方式

在使用dashboard配置路由后,浏览器中已经可以访问http://lab.com/hello-world,但trace-id插件未生效,可使用apisix admin api为路由添加:

$ curl http://10.109.76.252:9080/apisix/admin/routes/408684119430529881 -H 'X-API-KEY: your-secret' -X PATCH -d '
{

  "plugins": {
        "trace-id":{}
  }
}'
...

通过dashboard查看或实际测试确认trace-id生效:

$ curl http://lab.com/hello-world
{"method":"GET","url":"/hello-world","headers":{"host":"lab.com","x-real-ip":"10.244.0.1","x-forwarded-for":"192.168.168.168, 10.244.0.1","x-forwarded-proto":"http","x-forwarded-host":"lab.com","x-forwarded-port":"80","x-forwarded-ssl":"off","user-agent":"curl/7.68.0","accept":"*/*","x-trace-id":"a17399de-4742-40fb-a809-7d364a0f7be3"}}#

$ curl -H 'x-trace-id: a17399de-4742-40fb-a809-7d364a0f7be3' curl http://lab.com/thank-you
{"method":"GET","url":"/thank-you","headers":{"host":"lab.com","x-real-ip":"10.244.0.1","x-forwarded-for":"192.168.168.168, 10.244.0.1","x-forwarded-proto":"http","x-forwarded-host":"lab.com","x-forwarded-port":"80","x-forwarded-ssl":"off","user-agent":"curl/7.68.0","accept":"*/*","x-trace-id":"a17399de-4742-40fb-a809-7d364a0f7be3"}}#

未传x-trace-id时插件生成,传的时候保持透传,查看网关日志确认trace-id有效。

结语

通过本次动手实验完成了如下几件事情:

如果是在实际项目使用apisix,路由规则维护可能还需下些功夫,路由规则不多时可通过将dashboard schema.json同步到位并使用UI管理,而如果路由规则多笔者个人推荐用统一的配置文件描述,再通过脚本等方式进行自动同步管理。

动手系列的文章笔者时间主要花在动手上,文章只是把动手最后的结果用文字方式尽量呈现出来,可预见的是如果读者只是阅读而未实际动手,这文章的作用甚微,故有条件的读者极力推荐实际动手!

Ref