【k8s学习笔记二】腾讯云harbor私有仓库部署实践

2023-07-12 00:28:25 浏览数 (2)

Harbor是由VMware公司开源的企业级的Docker Registry管理项目，相比docker官方拥有更丰富的权限权利和完善的架构设计，适用大规模docker集群部署提供仓库服务,并且提供UI界面。 一般harbor可以通过helm或者docker-compose安装，本文以compose安装为例，介绍harbor如何配置腾讯云对象存储COS作为私有镜像仓库存储地址。 另外，除了使用自建harbor，也可使用腾讯云镜像仓库简化维护过程： https://cloud.tencent.com/product/tcr

一 部署过程

1 准备：

安装docker-compose

代码语言：txt复制
curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose

2 重启docker

代码语言：txt复制
systemctl daemon-reload

3 下载habor装包解压 harbor.v2.1.0.tar.gz

3.1 配置项注意：

  • hostname 修改主机地址，即访问域名
  • https 增加HTTPS证书配置，注意如果使用了CLB，需要在CLB同时配置证书
  • storage_service 中配置COS信息，注意harbor支持aws S3,可以在s3中配置COS桶信息，需要在对象存储提前建好对应桶信息
  • 注意harbor日志路径：默认 /var/log/harbor/registryctl.log
代码语言：txt复制
# harbor.v2.1.0.tar.gz
vi harbor.yml
-----------harbor.yml---------
# Configuration file of Harbor

# The IP address or hostname to access admin UI and registry service.
# DO NOT use localhost or 127.0.0.1, because Harbor needs to be accessed by external clients.
hostname: harbor.yourset.com

# http related config
http:
  # port for http, default is 80. If https enabled, this port will redirect to https port
  port: 80

# https related config
https:
  # https port for harbor, default is 443
  port: 443
  # The path of cert and key files for nginx
  certificate: /data/key/harbor.yourset.com.crt
  private_key: /data/key/harbor.yourset.xyz.key

# # Uncomment following will enable tls communication between all harbor components
# internal_tls:
#   # set enabled to true means internal tls is enabled
#   enabled: true
#   # put your cert and key files on dir
#   dir: /etc/harbor/tls/internal

# Uncomment external_url if you want to enable external proxy
# And when it enabled the hostname will no longer used
# external_url: https://reg.mydomain.com:8433

# The initial password of Harbor admin
# It only works in first time to install harbor
# Remember Change the admin password from UI after launching Harbor.
harbor_admin_password: Harborxxx

# Harbor DB configuration
database:
  # The password for the root user of Harbor DB. Change this before any production use.
  password: root123
  # The maximum number of connections in the idle connection pool. If it <=0, no idle connections are retained.
  max_idle_conns: 50
  # The maximum number of open connections to the database. If it <= 0, then there is no limit on the number of open connections.
  # Note: the default number of connections is 1024 for postgres of harbor.
  max_open_conns: 1000

# The default data volume
data_volume: /data

# Harbor Storage settings by default is using /data dir on local filesystem
# Uncomment storage_service setting If you want to using external storage
storage_service:
  s3:
     accesskey: xxxxxx
     secretkey: xxxxxxx
     regionendpoint: cos.ap-guangzhou.myzijiebao.com
     region: ap-guangzhou
     bucket: harborgz-xxxxx
     secure: true
   # ca_bundle is the path to the custom root ca certificate, which will be injected into the truststore
   # of registry's and chart repository's containers.  This is usually needed when the user hosts a internal storage with self signed certificate.

#   ca_bundle:

#   # storage backend, default is filesystem, options include filesystem, azure, gcs, s3, swift and oss
#   # for more info about this configuration please refer https://docs.docker.com/registry/configuration/
#   filesystem:
#     maxthreads: 100
#   # set disable to true when you want to disable registry redirect
#   redirect:
#     disabled: false

###....

4 启动，进入安装目录

代码语言：txt复制
$ docker-compose down -v

# 修改配置后，需要重新检查运行环境
$ ./prepare

# 启动harbor服务
$ docker-compose up -d

5 测试

5.1 网页测试，如果需要https访问，需要申请证书并配置在CLB及harbor服务器上

配置在harbor.yml的https中：

代码语言：txt复制
-----------------
  https:
  # https port for harbor, default is 443
  port: 443
  # The path of cert and key files for nginx
  certificate: /data/key/harbor.yourset.com.crt
  private_key: /data/key/harbor.yourset.xyz.key

5.2 本地测试，输入密码

5.3 测试推镜像

注意事项：需要在harbor中先建立项目，否则会推送失败

代码语言：txt复制
[root@centos ~/tmp]# docker tag hello-world 127.0.0.1/s3/hello-world:v1.0.0 
[root@centos ~/tmp]# docker push 127.0.0.1/s3/hello-world:v1.0.0
The push refers to repository [127.0.0.1/s3/hello-world]
f22b99068db9: Preparing 
unauthorized: project not found, name: s3: project not found, name: s3
[root@centos ~/tmp]# docker push 127.0.0.1/s3/hello-world:v1.0.0
The push refers to repository [127.0.0.1/s3/hello-world]
f22b99068db9: Pushed 
v1.0.0: digest: sha256:1b26826f602946860c279fce65829b57792 size: 525

5.4 对象存储中也生成了相关文件：

二 踩坑记录

2.1 坑一：S3配置踩坑

网上搜的文章及其他用户反馈S3部分的配置如下：

代码语言：txt复制
s3:
      region: ap-xxx
      bucket: xx-sigp-xxxxxxx
      accesskey: xxxxxxx
      secretkey: xxxxx
      endpoint: cos.ap-singapore.myzijiebao.com
      secure: true

使用该配置后，启动harbor后总会有harbor-registryclt等容器不断重启，造成无法推拉镜像：

查看错误日志：

代码语言：txt复制
tail -f /var/log/harbor/registryctl.log 

Aug  3 15:32:31 172.30.0.1 registryctl[28778]: 2021-08-03T07:32:31Z [ERROR] [/registryctl/config/config.go:63]: failed to load storage driver, err:No region parameter provided
Aug  3 15:32:31 172.30.0.1 registryctl[28778]: 2021-08-03T07:32:31Z [FATAL] [/registryctl/main.go:78]: Failed to load configurations with error: No region parameter provided

关键信息：

询问几个同事都没有结论，在google查了半天也没找到相关的文档，只好决定从源码入手，先去查看registryctl/main.go源码：

继续查看config.go:63

https://github.com/goharbor/harbor/blob/9e117539492b9e54658b8c4dd240af231c351cb5/src/registryctl/config/config.go#L71

查看setStorageDriver()

找出storagedriver中s3的相关代码

查看s3部分：

https://github.com/distribution/distribution/blob/01f589cf8726565aa3c5c053be12873bafedbedc/registry/storage/driver/s3-aws/s3.go#L109

发现一段特别的地方：

当"regionendpoint"为空时，程序会去aws的官方的 validRegins列表中查询可用区，而本次是要配置腾讯云COS地址，当然在aws的region列表里面没有，所以会提示 err:No region parameter provided。

因此需要传入"regionendpoint"的key才可避免查询aws自己的region list（网上的文章误导人啊），而不是传入"endpoint"

，所以需要在harbor.yml中把配置改为：

代码语言：txt复制
s3:
      region: ap-xxx
      bucket: xx-sigp-xxxxxxx
      accesskey: xxxxxxx
      secretkey: xxxxx
      regionendpoint: cos.ap-singapore.myzijiebao.com
      secure: true

修改后重载harbor启动成功，推拉镜像正常。

2.2 坑二：COS 强一致配置踩坑

有用户根据上面部署后，发现新的报错，现象是通过docker push文件成功，但是harbor总返回500报错：

经过与COS团队沟通，主要是list强一致问题，即put 文件后直接list不一定能list到文件，因为list是最终一致性的，需要COS运维同事下发强一致的配置后，会保障能list出来。

发配置需提供客户账号appid、可用区、桶名称信息（可向腾讯云提工单），发布配置后该报错可解决。

三 总结

代码语言：txt复制
1 部署过程及时记录自己操作过程，关注日志
代码语言：txt复制
2 如果网上没有现成的答案，请教身边的专家
代码语言：txt复制
3 实在解决不了，去查源码，所有的逻辑都已经写在代码里了

k8s学习笔记合集：

【k8s学习笔记一】容器及Docker基础

【k8s学习笔记二】腾讯云harbor私有仓库部署实践

【k8s学习笔记三】kubernetes本地环境部署测试

【k8s学习笔记四】kubernetes ingress测试

容器镜像服务 对象存储 存储

0 人点赞