Gitlab Update

| Dec 8, 2021| Devops
gitlab update 升级测试
升级路线
参考官网路线
10.7->10.8.7 -> 11.11.8 -> 12.0.12 -> 12.1.17 -> 12.10.14 -> 13.0.14 -> 13.1.11 -> 13.8.8 -> 13.12.15 -> latest 14.0.Z -> latest 14.1.Z -> latest 14.Y.Z
https://docs.gitlab.com/ee/update/#upgrade-paths
官网的升级路线有问题纠正如下
10.7->10.8.7 -> 11.11.8 -> 12.0.12 -> 12.1.17 -> 12.10.14 -> 13.0.14 -> 13.1.11 -> 13.8.8
-> 13.9.2-> 13.12.12 -> 14.0.11 -> 14.1.6->14.1.7->14.2.1->14.2.5->14.2.6->14.2.7-> 14.3.4->14.4.3->14.5.2
恢复备份集并修改了数据库链接
vim  gitlab.rb 
gitlab_rails['db_host'] = ""
重启gitlab

遇到的错误

WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.

解决 
vim /etc/sysctl.conf
net.core.somaxconn = 2048


2021-12-08_09:51:01.78141 612:M 08 Dec 09:51:01.781 # Fatal error loading the DB: Permission denied. Exiting.

==> /var/log/gitlab/gitlab-workhorse/current <==
2021-12-08_09:51:02.20495 time="2021-12-08T09:51:02Z" level=info msg="redis: dialing" address=/var/opt/gitlab/redis/redis.socket network=unix
2021-12-08_09:51:02.20498 time="2021-12-08T09:51:02Z" level=error msg="unknown error" error="keywatcher: dial unix /var/opt/gitlab/redis/redis.socket: connect: connection refused"

解决
docker  exec -ti gitlab bash   
chown gitlab-redis /var/opt/gitlab/redis/*
升级
拉取10.8.7 的镜像 
https://hub.docker.com/r/gitlab/gitlab-ce/  
需要修改git.sh 启动脚本的镜像版本号

docker run -d \
    -p 80:80 \
    -p 443:443 \
    -p 22:22 \
    --name gitlab \
    --restart unless-stopped \
    -v /data/gitlab/config:/etc/gitlab \
    -v /data/gitlab/logs:/var/log/gitlab \
    -v /data/gitlab/data:/var/opt/gitlab \
    gitlab-ce:11.11.8-ce.0
    
 #另外需要注意，每一个版本升级都需要替换gitlab.rb 配置文件
从10.8.7 升级到11.11.8 报错
..................
PG::UndefinedObject: ERROR:  operator class "gin_trgm_ops" does not exist for access method "gin"
    : CREATE  INDEX CONCURRENTLY "index_tags_on_name_trigram" ON "tags" USING gin ("name" gin_trgm_ops)
    ........................
解决
# 需要pg 的扩展
yum install postgresql12-contrib.x86_64 
psql -U root  -h ip  -d gitlabhq_production 
CREATE EXTENSION pg_trgm;

故障2 
There was an error running gitlab-ctl reconfigure:

service[gitlab-workhorse] (dynamically defined) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of /opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitlab-workhorse ----
STDOUT: fail: /opt/gitlab/service/gitlab-workhorse: runsv not running
STDERR: 
---- End output of /opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitlab-workhorse ----
Ran /opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitlab-workhorse returned 1

解决
gitlab-ctl prometheus-upgrade
从12.0.12 升级到12.1.17 故障
故障1 
/opt/gitlab/embedded/bin/runsvdir-start: line 24: ulimit: pending signals: cannot modify limit: Operation not permitted
/opt/gitlab/embedded/bin/runsvdir-start: line 37: /proc/sys/fs/file-max: Read-only file system

解决
docker exec -it gitlab update-permissions
docker restart gitlab

故障2 
Error executing action `restart` on resource 'service[gitlab-workhorse]'

解决
gitlab-rake gitlab:check SANITIZE=true
gitlab-ctl reconfigure

docker exec -it gitlab    gitlab-ctl reconfigure
13.8.8.8
1. 这个版本需要升级PG 数据库到12.9
2. 迁移存储库参考官网https://docs.gitlab.com/ee/administration/raketasks/storage.html#migrate-to-hashed-storage
pg 数据库迁移
# 备份数据
/usr/pgsql-12/bin/pg_dump  -h 10.39.15.2 -U root -C   gitlabhq_production > gitlabhq_production.sql

# 导入数据
sudo -u postgres /usr/pgsql-12/bin/psql -U  postgres < gitlabhq_production.sql

# 创建
CREATE USER dbuser WITH PASSWORD '*****';
GRANT ALL PRIVILEGES ON DATABASE gitlabhq_production TO gitlab;
ALTER USER myuser WITH SUPERUSER;

sudo -u postgres /usr/pgsql-12/bin/psql -U gitlab -d gitlabhq_production   -h 10.39.15.3 
CREATE EXTENSION pg_trgm;
存储库迁移步骤
https://gitlab.com/gitlab-org/gitlab/-/issues/289816 

# 进入 Rails 控制台
gitlab-rails console

# 查询 项目 read-only 打开的 
projects = Project.where(repository_read_only: true)

# 关闭 项目的 read-only
projects.each do |p|
  p.update!(repository_read_only:nil)
end

# 存储库迁移
 gitlab-rake gitlab:storage:migrate_to_hashed

#　全部迁移成功，以下命令查看所列出的项目总数与页面的理应一致
gitlab-rake gitlab:storage:hashed_projects

# 查看，全部迁移成功以下两条命令应该为 0 
gitlab-rake gitlab:storage:legacy_projects
gitlab-rake gitlab:storage:legacy_attachments

# 全部迁移成功，以下两条命令应该没有输出
gitlab-rake gitlab:storage:list_legacy_projects
gitlab-rake gitlab:storage:list_legacy_attachments
13.9.2 -> 13.12.12 -> 14.0.11故障

如下报错在14.1.+ 版本解决

ActionView::Template::Error (PG::UndefinedTable: ERROR: relation "services" does not exist

直接按照官网的升级路线，13.12.15 -> latest 14.0.Z -> latest 14.1.Z -> latest 14.Y.Z 会遇到如下

Finalize it manualy by running
        sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_stages,id,'[["id"]\, ["id_convert_to_bigint"]]']

For more information, check the documentation   https://docs.gitlab.com/ee/user/admin_area/monitoring/background_migrations.html#database-migrations-failing-because-of-batched-background-migration-not-finished


https://forum.gitlab.com/t/upgrade-problems-debian-14-2-4/58822

#如果有这样的报错，就返回到上一个升级的完整版本，
然后docker  exec -ti  gitlab bash 
执行官网提示的语句
gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,events,id,'[["id"]\, ["id_convert_to_bigint"]]']

gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,push_event_payloads,event_id,'[["event_id"]\, ["event_id_convert_to_bigint"]]']
14.1.17 升级到14.2.1 报错
#报错
Finalize it manualy by running
        sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_stages,id,'[["id"]\, ["id_convert_to_bigint"]]']

docker  restart  gitlab  
然后打开一个新的终端执行  docker exec -ti  gitlab gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_stages,id,'[["id"]\, ["id_convert_to_bigint"]]']
有done 显示标识成功或者$? 返回为0表示正常
#Devops

See Also

comments powered by Disqus