DEV Community

Deploying gitlab on Docker Swarm

Livio Ribeiro on January 14, 2018

Update: You may want to see the instructions on the gitlab documentation. Docker Swarm is a container orchestration system that is very easy to se...
Collapse
 
danielfrancora profile image
danielfrancora

Hi,

Sorry for the newbie question but, how do I change the volumes you created using nfs to a local driver path?
I'm using GlusterFS as a fileshare and I'm having some trouble to put your docker-compose to work because the volume mount type.

I've tryed:
grafana_data:
driver: local
driver_opts:
device: "/opt/local/docker/docker-data/gitlab/grafana"
type: none
o: bind

But I get error on mount.

Can you help?

Thx in advance

Collapse
 
livioribeiro profile image
Livio Ribeiro

Does the directory "/opt/local/docker/docker-data/gitlab/grafana" already exist? In my tests, I could only make it work when the directory to be mounted already exists on the host.

Collapse
 
danielfrancora profile image
danielfrancora

Yes, the directory exists. I created all the directory structure you pointed in the beginning.

Collapse
 
apetushkov profile image
Alexey Petushkov

Hi Livio,
Thank you for your article. One thing to add:

Add line to /etc/fstab

/srv/gitlab-swarm/      /exports/gitlab-swarm/  none    bind

In other case, it will fail to start after reboot (due to NFS shares)

Collapse
 
livioribeiro profile image
Livio Ribeiro

Thanks! I've updated the post with your suggestion.

Collapse
 
azaars profile image
azaars

Hi Livio,

This is an excellent article indeed. Thanks for sharing.

I'm going to try it out and so, I should be able to scale it up by running

docker service scale mystack_gitlab=3

Will it actually work in a cluster this way?

Collapse
 
livioribeiro profile image
Livio Ribeiro

I never tried that, so I cannot say if it would work.

You can find out about high availability on Gitlab's documentation. The stack I showed is not far from it.

Collapse
 
hagbard profile image
Nick Parlow

Hi Livio,
Thanks for this excellent article. really easy to follow, and i've now got gitlab up and running on a three node swarm... :D

any ideas on how to introduce more gitlab front end containers? i've tried following the article here:
docs.gitlab.com/ce/administration/...

but the second node just bounces up and down.

the gitlab article implies that the second and subsequent servers should only need access to the shared secrets, and otherwise use the same confg; they already have acess t othose because the gitlab data directory is on the shared nfs path....

any ideas?

Collapse
 
livioribeiro profile image
Livio Ribeiro

You need to get the logs from the container to see what went wrong.

I just tested this gitlab deploy (although using gitlab 11) and it worked with 3 replicas.

Collapse
 
hagbard profile image
Nick Parlow

right - still couldn't get subsequent containers to be stable, even with gitlab 11... but when i rearranged my shared storage, made tw oseparate gitlab services and altered the stack.yaml file, it works:
/srv/gitlab-swarm/
├── gitlab1
│   ├── config
│   ├── data
│   └── logs
├── gitlab2
│   ├── config
│   ├── data
│   └── logs
├── gitlabshared
│   └── data
│   ├── gitdata
│   ├── ssh
│   ├── gitlabrails
│   │ ├── uploads
│   │ └── shared
│   └── gitlabci
│   └── builds
├── grafana
├── postgres
└── prometheus

volumes:
gitlab1_data:
drive: local
driver_opts:
type: nfs4
o: "addr=10.17.17.10"
device: ":/gitlab-swarm/gitlab/data"
gitlab1_logs:
driver: local
driver_opts:
type: nfs4
o: "addr=10.17.17.10"
device: ":/gitlab-swarm/gitlab/logs"
gitlab1_config:
driver: local
driver_opts:
type: nfs4
o: "addr=10.17.17.10"
device: ":/gitlab-swarm/gitlab/config"
gitlab2_data:
drive: local
driver_opts:
type: nfs4
o: "addr=10.17.17.10"
device: ":/gitlab-swarm/gitlab/data"
gitlab2_logs:
driver: local
driver_opts:
type: nfs4
o: "addr=10.17.17.10"
device: ":/gitlab-swarm/gitlab/logs"
gitlab2_config:
driver: local
driver_opts:
type: nfs4
o: "addr=10.17.17.10"
device: ":/gitlab-swarm/gitlab/config"
gitlab_gitdata
driver: local
driver_opts:
type: nfs4
o: "addr=10.17.17.10"
device: ":/gitlab-swarm/gitlabshared/data"
gitlab_ssh
driver: local
driver_opts:
type: nfs4
o: "addr=10.17.17.10"
device: ":/gitlab-swarm/gitlabshared/ssh"
gitlab_gitrailsupload
driver: local
driver_opts:
type: nfs4
o: "addr=10.17.17.10"
device: ":/gitlab-swarm/gitlabshared/data/gitlabrails/upload"
gitlab_gitrailsshared
driver: local
driver_opts:
type: nfs4
o: "addr=10.17.17.10"
device: ":/gitlab-swarm/gitlabshared/gitlabrails/shared"
gitlab_gitcibuilds
driver: local
driver_opts:
type: nfs4
o: "addr=10.17.17.10"
device: ":/gitlab-swarm/gitlabshared/gitlabci/builds"

services:
gitlab1:
image: "gitlab/gitlab-ce:10.3.3-ce.0"
volumes:
- "gitlab1_data:/var/opt/gitlab"
- "gitlab1_logs:/var/log/gitlab"
- "gitlab1_config:/etc/gitlab"
- "gitlab_gitdata:/var/opt/gitlab/gitdata"
- "gitlab_ssh:/var/opt/gitlab/.ssh"
- "gitlab_gitrailsupload:/var/opt/gitlab/gitlab-rails/uploads"
- "gitlab_gitrailsshared:/var/opt/gitlab/gitlab-rails/shared"
- "gitlab_gitcibuilds:/var/opt/gitlab/gitlab-ci/builds"

gitlab2:
image: "gitlab/gitlab-ce:10.3.3-ce.0"
volumes:
- "gitlab2_data:/var/opt/gitlab"
- "gitlab2_logs:/var/log/gitlab"
- "gitlab2_config:/etc/gitlab"
- "gitlab_gitdata:/var/opt/gitlab/gitdata"
- "gitlab_ssh:/var/opt/gitlab/.ssh"
- "gitlab_gitrailsupload:/var/opt/gitlab/gitlab-rails/uploads"
- "gitlab_gitrailsshared:/var/opt/gitlab/gitlab-rails/shared"
- "gitlab_gitcibuilds:/var/opt/gitlab/gitlab-ci/builds"

also need to add the shared secrets stuff into gitlab.rb for both services

gitlab_shell['secret_token'] = 'fbfb19c355066a9afb030992231c4a363357f77345edd0f2e772359e5be59b02538e1fa6cae8f93f7d23355341cea2b93600dab6d6c3edcdced558fc6d739860'
gitlab_rails['otp_key_base'] = 'b719fe119132c7810908bba18315259ed12888d4f5ee5430c42a776d840a396799b0a5ef0a801348c8a357f07aa72bbd58e25a84b8f247a25c72f539c7a6c5fa'
gitlab_rails['secret_key_base'] = '6e657410d57c71b4fc3ed0d694e7842b1895a8b401d812c17fe61caf95b48a6d703cb53c112bc01ebd197a85da81b18e29682040e99b4f26594772a4a2c98c6d'
gitlab_rails['db_key_base'] = 'bf2e47b68d6cafaef1d767e628b619365becf27571e10f196f98dc85e7771042b9203199d39aff91fcb6837c8ed83f2a912b278da50999bb11a2fbc0fba52964'

(these are the ones from the gitlab docs, obviously they're not the ones i used. i span up a single node stack, copied the data from the secrets.json file, and then put those into my gitlab.rb file.

thanks for the brilliant article - it helped me a lot. now to make postgres HA. :|

Collapse
 
iahmadkhan profile image
Ijaz ahmad

Hi ,

thanks for sharing,

I have mounted an nfs share from a production nfs filer/server on a VM , there is a specific user that has RW access to that share , how can I use that share in setting up this stack , where in this compose configuration I should specific the user credentials so that docker machines can use that.

And how can I make this work if I want to land each service on a different docker host , and what changes should be made on NFS related. NFS share is mounted on each machine with the same user credentials.

thanks
ijaz

Collapse
 
livioribeiro profile image
Livio Ribeiro

Sorry but I cannot help, I don't know much about NFS and never had the chance to use this kind of setup in production.

The "o" option in the volume driver configuration can receive any nfs configuration (as far as I know), maybe you can do something with this and the server setup.

Collapse
 
rmrfetc profile image
Rob

Does this also run a private docker registry in the gitlab container?

Collapse
 
livioribeiro profile image
Livio Ribeiro

Since the storage is set up, then I believe yes