Deployment randomly failing on deploy:update_code

[Seb33300]

Deployer Version

v7.5.12

Target OS

Ubuntu

Which PHP version are you using?

PHP 8.3

Content of deploy.php or deploy.yaml

Hello,

Since 1 week, our deployments started to randomly fail with no reason on the deploy:update_code step and I do not really understand the error.
This is really strange because we did not change anything, if someone has an idea?

Latest change was 1 month ago when we upgraded from 7.5.8 to 7.5.12.
And everything was working fine so far (we have multiple deployment per day) until last week when it started to happen randomly (~50% of our deployment are now failing).

We tried to reboot the VM, but the issue still persists.

The deployment is triggered from GitHub Actions.

task deploy:update_code
  [dev-{0}.****.com] /usr/bin/php8.3 /home/runner/work/****/****/deployer.phar worker --port 46587 --task deploy:update_code --host dev-{0}.****.com --option branch=feature/external-users-related-tables-changes
   --option deploy_path=/app/dev-1391.****.com
   --option become=www-data
   --option keep_releases=3 --ansi --no-interaction --decorated -vvv
  [dev-{0}.****.com] ssh '-A' '-o' 'ControlMaster=auto' '-o' 'ControlPersist=60' '-o' 'ControlPath=/dev/shm/dev.****.com' 'dev.****.com' ': 389699cb7489e1c84ce6; sudo -H -u www-data bash -ls'
  [dev-{0}.****.com] run command -v 'git' || which 'git' || type -p 'git'
  [dev-{0}.****.com] /usr/bin/git
  [dev-{0}.****.com] ssh '-A' '-o' 'ControlMaster=auto' '-o' 'ControlPersist=60' '-o' 'ControlPath=/dev/shm/dev.****.com' 'dev.****.com' ': 02babe513f85dc327319; sudo -H -u www-data bash -ls'
  [dev-{0}.****.com] run [ -d /app/dev-1391.****.com/.dep/repo ] || mkdir -p /app/dev-1391.****.com/.dep/repo
  [dev-{0}.****.com] ssh '-A' '-o' 'ControlMaster=auto' '-o' 'ControlPersist=60' '-o' 'ControlPath=/dev/shm/dev.****.com' 'dev.****.com' ': 63d125dae678a905946e; sudo -H -u www-data bash -ls'
  [dev-{0}.****.com] run export GIT_TERMINAL_PROMPT='0' GIT_SSH_COMMAND='ssh -o StrictHostKeyChecking=accept-new'; [ -f /app/dev-1391.****.com/.dep/repo/HEAD ] || /usr/bin/git clone --mirror git@github-****:****/****.git /app/dev-1391.****.com/.dep/repo 2>&1
  [dev-{0}.****.com] Cloning into bare repository '/app/dev-1391.****.com/.dep/repo'...
  [dev-{0}.****.com] ssh '-A' '-o' 'ControlMaster=auto' '-o' 'ControlPersist=60' '-o' 'ControlPath=/dev/shm/dev.****.com' 'dev.****.com' ': 6911b36b9b6f57638534; sudo -H -u www-data bash -ls'
  [dev-{0}.****.com] run ps x | grep 63d125dae678a905946e | grep -v grep | awk '{print $1}'
  [dev-{0}.****.com] ssh '-A' '-o' 'ControlMaster=auto' '-o' 'ControlPersist=60' '-o' 'ControlPath=/dev/shm/dev.****.com' 'dev.****.com' ': 8194f261edb6574e6020; sudo -H -u www-data bash -ls'
  [dev-{0}.****.com] run kill -9 -
  [dev-{0}.****.com] bash: line 1: kill: -: arguments must be process or job IDs
  [dev-{0}.****.com]  error  in update_code.php on line 95:
  [dev-{0}.****.com] exit code 1 (General error)
ERROR: Task deploy:update_code failed!

Steps to reproduce

This happens randomly, I cannot find a way to reproduce it.

[antonmedv]

Maybe it is timeout?

[Seb33300]

Hi @antonmedv

Thanks, I think you are right.

When the deployment fails, before triggering the error, it's stuck for ~5min on that step:

[dev-{0}.port360.com] Cloning into bare repository '/app/dev-1392.****.com/.dep/repo'...

And when the deployment works, it looks like the git clone completed just before the timeout happens.

In our case, we are deploying into a fresh environment every PR created by our developers.
Which means deployer cannot just pull the latest changes from the previous deployment.
It's almost always a git clone triggered (unless the PR is updated).

In order to speedup this, we may need to be able to use the git --depth, but I just found this issue where you said it's not needed.

Is there any solution?

[Seb33300]

I just found in the documentation that the default update_code_strategy is archive

I did not override the update_code_strategy option, so why is it using git clone in my case?

[antonmedv]

The repo is cloned as bare just before doing an archive:

deployer/recipe/deploy/update_code.php

Line 95 in 30742fc

run("[ -f $bare/HEAD ] || $git clone --mirror $repository $bare 2>&1", env: $env);

I think we need to refactor this part. And make local archive a default option in v8.

Also I think we can add --depth 0 to clone of bare repo.

[Seb33300]

Any idea of workaround i could do to fix it until the refactor is done?

Never mind, just realized there is a default_timeout option. I will just increase it a bit for now.

[Seb33300]

Hi @antonmedv, I see that you marked this thread as resolved.
But increasing the default timeout is just a workaround IMO.

On big projects, always cloning the repo as bare can take a lot of time and this goes worse over the time.
I believe this should be improved as you suggested.

I think we need to refactor this part. And make local archive a default option in v8.

[antonmedv]

But why cloning is gappening every time? Do you deploy to fresh environment every time?

[Seb33300]

Almost. In our project, we deploy a fresh environment for every PR created.

[antonmedv]

I see, only for PR is fresh.