Skip to content
This repository was archived by the owner on Jun 27, 2025. It is now read-only.
This repository was archived by the owner on Jun 27, 2025. It is now read-only.

Levant deploy stuck in loop when deployment object is empty #263

Closed
Closed
Levant deploy stuck in loop when deployment object is empty#263
Assignees
jrasell

Description

@msvbhat
msvbhat
opened on Dec 19, 2018

We had a nomad job which was last deployed couple of months ago. And for some reason, the /v1/job//deployments returned an empty object. So when I try to run deploy the job (with no changes to job definition), levant got stuck in a loop for more than 20 minutes. The command I ran for levant deploy is levant deploy -ignore-no-changes -var-file <file> job.nomad

levant version
Levant v0.2.5
Date: 2018-10-25T13:22:22Z
Commit: 0514741
Branch: 0.2.5
State: 0.2.5
Summary: 0514741

nomad version
Nomad v0.8.5 (90fbfaba6a6d9af7febc39082b95ed832d8b8bd6)

Debug log outputs from Levant:

I don't have the DEBUG logs output yet (lost them when we did the workaround). But the debug had lot of below output

levant/deploy: Nomad returned an empty deployment for evaluation ; retrying

When I checked the levant code, I observed that the DeploymentID is empty and that is why it got stuck.

https://github.com/jrasell/levant/blob/cc275cb120fda9dfaf20ffaebb36c12305495acb/levant/deploy.go#L446

The steps to reproduce would be to have a job that doesn't have the deployment (not sure how a job can end up in this state). And then try to deploy that job without doing any changes to job definition with -ignore-no-changes option.

If I am able to reproduce this again in our environment, I will provide more information.

Activity

  1. msvbhat commented on Jan 25, 2019

    @msvbhat
    msvbhat
    on Jan 25, 2019
    ContributorAuthor

    I am facing this issue again. Below is the logs from the deploy command.

    2019-01-25T10:42:00+01:00 |INFO| helper/variable: using variable with key tcp_service_checks and value [map[interval:10s timeout:2s]] from file
2019-01-25T10:42:00+01:00 |INFO| helper/variable: using variable with key show_datadog_logs and value true from file
2019-01-25T10:42:00+01:00 |INFO| helper/variable: using variable with key consumers_path and value server/workers/consumers/ from file
2019-01-25T10:42:00+01:00 |DEBU| levant/plan: triggering Nomad plan
2019-01-25T10:42:00+01:00 |ERRO| levant/plan: no changes detected for job
2019-01-25T10:42:00+01:00 |INFO| levant/plan: no changes found in job but ignore-changes flag set to true
2019-01-25T10:42:01+01:00 |DEBU| levant/deploy: running dynamic job count updater job_id=webapp-backend
2019-01-25T10:42:01+01:00 |INFO| levant/deploy: using dynamic count 2 for group staging job_id=webapp-backend
2019-01-25T10:42:01+01:00 |INFO| levant/deploy: triggering a deployment job_id=webapp-backend
2019-01-25T10:42:02+01:00 |INFO| levant/deploy: evaluation f7d110ef-0fbe-8d70-3cf7-b22e421ac8a2 finished successfully job_id=webapp-backend
2019-01-25T10:42:02+01:00 |INFO| levant/deploy: beginning deployment watcher for job job_id=webapp-backend
2019-01-25T10:42:02+01:00 |DEBU| levant/deploy: Nomad returned an empty deployment for evaluation f7d110ef-0fbe-8d70-3cf7-b22e421ac8a2; retrying job_id=webapp-backend
2019-01-25T10:42:04+01:00 |DEBU| levant/deploy: Nomad returned an empty deployment for evaluation f7d110ef-0fbe-8d70-3cf7-b22e421ac8a2; retrying job_id=webapp-backend
2019-01-25T10:42:06+01:00 |DEBU| levant/deploy: Nomad returned an empty deployment for evaluation f7d110ef-0fbe-8d70-3cf7-b22e421ac8a2; retrying job_id=webapp-backend
2019-01-25T10:42:08+01:00 |DEBU| levant/deploy: Nomad returned an empty deployment for evaluation f7d110ef-0fbe-8d70-3cf7-b22e421ac8a2; retrying job_id=webapp-backend
2019-01-25T10:42:10+01:00 |DEBU| levant/deploy: Nomad returned an empty deployment for evaluation f7d110ef-0fbe-8d70-3cf7-b22e421ac8a2; retrying job_id=webapp-backend
2019-01-25T10:42:13+01:00 |DEBU| levant/deploy: Nomad returned an empty deployment for evaluation f7d110ef-0fbe-8d70-3cf7-b22e421ac8a2; retrying job_id=webapp-backend
2019-01-25T10:42:15+01:00 |DEBU| levant/deploy: Nomad returned an empty deployment for evaluation f7d110ef-0fbe-8d70-3cf7-b22e421ac8a2; retrying job_id=webapp-backend
2019-01-25T10:42:17+01:00 |DEBU| levant/deploy: Nomad returned an empty deployment for evaluation f7d110ef-0fbe-8d70-3cf7-b22e421ac8a2; retrying job_id=webapp-backend
2019-01-25T10:42:19+01:00 |DEBU| levant/deploy: Nomad returned an empty deployment for evaluation f7d110ef-0fbe-8d70-3cf7-b22e421ac8a2; retrying job_id=webapp-backend

    And it gets stuck in this forever. From what I can see, it is because the deployment endpoint from Nomad has returned an empty response.

    msvbhat@tortuga$ curl --header "X-Nomad-Token: $NOMAD_TOKEN" https://staging.nomad.com:4646/v1/job/webapp-backend/deployment; echo
null

msvbhat@tortuga$ curl --header "X-Nomad-Token: $NOMAD_TOKEN" https://staging.nomad.com:4646/v1/job/webapp-backend/deployments; echo
[]

    This may also be because Nomad cluster went into some inconsistent state. But would like to see levant get around it.

  2. msvbhat commented on Jan 31, 2019

    @msvbhat
    msvbhat
    on Jan 31, 2019
    ContributorAuthor

    Another update.

    Today, I hit this again even when the -ignore-no-changes option was NOT set. So I believe this happens everytime when there is a empty deployment object.

  3. stevenscg commented on Feb 1, 2019

    @stevenscg
    stevenscg
    on Feb 1, 2019

    @msvbhat Does #264 represent the issue you're seeing or something different??

    cc @jrasell

  4. msvbhat commented on Feb 1, 2019

    @msvbhat
    msvbhat
    on Feb 1, 2019
    ContributorAuthor

    No, I don't think they are related. Because recently I have also seen the issue even without -ignore-no-changes option.

  5. stevenscg commented on Feb 1, 2019

    @stevenscg
    stevenscg
    on Feb 1, 2019

    Ok. I know that jrasell has been swamped lately.

  6. jrasell commented on Feb 2, 2019

    @jrasell
    jrasell
    on Feb 2, 2019
    Member

    Hi @msvbhat sorry for the delay; as @stevenscg has pointed out I have been pretty busy in my day to day professional job so have let Levant slide a little so apologies for that. I am dedicating some time tomorrow to fix this problem, which I believe will be solved by adding some timeout logic into the code where it checks for a deployment object rather than continually try to get a deployment ID. I am thinking this can be short, maybe 30s before exiting and stating no deployment ID was generated. If you have any thoughts please let me know.

  7. jrasell
    self-assigned this
    on Feb 2, 2019
  8. jrasell
    added
    bug
    deployment
    in progress
    on Feb 2, 2019

  9. msvbhat commented on Feb 2, 2019

    @msvbhat
    msvbhat
    on Feb 2, 2019
    ContributorAuthor

    Hi @jrasell, Thanks for replying. And I understand about the job, So please you don't have to apologise. :)

    I think timeout login feels easy to implement and would fix the problem at hand. I can also try and send a PR for timeout.

    But I was trying to understand why a deployment watcher is necessary when there is no change in the job file and no deployment is triggered at Nomad.

  10. jrasell
    added 2 commits that reference this issue on Feb 25, 2019

    Fix deploy loop bug when evaluation didnt include a deployment ID.

    064f765

    Merge branch 'master' into gh-263

    69cbb92
  11. jrasell
    closed this as completedin #268on Feb 25, 2019
  12. jrasell
    added a commit that references this issue on Feb 25, 2019

    Merge pull request #268 from jrasell/gh-263

    4bff93a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions
    ⌂ 换网址