Take the 2-minute tour ×
Ask Ubuntu is a question and answer site for Ubuntu users and developers. It's 100% free, no registration required.

I was doing some experimentation with a test charm on Juju on AWS and managed to get my service in to a completely hung state. juju service returns the following.

environment: amazon
machines:
  "0":
    agent-state: started
    agent-version: 1.16.5
    dns-name: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    instance-id: i-7c2f4c52
    instance-state: running
    series: precise
    hardware: arch=amd64 cpu-cores=1 cpu-power=100 mem=1740M root-disk=8192M
  "5":
    agent-state: down
    agent-state-info: (started)
    agent-version: 1.16.5
    instance-id: i-9cb9cbb2
    instance-state: missing
    series: precise
    hardware: arch=amd64 cpu-cores=1 cpu-power=100 mem=1740M root-disk=8192M
services:
  metest:
    charm: local:precise/metest-0
    exposed: false
    life: dying
    relations:
      cluster:
      - metest
    units:
      metest/0:
        agent-state: down
        agent-state-info: (started)
        agent-version: 1.16.5
        life: dying
        machine: "5"
        open-ports:
        - 80/tcp
        public-address: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

(I've removed the DNS names just in case!). The instance-id for machine 5 has been terminated according to the AWS management console. None of "destroy-unit metest/0", "destroy-service metest" and "destroy-machine 5" clear the problem, and I can't redeploy the service with it in this state. juju resolve seems to have no effect either.

Googling the issue, the only resolution I can find is to completely blow away my environment - which is not a great option. Is there any way I can clear the problem otherwise? What is the general method for debugging this kind of issue?

The root cause of the problem: We use Chef for most of our orchestration and have found that occasional failure between Chef and AWS API leave orphaned instances around. Since all the instances we launch from Chef are tagged with a name, and these orphaned instances are unnamed, to avoid giving Amazon money needlessly we added code to our knife plugins to terminate unnamed instances. I'm sure you can see where this is going ...

Is there any way to clean up machines once they are in this state (--force doesn't help) - and I would also like to know if there are any plans to allow instances to be named so they are identifiable in the EC2 management console (something like juju-- would be ideal)?

Things I have tried:

  • destroy-machine --force doesn't seem to clean things up. I don't get an error, but it appears like nothing changed in the status.
share|improve this question
    
I found out the root cause of the problem. We use Chef for most of our orchestration and have found that occasional failure between Chef and AWS API leave orphaned instances around. Since all the instances we launch from Chef are tagged with a name, and these orphaned instances are unnamed, to avoid giving Amazon money needlessly we added code to our knife plugins to terminate unnamed instances. I'm sure you can see where this is going ... –  Mike Evans Feb 12 at 11:25
    
Unfortunately juju doesn't name instances on EC2, and since we are doing Juju experiments on the same AWS account as people doing Chef orchestration, our Juju instances keep getting blow away from underneath us. I took a quick look at the Juju EC2 code, and it appears there is no way to turn on instance naming. –  Mike Evans Feb 12 at 11:27
    
So my original question still stands - is there any way to clean up machines once they are in this state (--force doesn't help) - and I would also like to know if there are any plans to allow instances to be named so they are identifiable in the EC2 management console (something like juju-<environment>-<machine number> would be ideal)? –  Mike Evans Feb 12 at 11:29
1  
Hi Mike, I've redited your question to reflect the new information. Please feel free to continue to edit it to make it clear (don't put the new information in comments, that just confuses people). Thanks! –  Jorge Castro Feb 12 at 14:53
    
@MikeEvans You could identify juju instances in the EC2 console by the security groups they're part of, e.g. juju-<envname>-0 for machine 0. –  dimitern Feb 12 at 15:23
add comment

1 Answer 1

You could try:

juju destroy-machine --force 5

The --force option of destroy-machine is available since 1.16.5 and should remove the hung machine and all units on it. Then you should be able to redeploy your service, but if it says "service already exists" just deploy it with a different name.

If all else fails, juju destroy-environment -e <name> is always an option. I'm not sure if it did support --force as well in 1.16.5.

share|improve this answer
    
destroy-machine --force doesn't seem to clean things up. I don't get an error, but it appears like nothing changed in the status. –  Mike Evans Feb 12 at 11:23
    
Can you also add --show-log --debug after --force and paste the log you're getting? Please use paste.ubuntu.com. Also, it might take a minute after running destroy-machine for the status to get updated. –  dimitern Feb 12 at 11:53
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.