I was doing some experimentation with a test charm on Juju on AWS and managed to get my service in to a completely hung state. juju service returns the following.
environment: amazon machines: "0": agent-state: started agent-version: 1.16.5 dns-name: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx instance-id: i-7c2f4c52 instance-state: running series: precise hardware: arch=amd64 cpu-cores=1 cpu-power=100 mem=1740M root-disk=8192M "5": agent-state: down agent-state-info: (started) agent-version: 1.16.5 instance-id: i-9cb9cbb2 instance-state: missing series: precise hardware: arch=amd64 cpu-cores=1 cpu-power=100 mem=1740M root-disk=8192M services: metest: charm: local:precise/metest-0 exposed: false life: dying relations: cluster: - metest units: metest/0: agent-state: down agent-state-info: (started) agent-version: 1.16.5 life: dying machine: "5" open-ports: - 80/tcp public-address: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
(I've removed the DNS names just in case!). The instance-id for machine 5 has been terminated according to the AWS management console. None of "destroy-unit metest/0", "destroy-service metest" and "destroy-machine 5" clear the problem, and I can't redeploy the service with it in this state. juju resolve seems to have no effect either.
Googling the issue, the only resolution I can find is to completely blow away my environment - which is not a great option. Is there any way I can clear the problem otherwise? What is the general method for debugging this kind of issue?
The root cause of the problem: We use Chef for most of our orchestration and have found that occasional failure between Chef and AWS API leave orphaned instances around. Since all the instances we launch from Chef are tagged with a name, and these orphaned instances are unnamed, to avoid giving Amazon money needlessly we added code to our knife plugins to terminate unnamed instances. I'm sure you can see where this is going ...
Is there any way to clean up machines once they are in this state (--force doesn't help) - and I would also like to know if there are any plans to allow instances to be named so they are identifiable in the EC2 management console (something like juju-- would be ideal)?
Things I have tried:
destroy-machine --forcedoesn't seem to clean things up. I don't get an error, but it appears like nothing changed in the status.