4

I deployed wordpress and mysql and add relation between wordpress and mysql. I try to destroy wordpress before removing the relation between wordpress and mysql.

Now, both services are hanging at dying. What should I do? is there any way to manually delete the service cleanly?

I am running Ubuntu 12.04 LTS.

Here is the output from juju status mysql:

controller:~$ juju status mysql
environment: maas
machines:
  "0":
    agent-state: started
    agent-version: 1.16.6.1
    dns-name: node-1.master
    instance-id: /MAAS/api/1.0/nodes/node-345fea0a-9f84-11e3-88be-525400429c50/
    series: precise
services:
  mysql:
    charm: cs:precise/mysql-35
    exposed: false
    life: dying
    relations:
      cluster:
      - mysql
      db:
      - wordpress
    units:
      mysql/0:
        agent-state: started
        agent-version: 1.16.6.1
        life: dying
        machine: "0"
        public-address: node-1.master

Part of the output from juju status (yes, it ends at db: - mysql)

  wordpress:
    charm: cs:precise/wordpress-21
    exposed: false
    life: dying
    relations:
      db:
      - mysql

Log related(juju debug-log):

node-1:014-03-03 19:32:12 INFO juju runner.go:253 worker: start "uniter"
node-1:014-03-03 19:32:12 INFO juju.worker.uniter uniter.go:83 unit "mysql/0" started
node-1:014-03-03 19:32:12 INFO juju.worker.uniter modes.go:421 ModeInit starting
node-1:014-03-03 19:32:12 INFO juju.worker.uniter modes.go:29 updating unit addresses
node-1:014-03-03 19:32:12 INFO juju.worker.uniter.filter filter.go:454 unit is dying
node-1:014-03-03 19:32:12 DEBUG juju.worker.uniter.filter filter.go:504 charm check         skipped, unit is dying
node-1:014-03-03 19:32:12 INFO juju.worker.uniter modes.go:54 reconciling relation state
node-1:014-03-03 19:32:12 DEBUG juju.worker.uniter.filter filter.go:322 got service change
node-1:014-03-03 19:32:12 INFO juju.worker.uniter uniter.go:517 joining relation "wordpress:db mysql:db"
node-1:014-03-03 19:32:12 DEBUG juju.worker.uniter.filter filter.go:504 charm check     skipped, unit is dying
node-1:014-03-03 19:32:12 DEBUG juju.worker.uniter.filter filter.go:338 got relations change
node-1:014-03-03 19:32:12 DEBUG juju.worker.uniter.filter filter.go:314 got unit change
node-1:014-03-03 19:32:12 INFO juju.worker.uniter uniter.go:543 joined relation "wordpress:db mysql:db"
node-1:014-03-03 19:32:12 DEBUG juju.worker.uniter modes.go:423 ModeInit exiting
node-1:014-03-03 19:32:12 INFO juju.worker.uniter modes.go:421 ModeContinue starting
node-1:014-03-03 19:32:12 INFO juju.worker.uniter modes.go:67 loading uniter state
node-1:014-03-03 19:32:12 INFO juju.worker.uniter modes.go:108 found uncommitted     "config-changed" hook
node-1:014-03-03 19:32:12 INFO juju.worker.uniter uniter.go:363 committing "config-    changed" hook
node-1:014-03-03 19:32:12 DEBUG juju.worker.uniter.filter filter.go:330 got config change
node-1:014-03-03 19:32:12 DEBUG juju.worker.uniter.filter filter.go:334 preparing new config event
node-1:014-03-03 19:32:12 ERROR juju git.go:188 worker/uniter/charm: git command failed: exit status 128
node-1:ath: /var/lib/juju/agents/unit-mysql-0/charm
node-1:rgs: []string{"commit", "--allow-empty", "-m", "Completed \"config-changed\" hook."}
node-1:rror: object file .git/objects/d4/7f136f29e2319929b668b4e7917dca934b462f is     empty
node-1:atal: loose object d47f136f29e2319929b668b4e7917dca934b462f (stored in .git/objects/d4/7f136f29e2319929b668b4e7917dca934b462f) is corrupt
node-1:014-03-03 19:32:12 DEBUG juju.worker.uniter modes.go:423 ModeContinue exiting
node-1:014-03-03 19:32:12 INFO juju.worker.uniter uniter.go:105 unit "mysql/0"     shutting down: ModeContinue: git commit failed: exit status 128
node-1:014-03-03 19:32:12 ERROR juju.worker.uniter.filter filter.go:117 tomb: dying
node-1:014-03-03 19:32:12 ERROR juju runner.go:211 worker: exited "uniter":     ModeContinue: git commit failed: exit status 128
node-1:014-03-03 19:32:12 INFO juju runner.go:245 worker: restarting "uniter" in 3s
node-1:014-03-03 19:32:15 INFO juju runner.go:253 worker: start "uniter"
node-1:014-03-03 19:32:15 INFO juju.worker.uniter uniter.go:83 unit "mysql/0" started
node-1:014-03-03 19:32:15 INFO juju.worker.uniter modes.go:421 ModeInit starting
node-1:014-03-03 19:32:15 INFO juju.worker.uniter modes.go:29 updating unit addresses
node-1:014-03-03 19:32:15 INFO juju.worker.uniter.filter filter.go:454 unit is dying
node-1:014-03-03 19:32:15 DEBUG juju.worker.uniter.filter filter.go:504 charm check     skipped, unit is dying
node-1:014-03-03 19:32:15 INFO juju.worker.uniter modes.go:54 reconciling relation     state
node-1:014-03-03 19:32:15 DEBUG juju.worker.uniter.filter filter.go:322 got service     change
node-1:014-03-03 19:32:15 INFO juju.worker.uniter uniter.go:517 joining relation     "wordpress:db mysql:db"
node-1:014-03-03 19:32:15 DEBUG juju.worker.uniter.filter filter.go:504 charm check     skipped, unit is dying
node-1:014-03-03 19:32:15 DEBUG juju.worker.uniter.filter filter.go:338 got relations     change
node-1:014-03-03 19:32:15 DEBUG juju.worker.uniter.filter filter.go:314 got unit     change
node-1:014-03-03 19:32:15 INFO juju.worker.uniter uniter.go:543 joined relation     "wordpress:db mysql:db"
node-1:014-03-03 19:32:15 DEBUG juju.worker.uniter modes.go:423 ModeInit exiting
node-1:014-03-03 19:32:15 INFO juju.worker.uniter modes.go:421 ModeContinue starting
node-1:014-03-03 19:32:15 INFO juju.worker.uniter modes.go:67 loading uniter state
node-1:014-03-03 19:32:15 INFO juju.worker.uniter modes.go:108 found uncommitted     "config-changed" hook
node-1:014-03-03 19:32:15 INFO juju.worker.uniter uniter.go:363 committing "config-    changed" hook
node-1:014-03-03 19:32:15 DEBUG juju.worker.uniter.filter filter.go:330 got config     change
node-1:014-03-03 19:32:15 DEBUG juju.worker.uniter.filter filter.go:334 preparing new     config event
node-1:014-03-03 19:32:15 ERROR juju git.go:188 worker/uniter/charm: git command     failed: exit status 128
node-1:ath: /var/lib/juju/agents/unit-mysql-0/charm
node-1:rgs: []string{"commit", "--allow-empty", "-m", "Completed \"config-changed\"     hook."}
node-1:rror: object file .git/objects/d4/7f136f29e2319929b668b4e7917dca934b462f is empty
node-1:atal: loose object d47f136f29e2319929b668b4e7917dca934b462f (stored in     .git/objects/d4/7f136f29e2319929b668b4e7917dca934b462f) is corrupt
node-1:014-03-03 19:32:15 DEBUG juju.worker.uniter modes.go:423 ModeContinue exiting
node-1:014-03-03 19:32:15 INFO juju.worker.uniter uniter.go:105 unit "mysql/0"     shutting down: ModeContinue: git commit failed: exit status 128
node-1:014-03-03 19:32:15 ERROR juju.worker.uniter.filter filter.go:117 tomb: dying
node-1:014-03-03 19:32:15 ERROR juju runner.go:211 worker: exited "uniter":     ModeContinue: git commit failed: exit status 128
node-1:014-03-03 19:32:15 INFO juju runner.go:245 worker: restarting "uniter" in 3s

Please let me know if there is an easy fix? Any suggestion would be appreciated. Thanks.

1 Answer 1

2

I'm not 100% positive this works with MAAS - but I know this method works with other providers. When I want to destroy a service and its "stuck" in a dying state, using your wordpress deployment as an example:

juju resolve wordpress/0

now if that fails to help the situation, ergo, it continues to go from hook to hook in an error state, I will destroy the machine with extreme predjudice. (note, this yields an unrecoverable machine, and should be treated with caution, as you would any rm -rf operation - this WILL destroy the machine in question)

Obtain the machine id from the command juju status - then:

 juju destroy-machine --force <machine_id>

If this continues to leave the mysql service in a state of distress, you can resolve it manually following the above workflow:

juju resolve mysql/0

if all else fails, and you dont care about the data

juju destroy-machine --force <machine_id>

In relation to the log output - know that there is an effort to move juju away from using git, so stray issues like this do not crop up for end users. I don't have an ETA on when this feature will land but its WIP at current.

2
  • Thank you lazyPower. I believe I have tried juju resolved wordpress/0 but it doesn't work neither does destroy-machine. It gave me error message like: mysql service is still on. I don't remember clearly, but even the --force option cannot terminate a machine with service on it? If the machine is pending with error or have no response, the --force does work. I finally destroy the environment and bootstrap again. I thought there might have some other issues with the node instead of juju.
    – jzc.emacs
    Mar 4, 2014 at 22:57
  • @lazyPower This is real. It is also an issue if you have an error in your relation joined hook. Service goes into error state, and you can't kill it, except by killing the machines. This is a PITA when developing hooks, as I have need to bootstrap every iteration. Now I'm not deploying to the bootstrap/juju-gui server, so it goes faster.
    – rmustakos
    Apr 2, 2015 at 17:55

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .