Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

how to migrate from hopsoft/graphite-statsd to graphiteapp/graphite-statsd #23

Closed
codexiao opened this issue Dec 4, 2017 · 24 comments
Closed
Labels

Comments

@codexiao
Copy link

codexiao commented Dec 4, 2017

Hi,

I'm currently using docker-compose managing a grafana and hopsoft/graphite-statsd deployment, where a host volume is attached to the hopsoft/graphite-statsd container's /opt/graphite/storage. Everything is working fine, except hopsoft/graphite-statsd is still at 0.91 that a few functions I need is missing. So, when i'm trying to migrate my hopsoft/graphite-statsd to graphiteapp/graphite-statsd:1.0.2-3. I thought it's just simply replacing the image name. Looks like i'm totally wrong. I noticed that graphiteapp/graphite-statsd:1.0.2-3 can't initialize itself with the same host volume. Anyone can give me some advice how to do this migration smoothly? I would like to preserve all the collected data that currently stored on my local volume by hopsoft/graphite-statsd, and I would like Grafana to function normally after migration. Anyway to achieve this?

My docker-compose snippet is as follow. Thanks a lot!

my_graphite:
image: graphiteapp/graphite-statsd:1.0.2-3 #hopsoft/graphite-statsd #graphiteapp/graphite-statsd:1.0.2-3
volumes:
- my_data/graphite/storage:/opt/graphite/storage

@deniszh
Copy link
Member

deniszh commented Dec 4, 2017

Hello @codexiao ,
Could you please elaborate what is not working when you just replacing hopsoft image with this one?
Any errors in logs? What browser says?

@codexiao
Copy link
Author

codexiao commented Dec 4, 2017 via email

@deniszh
Copy link
Member

deniszh commented Dec 4, 2017

Django version was upgraded in the process of migration to 1.0.x, so, you should upgrade your database's schema. Try to run

PYTHONPATH=/opt/graphite/webapp python django-admin.py migrate auth --settings=graphite.settings
PYTHONPATH=/opt/graphite/webapp python django-admin.py migrate --settings=graphite.settings

in target docker container after mounting old storage (use docker exec -it <container_id_or_name> /bin/bash first).

@codexiao
Copy link
Author

codexiao commented Dec 5, 2017

Thanks a lot, @deniszh . Bear with me, I'm not very experience with this. So I should follow these steps, right?

################
1). Chang the image name to my target image.
-- image: hopsoft/graphite-statsd
++ image: graphiteapp/graphite-statsd:1.0.2-3

2). docker-compose up -d
3). docker exec -it /bin/bash
4). docker exec -it PYTHONPATH=/opt/graphite/webapp python django-admin.py migrate auth --settings=graphite.settings
5). docker exec -it PYTHONPATH=/opt/graphite/webapp python django-admin.py migrate --settings=graphite.settings
################

By doing this, all my data collected by hopoft's graphite will be kept and recognized by graphiteapp/graphite-statsd, and my grafana will function as normal, right?

Thank you!!!
Xiao Wang

@codexiao
Copy link
Author

codexiao commented Dec 5, 2017

@deniszh I just tried it.
First, I replace hopsoft/graphite-statsd with graphiteapp/graphite-statsd:1.0.2-3
Then, I docker-compose up the service.
And then i follow your suggestions, but it gives me the following error. Can you help provide some guidance?

docker exec -it fba2dc92dceb /bin/bash

root@fba2dc92dceb:~# PYTHONPATH=/opt/graphite/webapp python django-admin.py migrate auth --settings=graphite.settings
python: can't open file 'django-admin.py': [Errno 2] No such file or directory

@deniszh
Copy link
Member

deniszh commented Dec 5, 2017

@codexiao ,
Oh, sorry, didn't test it. Try to add full path to django-admin.py, that should work:

PYTHONPATH=/opt/graphite/webapp python /usr/local/bin/django-admin.py migrate auth --settings=graphite.settings
PYTHONPATH=/opt/graphite/webapp python /usr/local/bin/django-admin.py migrate --settings=graphite.settings

@codexiao
Copy link
Author

codexiao commented Dec 6, 2017

@deniszh Thanks. I tested it again. It went a step further but still give me errors, see below. any idea?


root@a1f7bfe26094:/# PYTHONPATH=/opt/graphite/webapp python /usr/local/bin/django-admin.py migrate auth --settings=graphite.settings
Operations to perform:
Apply all migrations: auth
Running migrations:
Rendering model states... DONE
Applying contenttypes.0001_initial...Traceback (most recent call last):
File "/usr/local/bin/django-admin.py", line 5, in
management.execute_from_command_line()
File "/usr/local/lib/python2.7/dist-packages/django/core/management/init.py", line 354, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python2.7/dist-packages/django/core/management/init.py", line 346, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 394, in run_from_argv
self.execute(*args, **cmd_options)
File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 445, in execute
output = self.handle(*args, **options)
File "/usr/local/lib/python2.7/dist-packages/django/core/management/commands/migrate.py", line 222, in handle
executor.migrate(targets, plan, fake=fake, fake_initial=fake_initial)
File "/usr/local/lib/python2.7/dist-packages/django/db/migrations/executor.py", line 110, in migrate
self.apply_migration(states[migration], migration, fake=fake, fake_initial=fake_initial)
File "/usr/local/lib/python2.7/dist-packages/django/db/migrations/executor.py", line 148, in apply_migration
state = migration.apply(state, schema_editor)
File "/usr/local/lib/python2.7/dist-packages/django/db/migrations/migration.py", line 115, in apply
operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
File "/usr/local/lib/python2.7/dist-packages/django/db/migrations/operations/models.py", line 59, in database_forwards
schema_editor.create_model(model)
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/schema.py", line 286, in create_model
self.execute(sql, params or None)
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/schema.py", line 111, in execute
cursor.execute(sql, params)
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
File "/usr/local/lib/python2.7/dist-packages/django/db/utils.py", line 98, in exit
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/utils.py", line 62, in execute
return self.cursor.execute(sql)
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/sqlite3/base.py", line 316, in execute
return Database.Cursor.execute(self, query)
django.db.utils.OperationalError: table "django_content_type" already exists
root@a1f7bfe26094:/# PYTHONPATH=/opt/graphite/webapp python /usr/local/bin/django-admin.py migrate --settings=graphite.settings
Operations to perform:
Synchronize unmigrated apps: account, render, staticfiles, whitelist, metrics, url_shortener, dashboard, composer, events, browser
Apply all migrations: admin, contenttypes, tagging, auth, sessions
Synchronizing apps without migrations:
Creating tables...
Creating table dashboard_template
Running deferred SQL...
Installing custom SQL...
Running migrations:
Rendering model states... DONE
Applying contenttypes.0001_initial...Traceback (most recent call last):
File "/usr/local/bin/django-admin.py", line 5, in
management.execute_from_command_line()
File "/usr/local/lib/python2.7/dist-packages/django/core/management/init.py", line 354, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python2.7/dist-packages/django/core/management/init.py", line 346, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 394, in run_from_argv
self.execute(*args, **cmd_options)
File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 445, in execute
output = self.handle(*args, **options)
File "/usr/local/lib/python2.7/dist-packages/django/core/management/commands/migrate.py", line 222, in handle
executor.migrate(targets, plan, fake=fake, fake_initial=fake_initial)
File "/usr/local/lib/python2.7/dist-packages/django/db/migrations/executor.py", line 110, in migrate
self.apply_migration(states[migration], migration, fake=fake, fake_initial=fake_initial)
File "/usr/local/lib/python2.7/dist-packages/django/db/migrations/executor.py", line 148, in apply_migration
state = migration.apply(state, schema_editor)
File "/usr/local/lib/python2.7/dist-packages/django/db/migrations/migration.py", line 115, in apply
operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
File "/usr/local/lib/python2.7/dist-packages/django/db/migrations/operations/models.py", line 59, in database_forwards
schema_editor.create_model(model)
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/schema.py", line 286, in create_model
self.execute(sql, params or None)
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/schema.py", line 111, in execute
cursor.execute(sql, params)
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
File "/usr/local/lib/python2.7/dist-packages/django/db/utils.py", line 98, in exit
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/utils.py", line 62, in execute
return self.cursor.execute(sql)
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/sqlite3/base.py", line 316, in execute
return Database.Cursor.execute(self, query)
django.db.utils.OperationalError: table "django_content_type" already exists


my_graphite:
image: hopsoft/graphite-statsd
volumes:
- my_data/graphite/storage:/opt/graphite/storage

Steps to reproduce this migration problem:
1). Attach a empty host folder to hopsoft/graphite-statsd
2). "docker-compose up -d"
3). Add graphite datasource in Grafana. Save and test will show it's working.
4). "Docker-compose stop"
5). Replace image name to graphiteapp/graphite-statsd:1.0.2-3
6). "Docker-compose up -d"
7). Go to Grafana datasource, save and test. It will time out.
8). Following the steps to migration. And it throw out above errors.
docker exec -it fba2dc92dceb /bin/bash

root@fba2dc92dceb:~# PYTHONPATH=/opt/graphite/webapp python django-admin.py migrate auth --settings=graphite.settings

root@fba2dc92dceb:~# PYTHONPATH=/opt/graphite/webapp python /usr/local/bin/django-admin.py migrate --settings=graphite.settings

@codexiao
Copy link
Author

codexiao commented Dec 7, 2017

@deniszh Hi Denis, sorry to bug you. Any suggestion?

@deniszh
Copy link
Member

deniszh commented Dec 7, 2017

Try to add --fake-initial:

PYTHONPATH=/opt/graphite/webapp python /usr/local/bin/django-admin.py migrate auth --settings=graphite.settings --fake-initial
PYTHONPATH=/opt/graphite/webapp python /usr/local/bin/django-admin.py migrate --settings=graphite.settings --fake-initial

@codexiao
Copy link
Author

codexiao commented Dec 8, 2017

Thanks, @deniszh . Those two commands ran successfully. But save&test from graphite still timeout. Checking the ngnix says upstream is not available. Looks like graphiteapp/graphite-statsd:1.0.2-3 still can't initial itself correctly? Do you know where should I look at? Thanks a lot!

PYTHONPATH=/opt/graphite/webapp python /usr/local/bin/django-admin.py migrate auth --settings=graphite.settings --fake-initial
PYTHONPATH=/opt/graphite/webapp python /usr/local/bin/django-admin.py migrate --settings=graphite.settings --fake-initial

root@499a718c484c:/# PYTHONPATH=/opt/graphite/webapp python /usr/local/bin/django-admin.py migrate auth --settings=graphite.settings --fake-initial
Operations to perform:
Apply all migrations: auth
Running migrations:
Rendering model states... DONE
Applying contenttypes.0001_initial... FAKED
Applying contenttypes.0002_remove_content_type_name... OK
Applying auth.0001_initial... FAKED
Applying auth.0002_alter_permission_name_max_length... OK
Applying auth.0003_alter_user_email_max_length... OK
Applying auth.0004_alter_user_username_opts... OK
Applying auth.0005_alter_user_last_login_null... OK
Applying auth.0006_require_contenttypes_0002... OK

root@499a718c484c:/# PYTHONPATH=/opt/graphite/webapp python /usr/local/bin/django-admin.py migrate --settings=graphite.settings --fake-initial
Operations to perform:
Synchronize unmigrated apps: account, render, staticfiles, whitelist, metrics, url_shortener, dashboard, composer, events, browser
Apply all migrations: admin, contenttypes, tagging, auth, sessions
Synchronizing apps without migrations:
Creating tables...
Running deferred SQL...
Installing custom SQL...
Running migrations:
Rendering model states... DONE
Applying admin.0001_initial... FAKED
Applying sessions.0001_initial... FAKED
Applying tagging.0001_initial... FAKED
Applying tagging.0002_on_delete... OK
root@499a718c484c:/# exit

cat /var/log/nginx/error.log
2017/12/08 02:22:52 [error] 40#40: 1 upstream prematurely closed connection while reading response header from upstream, client: 172.18.0.4, server: , request: "GET /metrics/find?query= HTTP/1.1", upstream: "fastcgi://127.0.0.1:8080", host: "graphite", referrer: "http://localhost/datasources/edit/3"

@deniszh
Copy link
Member

deniszh commented Dec 8, 2017

That error is really strange. Are you sure that you connecting to the right instance? We're not using fastcgi, we're using proxying to gunicorn.
Check your Grafana's datasource config, looks like you're connecting to old IP or something.

What ps awx | grep python in container says?
Should be something like

root@6ecdfa3ac730:/# ps awx | grep python
    1 ?        Ss     0:00 /usr/bin/python3 -u /sbin/my_init
   27 ?        Sl     0:00 /usr/bin/python /opt/graphite/bin/carbon-cache.py start --debug
   30 ?        S      0:00 /usr/bin/python /usr/local/bin/gunicorn wsgi --workers=4 --bind=127.0.0.1:8080 --log-file=/var/log/gunicorn.log --preload --pythonpath=/opt/graphite/webapp/graphite
   32 ?        S      0:00 /usr/bin/python /opt/graphite/bin/carbon-aggregator.py start --debug
   70 ?        S      0:00 /usr/bin/python /usr/local/bin/gunicorn wsgi --workers=4 --bind=127.0.0.1:8080 --log-file=/var/log/gunicorn.log --preload --pythonpath=/opt/graphite/webapp/graphite
   71 ?        S      0:00 /usr/bin/python /usr/local/bin/gunicorn wsgi --workers=4 --bind=127.0.0.1:8080 --log-file=/var/log/gunicorn.log --preload --pythonpath=/opt/graphite/webapp/graphite
   72 ?        S      0:00 /usr/bin/python /usr/local/bin/gunicorn wsgi --workers=4 --bind=127.0.0.1:8080 --log-file=/var/log/gunicorn.log --preload --pythonpath=/opt/graphite/webapp/graphite
   73 ?        S      0:00 /usr/bin/python /usr/local/bin/gunicorn wsgi --workers=4 --bind=127.0.0.1:8080 --log-file=/var/log/gunicorn.log --preload --pythonpath=/opt/graphite/webapp/graphite
  106 pts/0    S+     0:00 grep --color=auto python

@codexiao
Copy link
Author

codexiao commented Dec 8, 2017

@deniszh yes, during the whole testing, the only change i made is the image name in docker-compose file, from hopsoft/graphite-statsd to graphiteapp/graphite-statsd:1.0.2-3. docker-compose down, make the changes, and then docker-compose up

I will check the ps aws output and report back.

Thanks a lot!

@deniszh
Copy link
Member

deniszh commented Dec 8, 2017

yes, during the whole testing, the only change i made is the image name in docker-compose file, from hopsoft/graphite-statsd to graphiteapp/graphite-statsd:1.0.2-3. docker-compose down, make the changes, and then docker-compose up

But our image doesn't contain Grafana, but according to error above it came from Grafana. Are you sure that Grafana connecting to correct instance?

@codexiao
Copy link
Author

codexiao commented Dec 8, 2017

oh, sorry for the confusion. @deniszh my docker-compose has two services, one is grafana for UI, the other is graphite. The only change i made is the graphite image.

graphite:
image: graphiteapp/graphite-statsd:1.0.2-3
volumes:
- mydata/graphite/storage:/opt/graphite/storage

grafana:
image: grafana/grafana


Steps to reproduce this migration problem:
1). Attach a empty host folder to hopsoft/graphite-statsd
2). "docker-compose up -d"
3). Add graphite datasource in Grafana. Save and test will show it's working.
4). "Docker-compose stop"
5). Replace image name to graphiteapp/graphite-statsd:1.0.2-3
6). "Docker-compose up -d"
7). Go to Grafana datasource, save and test. It will time out.
8). Following the steps to migration.
9). go back to Grafana UI, test the datasource again. Still timeout

@deniszh
Copy link
Member

deniszh commented Dec 8, 2017

Ah, OK. But fastcgi error still looks very suspicious. Could you please share your docker-compose config also? And ps output above too.
Thanks!

@codexiao
Copy link
Author

codexiao commented Dec 9, 2017

@deniszh yup. Thanks a lot for helping me.

Below is my docker-compose file, docker-compose config output, the ps output, and also the related log i found in /var/log/gunicorn.log

Like i mentioned above, I can reproduce this one every time. With my docker-compose file:

1). "docker-compose up -d"
2). Go to http://http://localhost, add graphite datasource in Grafana. Save and test will show it's working.
3). "Docker-compose stop"
4). Replace image name to graphiteapp/graphite-statsd:1.0.2-3
5). "Docker-compose up -d"
6). Go to Grafana datasource, save and test. It will stay in "Testing" and then timeout, throw out "HTTP Error Bad Gateway"
8). Following the steps to migration. Everything says OK.
9). go back to Grafana UI, test the datasource again. Still timeout


version: "3"

services:
graphite:
image: graphiteapp/graphite-statsd:1.0.2-3 #hopsoft/graphite-statsd #graphiteapp/graphite-statsd:1.0.2-3
volumes:
- /tmp/mydata/graphite/storage:/opt/graphite/storage

grafana:
image: grafana/grafana
ports:
- "80:3000"


services:
grafana:
image: grafana/grafana
ports:
- 80:3000/tcp
graphite:
image: graphiteapp/graphite-statsd:1.0.2-3
volumes:
- /tmp/mydata/graphite/storage:/opt/graphite/storage:rw
version: '3.0'

docker exec adb77da6061f ps awx |grep python
1 ? Ss 0:00 /usr/bin/python3 -u /sbin/my_init
28 ? S 0:00 /usr/bin/python /opt/graphite/bin/carbon-aggregator.py start --debug
29 ? S 0:00 /usr/bin/python /usr/local/bin/gunicorn wsgi --workers=4 --bind=127.0.0.1:8080 --log-file=/var/log/gunicorn.log --preload --pythonpath=/opt/graphite/webapp/graphite
66 ? S 0:00 /usr/bin/python /usr/local/bin/gunicorn wsgi --workers=4 --bind=127.0.0.1:8080 --log-file=/var/log/gunicorn.log --preload --pythonpath=/opt/graphite/webapp/graphite
67 ? S 0:00 /usr/bin/python /usr/local/bin/gunicorn wsgi --workers=4 --bind=127.0.0.1:8080 --log-file=/var/log/gunicorn.log --preload --pythonpath=/opt/graphite/webapp/graphite
68 ? S 0:00 /usr/bin/python /usr/local/bin/gunicorn wsgi --workers=4 --bind=127.0.0.1:8080 --log-file=/var/log/gunicorn.log --preload --pythonpath=/opt/graphite/webapp/graphite
281 ? S 0:00 /usr/bin/python /usr/local/bin/gunicorn wsgi --workers=4 --bind=127.0.0.1:8080 --log-file=/var/log/gunicorn.log --preload --pythonpath=/opt/graphite/webapp/graphite

cat /var/log/gunicorn.log
[2017-12-09 00:01:53 +0000] [31] [CRITICAL] WORKER TIMEOUT (pid:68)
[2017-12-09 00:01:53 +0000] [68] [INFO] Worker exiting (pid: 68)
[2017-12-09 00:01:53 +0000] [2186] [INFO] Booting worker with pid: 2186

@deniszh
Copy link
Member

deniszh commented Dec 11, 2017

OK, making a clear test.

  1. Creating an empty directory named test.
  2. Running Hopsoft docker on this directory:
docker run -d \
 --name graphite \
 --restart=always \
 -p 80:80 \
 -p 2003-2004:2003-2004 \
 -p 2023-2024:2023-2024 \
 -p 8125:8125/udp \
 -p 8126:8126 -v /Users/deniszh/test:/opt/graphite/storage hopsoft/graphite-statsd

You can go to http://localhost:80 and check that it's working. Graphite version is 0.9.15

  1. Stopping and removing the container. You can check that test directory now contains some files, including graphite.db and whisper directory.

  2. Running latest image on top of it:

docker run -d \
 --name graphite \
 --restart=always \
 -p 80:80 \
 -p 2003-2004:2003-2004 \
 -p 2023-2024:2023-2024 \
 -p 8125:8125/udp \
 -p 8126:8126 -v /Users/deniszh/test:/opt/graphite/storage graphiteapp/graphite-statsd
  1. Graphite is instantly working on http://localhost:80 - I didn't even run any migration command. Old files are also visible.
[deniszh:~/test] $ curl http://localhost/version/
1.1.0-dev

@deniszh
Copy link
Member

deniszh commented Dec 11, 2017

You can try to to save your whisper directory somewhere, start new graphite instance with empty storage directory and copy old whisper files on top of it - they should be visible.

@deniszh
Copy link
Member

deniszh commented Dec 11, 2017

Ah, now I understand what's going on. When you running docker-compose up with new image you're mounting only storage volume, but not config, so, it's picking it up from previous image.
This will not work - configs are quite different and not compatible. That's why you was getting wsgi errors - it was an old config for nginx running. So, solution is simple - mount /opt/graphite/conf volume for the new image on a separate empty directory. Or delete old container after stopping it.
I tested both ways - both are working.

@codexiao
Copy link
Author

codexiao commented Dec 11, 2017 via email

@deniszh
Copy link
Member

deniszh commented Dec 11, 2017

Well, it was also surprising to me, but after some thinking, it looks completely logical, looks like it works as intended. You can use docker inspect command to investigate. Mounted volume looks like

            {
                "Type": "bind",
                "Source": "/Users/deniszh/test/storage",
                "Destination": "/opt/graphite/storage",
                "Mode": "rw",
                "RW": true,
                "Propagation": "rprivate"
            },

but config volume for both containers looks like:

            {
                "Type": "volume",
                "Name": "5f78754194033fec82c4eb869e0979389074898fca547c5f517990e00928d8c0",
                "Source": "/var/lib/docker/volumes/5f78754194033fec82c4eb869e0979389074898fca547c5f517990e00928d8c0/_data",
                "Destination": "/opt/graphite/conf",
                "Driver": "local",
                "Mode": "",
                "RW": true,
                "Propagation": ""
            },

I found out that source ("/var/lib/docker/volumes/...") is the same for both containers.

@codexiao
Copy link
Author

@deniszh yes! it's working!!! Finally!!! Thanks a lot for your help. You are awesome!

@deniszh
Copy link
Member

deniszh commented Dec 12, 2017

Good, thanks for asking, that was educational also. :)

@deniszh deniszh closed this as completed Dec 12, 2017
@codexiao
Copy link
Author

codexiao commented Dec 13, 2017 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants