[ZODB] Help with ZEO performance

[Colin Garriga-Salaün]

Hello list.

I have a 250Go ZEO5 database and ~140 clients connected to it.
~120 clients are uwsgi workers reading data.
~20 clients are daemon-workers handling write operations.

On heavy write operations, uwsgi workers are stalling.

Would you have some recommandations for me ?

I am very surprised to see that my server is handling all those
connections through only one process. Is it a normal behaviour ?

Help :)

[Colin Garriga-Salaün]

Sorry I have to correct my own statement : uwsgi are mostly reading but
also handling small writings. So that connecting them to a replicating
database won't do the job.

Post by Colin Garriga-Salaün
Hello list.
I have a 250Go ZEO5 database and ~140 clients connected to it.
~120 clients are uwsgi workers reading data.
~20 clients are daemon-workers handling write operations.
On heavy write operations, uwsgi workers are stalling.
Would you have some recommandations for me ?
I am very surprised to see that my server is handling all those
connections through only one process. Is it a normal behaviour ?
Help :)

[Jim Fulton]

Post by Colin Garriga-Salaün
Hello list.
I have a 250Go ZEO5 database and ~140 clients connected to it.
~120 clients are uwsgi workers reading data.
~20 clients are daemon-workers handling write operations.
On heavy write operations, uwsgi workers are stalling.
Would you have some recommandations for me ?
I am very surprised to see that my server is handling all those
connections through only one process. Is it a normal behaviour ?

That's a lot of connections. At ZC (Zope Corporation) I think we got up to
~80 connections.

What version of ZEO are you using?

Jim

Post by Colin Garriga-Salaün
Help :)
--
You received this message because you are subscribed to the Google Groups "zodb" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/d/optout.

[Colin Garriga-Salaün]

ZEO 5.2.0 with Python2.7

That's a lot of connections.  At ZC (Zope Corporation) I think we got
up to ~80 connections.
What version of ZEO are you using?

[Jim Fulton]

Post by Colin Garriga-Salaün
ZEO 5.2.0 with Python2.7

So some cheap things to try (not necessarily in order):

- Run the server on PyPy.
See the graphs at the end of:
https://github.com/jimfulton/byteserver/wiki/performance-2016-10-26

I'm not sure this will help you, given that writing seems to be slower,
but it might be worth a try.

- Upgrade to Python 3 (if practical).
- I was hoping you could use a Python 3 storage server with Python 2
clients if you configured it to use msgpack, but alas, this is not the
case. I'm working on trying to make this possible.

- Use biggish persistent ZEO caches, if you aren't already.
- Maybe use bigger object caches.
- Maybe a faster storage server machine.
- Double check that you aren't saturating network communication to the
storage server. That was something we had to deal with at ZC.

Some more involved things to try:

- If you can arrange for read-only servers (maybe with clever
load-balancer rules), then use ZRS secondaries for these. You've already
dismissed this, but maybe think harder about making it work.
- Maybe use fewer clients. For example, if you're CPU bound and using a
cloud provider, maybe use faster machines.
- If you're not CPU bound, then maybe use fewer clients and more
threads per client.
- Maybe try a ZEO fan-out configuration, where you have multiple ZEO
servers that wrap ZEO clients of a single ZEO server. I'm not aware of
anyone who's run this in production. :)

There are probably other options depending on your application. :)

Jim

Post by Colin Garriga-Salaün

Post by Jim Fulton
That's a lot of connections. At ZC (Zope Corporation) I think we got
up to ~80 connections.
What version of ZEO are you using?

--
You received this message because you are subscribed to the Google Groups "zodb" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/d/optout.

[Thierry Florac]

Hi Jim,
After a small email discussion with you (this was a long long time ago...),
I'm using a fan-out configuration in production since more than ten years!
Because of a slow network connection between the main ZEO server which is
storing our CMS contents, and our management servers which are hosted in
our local network, this kind of configuration was really helpful to keep
good performances!!
I've written a small topic on this configuration (
http://www.ztfy.org/articles-et-tutoriels/2010/09/installation-dun-serveur-zeo-en-mode.html
).
Best regards,
Thierry

Post by Jim Fulton

Post by Colin Garriga-Salaün
ZEO 5.2.0 with Python2.7

- Run the server on PyPy.
https://github.com/jimfulton/byteserver/wiki/performance-2016-10-26
I'm not sure this will help you, given that writing seems to be
slower, but it might be worth a try.
- Upgrade to Python 3 (if practical).
- I was hoping you could use a Python 3 storage server with Python
2 clients if you configured it to use msgpack, but alas, this is not the
case. I'm working on trying to make this possible.
- Use biggish persistent ZEO caches, if you aren't already.
- Maybe use bigger object caches.
- Maybe a faster storage server machine.
- Double check that you aren't saturating network communication to the
storage server. That was something we had to deal with at ZC.
- If you can arrange for read-only servers (maybe with clever
load-balancer rules), then use ZRS secondaries for these. You've already
dismissed this, but maybe think harder about making it work.
- Maybe use fewer clients. For example, if you're CPU bound and using
a cloud provider, maybe use faster machines.
- If you're not CPU bound, then maybe use fewer clients and more
threads per client.
- Maybe try a ZEO fan-out configuration, where you have multiple ZEO
servers that wrap ZEO clients of a single ZEO server. I'm not aware of
anyone who's run this in production. :)
There are probably other options depending on your application. :)
Jim

Post by Colin Garriga-Salaün

Post by Jim Fulton
That's a lot of connections. At ZC (Zope Corporation) I think we got
up to ~80 connections.
What version of ZEO are you using?

--
You received this message because you are subscribed to the Google Groups "zodb" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/d/optout.

--
Jim Fulton
http://jimfulton.info
--
You received this message because you are subscribed to the Google Groups "zodb" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/d/optout.

...

[Jim Fulton]

Oh nice!

Jim

Post by Thierry Florac
Hi Jim,
After a small email discussion with you (this was a long long time
ago...), I'm using a fan-out configuration in production since more than
ten years!
Because of a slow network connection between the main ZEO server which is
storing our CMS contents, and our management servers which are hosted in
our local network, this kind of configuration was really helpful to keep
good performances!!
I've written a small topic on this configuration (
http://www.ztfy.org/articles-et-tutoriels/2010/09/installation-dun-serveur-zeo-en-mode.html
).
Best regards,
Thierry

Post by Jim Fulton

Post by Colin Garriga-Salaün
ZEO 5.2.0 with Python2.7

- Run the server on PyPy.
https://github.com/jimfulton/byteserver/wiki/performance-2016-10-26
I'm not sure this will help you, given that writing seems to be
slower, but it might be worth a try.
- Upgrade to Python 3 (if practical).
- I was hoping you could use a Python 3 storage server with Python
2 clients if you configured it to use msgpack, but alas, this is not the
case. I'm working on trying to make this possible.
- Use biggish persistent ZEO caches, if you aren't already.
- Maybe use bigger object caches.
- Maybe a faster storage server machine.
- Double check that you aren't saturating network communication to
the storage server. That was something we had to deal with at ZC.
- If you can arrange for read-only servers (maybe with clever
load-balancer rules), then use ZRS secondaries for these. You've already
dismissed this, but maybe think harder about making it work.
- Maybe use fewer clients. For example, if you're CPU bound and using
a cloud provider, maybe use faster machines.
- If you're not CPU bound, then maybe use fewer clients and more
threads per client.
- Maybe try a ZEO fan-out configuration, where you have multiple ZEO
servers that wrap ZEO clients of a single ZEO server. I'm not aware of
anyone who's run this in production. :)
There are probably other options depending on your application. :)
Jim

Post by Colin Garriga-Salaün

Post by Jim Fulton
That's a lot of connections. At ZC (Zope Corporation) I think we got
up to ~80 connections.
What version of ZEO are you using?

--
You received this message because you are subscribed to the Google Groups "zodb" group.
To unsubscribe from this group and stop receiving emails from it, send
For more options, visit https://groups.google.com/d/optout.

--
Jim Fulton
http://jimfulton.info
--
You received this message because you are subscribed to the Google Groups "zodb" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/d/optout.

--
http://www.imagesdusport.com -- http://pyams.readthedocs.io
--
You received this message because you are subscribed to the Google Groups "zodb" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/d/optout.

...