Speed up PostGIS database with with synchronous_commit=off and more#94
Draft
anthonyfok wants to merge 1 commit intoOpenDRR:masterfrom
Draft
Speed up PostGIS database with with synchronous_commit=off and more#94anthonyfok wants to merge 1 commit intoOpenDRR:masterfrom
anthonyfok wants to merge 1 commit intoOpenDRR:masterfrom
Conversation
Member
Author
|
Some rudimentary benchmarks: TODO Beforeboundaries populated with database dump:with fsync=off, /var/lib/docker/volumes on NFSwith fsync=off sychronous_commit full_page_writes=off; /var/lib/docker/volumes on NFS |
Contributor
|
So with fsync off it's 51 minutes faster? |
Member
Author
|
Hi @jvanulde, thanks for looking into this... ...even though this PR as well as the "benchmarks" are at a very premature stage. I am using this space sort of as a "brain-dump" for convenience sake, even though the results aren't ready. The tests that I have made thus far are actually very sloppy 😅, as in:
I was expecting to see 10× difference in speed as some reported, but I am not seeing that, getting pretty confused by the sloppy logs. For example: Before, without fsync=off (supposedly): 1m14s After, with fsync=off (supposedly): 1m28s After, with fsync=off synchronous_commit=off full_page_writes=off (supposedly): 1m9s So, very inconclusive, i.e. junk benchmark results. Sorry! I'd better double check to see this PR is doing anything by using a special benchmark script like the one at https://pythonspeed.com/articles/faster-db-tests/ ... Sorry again! |
Member
Author
|
Good news: Setting Using an adopted test https://pythonspeed.com/articles/faster-db-tests/ (see the test-pgsql-speed branch, running while modifying Real-life benchmark with actual OpenDRR data will have to wait. 😅 <grin, duck, run> |
Member
Author
|
New idea: Use add_data.sh to change the three settings (fsync=off, etc.) instead? I realized just today that these settings (fsync, synchronous_commit and full_page_writes) can be modified on-the-fly with, say, Maybe References:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #77
Not quite ready yet:
ps auxwwwdoes shows apostgres -c fsync=off -c full_page_writes=off -c synchronous_commit=offprocess, but do need to verify if the command-line options actually reach the PostgreSQL server