[K8sDnsNameResolver] Fix Netty panic logging on JVM shutdown#8
Merged
Conversation
Removes this erroneous log message on JVM shutdown: io.grpc.StatusRuntimeException: INTERNAL: Panic! This is a bug! Why it was logged: - it seems to happen when LookupSession.lookupAsync tries to establish a new DNS server connection after the JVM shutdown has already started (it could happen because JVM shutdown hooks are executed in unpredictable order) - under the hood that function tries to register a JVM shutdown hook to clean up NIO resources registering a shutdown hook during the shutdown sequence results in java.lang.IllegalStateException: Shutdown in progress - this exception is not reported as a failure case in LookupSession.lookupAsync result completion stage but is thrown directly on the method call - the code inside the resolver doesn't handle such direct exception gracefully, which causes it to propagate to Netty internals, resulting in a scary "Panic! This is a bug!" log message written to JUL by Netty internals The fix: - catch and handle the "error on JVM shutdown" case - reduce the log level for such errors below INFO - additionally, the resolver shutdown logic was hardened for all the cases where currently running resolution is in a race with the shutdown
605141e to
024b1ca
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Removes this erroneous log message on JVM shutdown:
Why it was logged:
The fix: