Skip to content

[K8sDnsNameResolver] Fix Netty panic logging on JVM shutdown#8

Merged
migesok merged 1 commit intomainfrom
bugfix/k8s-resolver-panic-on-shutdown
Mar 5, 2026
Merged

[K8sDnsNameResolver] Fix Netty panic logging on JVM shutdown#8
migesok merged 1 commit intomainfrom
bugfix/k8s-resolver-panic-on-shutdown

Conversation

@migesok
Copy link
Collaborator

@migesok migesok commented Mar 4, 2026

Removes this erroneous log message on JVM shutdown:

io.grpc.StatusRuntimeException: INTERNAL: Panic! This is a bug!

Why it was logged:

  • it seems to happen when LookupSession.lookupAsync tries to establish a new DNS server connection after the JVM shutdown has already started (it could happen because JVM shutdown hooks are executed in unpredictable order)
  • under the hood that function tries to register a JVM shutdown hook to clean up NIO resources registering a shutdown hook during the shutdown sequence results in java.lang.IllegalStateException: Shutdown in progress
  • this exception is not reported as a failure case in LookupSession.lookupAsync result completion stage but is thrown directly on the method call
  • the code inside the resolver doesn't handle such direct exception gracefully, which causes it to propagate to Netty internals, resulting in a scary "Panic! This is a bug!" log message written to JUL by Netty internals

The fix:

  • catch and handle the "error on JVM shutdown" case
  • reduce the log level for such errors below INFO
  • additionally, the resolver shutdown logic was hardened for all the cases where currently running resolution is in a race with the shutdown

@migesok migesok self-assigned this Mar 4, 2026
Removes this erroneous log message on JVM shutdown:
io.grpc.StatusRuntimeException: INTERNAL: Panic! This is a bug!

Why it was logged:
- it seems to happen when LookupSession.lookupAsync tries to establish a new DNS server connection after the JVM shutdown has already started (it could happen because JVM shutdown hooks are executed in unpredictable order)
- under the hood that function tries to register a JVM shutdown hook to clean up NIO resources
registering a shutdown hook during the shutdown sequence results in java.lang.IllegalStateException: Shutdown in progress
- this exception is not reported as a failure case in LookupSession.lookupAsync result completion stage but is thrown directly on the method call
- the code inside the resolver doesn't handle such direct exception gracefully, which causes it to propagate to Netty internals, resulting in a scary "Panic! This is a bug!" log message written to JUL by Netty internals

The fix:
- catch and handle the "error on JVM shutdown" case
- reduce the log level for such errors below INFO
- additionally, the resolver shutdown logic was hardened for all the cases where currently running resolution is in a race with the shutdown
@migesok migesok force-pushed the bugfix/k8s-resolver-panic-on-shutdown branch from 605141e to 024b1ca Compare March 5, 2026 11:58
Copy link

@fillson-shady fillson-shady left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice fix, thank you!

@migesok migesok merged commit faf1233 into main Mar 5, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants