rpc: Ignore and log errors during cancel

Cancelling the RPC acceptors can sometimes result in an error about
a bad file descriptor.

As this is the shutdown sequence we need to continue nevertheless,
ignore these errors, log a warning and proceed.

Fixes #4352.