C#: Automatic reconnection of UdpClient after server restart?

Question

.NET

I am writing a service application that requests status data from the server via UDP and displays alerts if necessary.
The problem is that the server lives its own life. It may or may not be running when the application starts. Or it may even crash / restart / etc, and the service needs to wait and periodically check if the server has risen and whether it is possible to connect again.

I implemented it this way.

Lots of code

public Server(string name, string host, int? queryPort = null, int? rConPort = null)
        {
            Name = name;
            Host = Dns.GetHostAddresses(host)[0];
            QueryPort = queryPort;
            RConPort = rConPort;

            if (QueryPort == null && RConPort == null)
            {
                throw new IncorrectServerEntryPoint(Host);
            }
        }
        
        public async void Watch()
        {
            if (QueryPort == null) return;
            _statusWatcherClient = new UdpClient(Host.ToString(), QueryPort.Value);
            
            UpdateChallengeTokenTimer = new Timer(async obj =>
            {
                Console.WriteLine($"[INFO] [{Name}] Send handshake request");
                    
                Request handshakeRequest = Request.GetHandshakeRequest();
                byte[] response = null;
                try
                {
                    response = await SendResponseService.SendReceive(_statusWatcherClient, handshakeRequest.Data, ReceiveAwaitIntervalSeconds);
                    IsOnline = true;
                }
                catch (SocketException)
                {
                    WaitForServerAlive(QueryPort.Value);
                }
                    
                if (response == null) return;
                
                var challengeTokenRaw = Response.ParseHandshake(response);
                lock (_challengeTokenLock)
                {
                    SetChallengeToken(challengeTokenRaw);
                }
                    
                Console.WriteLine($"[INFO] [{Name}] ChallengeToken is set up: " + BitConverter.ToString(challengeTokenRaw));
            }, null, 0, GettingChallengeTokenInterval);
                
            UpdateServerStatusTimer = new Timer(async obj =>
            {
                Console.WriteLine($"[INFO] [{Name}] Send full status request");
                    
                var challengeToken = new byte[4];
                lock (_challengeTokenLock)
                {
                    Buffer.BlockCopy(_challengeToken, 0, challengeToken, 0, 4);
                }
                    
                var fullStatusRequest = Request.GetFullStatusRequest(challengeToken);

                byte[] response = null;
                try
                {
                    response = await SendResponseService.SendReceive(_statusWatcherClient, fullStatusRequest.Data, ReceiveAwaitIntervalSeconds);
                    IsOnline = true;
                }
                
                catch (SocketException)
                {
                    WaitForServerAlive(QueryPort.Value);
                }
                
                if (response == null) return;

                ServerFullState fullState = Response.ParseFullState(response);
                    
                Console.WriteLine($"[INFO] [{Name}] Full status is received");
                    
                OnFullStatusUpdated?.Invoke(this, new ServerStateEventArgs(Name, fullState));
                    
            }, null, 500, GettingStatusInterval);
        }

        public async Task Unwatch()
        {
            await UpdateChallengeTokenTimer.DisposeAsync();
            await UpdateServerStatusTimer.DisposeAsync();
            _statusWatcherClient.Dispose();
            _statusWatcherClient = null;
        }

        public async void WaitForServerAlive(int port)
        {
            Console.WriteLine($"[WARNING] [{Name}] Server is unavailable. Waiting for reconnection...");
            IsOnline = false;
            await Unwatch();
            Timer waitTimer = null;
            waitTimer = new Timer(async obj => {
                try
                {
                    using TcpClient tcpClient = new TcpClient();
                    await tcpClient.ConnectAsync(Host, port);
                    if (waitTimer == null) return;
                    await waitTimer.DisposeAsync();
                    Console.WriteLine($"[INFO] [{Name}] Server is available again");
                    Watch();
                }  catch (SocketException) { }
            }, null, 500, 5000);
        }

        public static int ReceiveAwaitIntervalSeconds = 10;
        public static int GettingChallengeTokenInterval = 30000;
        public static int GettingStatusInterval = 5000;

Here we instantiate the server and start watching it with Watch(), which does the job of keeping the request tokens up to date (every 30 seconds) and querying the server status every 5 seconds.
If the connection is interrupted or not available from the very beginning, WaitForServerAlive() comes into play, which stops the server request timers, destroys the old socket and starts knocking on the server port every 5 seconds in an attempt to get a connection. If the connection goes through, it resumes monitoring the server by calling Watch().

The code for sending and receiving packets over UDP looks like this

public static async Task<byte[]> SendReceive(UdpClient client, byte[] data, int receiveAwaitIntervalSeconds)
        {
            if (client == null)
            {
                throw new NullReferenceException("UdpClient client is null");
            }
            
            IPEndPoint ipEndPoint = null;
            byte[] response = null;
            
            await client.SendAsync(data, data.Length);
            var responseToken = client.BeginReceive(null, null);
            responseToken.AsyncWaitHandle.WaitOne(TimeSpan.FromSeconds(receiveAwaitIntervalSeconds));
            if (responseToken.IsCompleted)
            {
                try
                {
                    response = client.EndReceive(responseToken, ref ipEndPoint);
                }

                catch (Exception)
                {
                    // can't end receive
                }
            }

            if (response == null)
                throw new SocketException();

            return response;
        }

Actually, while the server is running stably, everything goes smoothly, but as soon as the service starts before the server starts or the server crashes in the middle of work, as at the moment the server "returns" the application starts to issue something like

Work logs

[INFO] [ML_VDS] Send handshake request
[WARNING] [ML_VDS] Server is unavailable. Waiting for reconnection...
[INFO] [ML_VDS] Server is available again
[INFO] [ML_VDS] Send handshake request
[WARNING] [ML_VDS] Server is unavailable. Waiting for reconnection...
[INFO] [ML_VDS] Server is available again
...
[INFO] [ML_VDS] Send handshake request
[WARNING] [ML_VDS] Server is unavailable. Waiting for reconnection...
[INFO] [ML_VDS] Server is available again
[INFO] [ML_VDS] Send handshake request
[INFO] [ML_VDS] Send full status request
[WARNING] [ML_VDS] Server is unavailable. Waiting for reconnection...
[WARNING] [ML_VDS] Server is unavailable. Waiting for reconnection...
[INFO] [ML_VDS] Server is available again
[INFO] [ML_VDS] Send handshake request
[INFO] [ML_VDS] Server is available again
[INFO] [ML_VDS] Send handshake request
[ML_VDS] Server is online
[INFO] [ML_VDS] ChallengeToken is set up: 00-93-95-1E
[INFO] [ML_VDS] ChallengeToken is set up: 00-98-D8-8B
[INFO] [ML_VDS] Send full status request
[INFO] [ML_VDS] Send full status request
[INFO] [ML_VDS] Send full status request
[WARNING] [ML_VDS] Server is unavailable. Waiting for reconnection...
[ML_VDS] Server is offline
[WARNING] [ML_VDS] Server is unavailable. Waiting for reconnection...
[INFO] [ML_VDS] Send full status request
[WARNING] [ML_VDS] Server is unavailable. Waiting for reconnection...
[WARNING] [ML_VDS] Server is unavailable. Waiting for reconnection...
[WARNING] [ML_VDS] Server is unavailable. Waiting for reconnection...
Unhandled exception. [WARNING] [ML_VDS] Server is unavailable. Waiting for reconnection...
Unhandled exception. Unhandled exception. Unhandled exception. Unhandled exception. System.NullReferenceException: Object reference not set to an instance of an object.
   at MCServerNotifier.Server.Unwatch() in /Users/maxlevs/RiderProjects/MCServerNotifier/MCServerNotifier/Server.cs:line 142
   at MCServerNotifier.Server.WaitForServerAlive(Int32 port) in /Users/maxlevs/RiderProjects/MCServerNotifier/MCServerNotifier/Server.cs:line 150
   at System.Threading.Tasks.Task.<>c.<ThrowAsync>b__140_1(Object state)
   ...
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
System.NullReferenceException: Object reference not set to an instance of an object.
   at MCServerNotifier.Server.Unwatch() in  
   ...
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
System.NullReferenceException: Object reference not set to an instance of an object.
   at MCServerNotifier.Server.Unwatch() in  
   ...
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
System.NullReferenceException: Object reference not set to an instance of an object.
   at MCServerNotifier.Server.Unwatch() in 
   ...
   at System.Threading.ThreadPoolWorkQueue.Dispatch()

System.NullReferenceException: Object reference not set to an instance of an object.
   at MCServerNotifier.Server.Unwatch() in 
   ...
   at System.Threading.ThreadPoolWorkQueue.Dispatch()

I ask knowledgeable people to help understand the cause of the problem. What is the correct approach to trying to reconnect after a disconnect?

Answer 1

Resolved the issue of refusing to use TCP connection as a test of server operation. (Who can tell if such a replacement could be viable at all?).
Replaced the WaitForServerAlive() code with the following:

Code here

public async void WaitForServerAlive()
        {
            if(Debug)
                Console.WriteLine($"[WARNING] [{ServerName}] Server is unavailable. Waiting for reconnection...");
            
            IsOnline = false;
            await Unwatch();
            
            _mcQuery.InitSocket();
            
            Timer waitTimer = null;
            waitTimer = new Timer(async obj => {
                try
                {
                    await _mcQuery.GetHandshake();
                    
                    IsOnline = true;
                    Watch();
                    lock (_retryCounterLock)
                    {
                        RetryCounter = 0;
                    }
                    
                    waitTimer.Dispose();
                }
                catch (SocketException)
                {
                    if(Debug)
                        Console.WriteLine($"[WARNING] [{ServerName}] [WaitForServerAlive] Server doesn't response. Try to reconnect: {RetryCounter}");
                    
                    lock (_retryCounterLock)
                    {
                        RetryCounter++;
                        if (RetryCounter >= RetryMaxCount)
                        {
                            if(Debug)
                                Console.WriteLine($"[WARNING] [{ServerName}] [WaitForServerAlive] Recreate socket");
                            
                            RetryCounter = 0;
                            _mcQuery.InitSocket();
                        }
                    }
                }
            }, null, 500, 5000);
        }

Here, every 5 seconds we knock on the server, requesting a new token. 5 attempts are given to receive the token, after which the socket is recreated to avoid situations when the server is up, but the data still does not want to go through the socket. Any successfully transmitted packet resets the counter. Actually, WaitForServerAlive() itself is called if an unsuccessful attempt to receive data from the server has occurred 5 times in a row, this puts the application into the connection recovery waiting mode.
The higher protocol implies obtaining a token through a handshake request, which always returns a result with a correctly composed request. If the server is up, then we will get a token. Usually it is enough to receive it once every 30 seconds for requests to pass. In order to quickly find out that the server is up again, I cancel the refresh of the token and call a similar code every 5 seconds until I receive data from the server.
After that, we can assume that the server is alive again and return to regular work.