Disconnected Sessions stay active

[mantux5]

I think I found the problem why my server has been crashing recently. This time we managed to control the situation and the server is still running, but need a restart, otherwise it will crash after a few hours. The issue is that at some point (Probably something wrong happens, but I'm not sure what) connections are stuck at "Discon" state. And they just stack on each other, without disappearing. The API calls state there are like 150 players online, but the client shows 500+ We have reached ~750+ and still didn't crash. After comparing the results, I see that the client actually is showing the Connections, not the Logins themselves.

I should mention, that all actions are taken via Remote Admin, because the Server Program is not responding. It's even impossible to shut it down with the red X button. I have to kill it using Task Manager. However, if we manage to Terminate such connections manually, the server does not crash. We succeeded in doing it for over 3 hours now. However, even when all the connections are terminated, the Server Program still does not respond and the "Discon" connections are still appearing and never disappearing.

My thoughts are that something happens in the part of program which is responsible for handling such connections and it just stops working. I'm not even sure how the whole program works in this section, but still trying to make logical assumptions. If you could, please make as much debugging as possible in this part of the software and send that modified version for us, to test it out and finally find a solution to this problem.

Best regards,
Mantas S.

[mantux5]

One more thing I noticed:
Some of the Connections have no value in the "Player" field. I don't know if this is Ok or not.

[Kent Briggs]

The blank Player field shows that a connection was made but the player never completed the login. Those connections are still using up a thread and could cause your server to run out of system resources, which would cause the non-responsive behavior of the Windows console. You should run TCPView or similar program and see how many active connections there really are on your packet port. Do you have a value set for Maximum PP Connections in the settings?

[mantux5]

Yes, I have the Maximum PP and FP Connections set. The issue is not that I'm reaching the limit, but that tons of connections are just stuck at the state of "Discon" and not going anywhere. I have to manually Terminate them. Also, these connections with no player name are really rare, so they are not the main problem here.

[mantux5]

Also, if this would cause the Server Program to freeze, after Terminating the connections I should be able to use it again, but it's in the "Not Responding" state since the connections started to grow without disappearing automatically.

[mantux5]

One more thing I noticed - The UpTime is not increasing. But the UTC and Local times are correct.

[Kent Briggs]

The issue is not that I'm reaching the limit, but that tons of connections are just stuck at the state of "Discon" and not going anywhere.

Poker Mavens uses a system timer that goes off every 60 seconds and terminates any disconnects that are 2 minutes old. If the termination fails, you should see "KillInactiveSessions error" messages in the error log. If you are not, my guess is that the timer stopped working because the system is in an unstable state for some reason.

[Kent Briggs]

One more thing I noticed - The UpTime is not increasing. But the UTC and Local times are correct.

The UpTime displayed at the bottom of the console window gets updated by a different system timer. That timer also starts timed tournaments and should fire every 15 seconds. So if you have any tournaments with a set Start Time, they probably aren't working either. In any case, the poker server needs to be shut down and restarted. Those disconnects are more likely just a symptom of the problem, not the actual cause. Are you running a Perfmon log to capture memory, thread, and CPU usage?

[mantux5]

Alright, I'm going to write a few suggestions here, because I feel that some of the things you are doing are wrong.

First of all, you should calculate UpTime using the Start and Current times. This would make it show the correct value all the time. Next, there should be some sort of process to check for other processes status. And if one or more processes are failing, they should be killed and new ones made. This would fix the issue of the "Session terminator" getting stuck. Well, it would fix most of the issues my team is having right now. I'm almost 100% sure that the "Session terminator" process is getting stuck because of some not considered and unexpected data from the user or the server itself. That's why some debugging tools should be implemented, to make the problem come out easier. Thinking of workaround, such as limiting connections and such won't fix the problem. The cause of problem should be found and eliminated. This is how I do things.

Also, regarding the System timer, which is working with Uptime and Tournaments, I have experienced the problem of the tournaments not starting at the time specified a long time before. Even when system gets back up and start counting again, it does not start the tournament even if time has passed. I think we should be allowed to at least have an option to allow it to be started at this situation. Well, I won't need it now, because I have made a CRON task to start the tournaments a minute after they should be started, so if the server itself did not start them, they are started via API call. This works quite well.

Another thing that interests me, how do the PerfMon logs help you to determine the problem? Yes, you can see an increases usage of resources. But you can't see any reason why it is like that. As I mentioned before, you should add some debugging tools for the software. Restarting server is not an option. Because it crashes at least once a week. If there is no one to restart it, we are simply losing money with that. Even if there is someone to restart the server, we are still losing money and clients. People consider website unreliable because it crashes constantly. Also, it takes at least 40 minutes to restart the server (This may not apply to everyone, because everyone have different amount of users). We have over 76 000 registered users, so it takes a ton of time to load all the data files. Also, all the logs older than a month are being saved, so we could safely retrieve them via API, without any need to create any external system to store them. You should consider implementing a different data handling system, because it takes tons of time to load and I can't find any advantages of it (Well, I might be wrong, because I'm not sure about the exact technology you are using). Either way the issue stays - long loading time. We are growing quite fast, so it will soon take 1 hour or even more just to load all the data. During that hour we could earn a hundred or two.

Thank you for your time and efforts releasing the previous updates, but I expect what I wrote here will be considered and at least some of it will be implemented, even though not that many new purchases of the software are made (I might be wrong about that too. This is just my assumption due to low activity on the forums) so this might make you less motivated to do big changes to the software.

Anyway, thank you for the help you gave us before.

[Kent Briggs]

First of all, you should calculate UpTime using the Start and Current times.

That misses the point of what has happened. The server has (likely) run out of system resources, which killed the system timers, and is now in an unstable state.

Even when system gets back up and start counting again, it does not start the tournament even if time has passed.

Once a full minute has passed, the tournament is considered expired and won't start automatically. You could have been down for hours and the players all left. It wouldn't be fair to start a tournament and accept those buy-ins at that point.

Another thing that interests me, how do the PerfMon logs help you to determine the problem?

It shows if the problems are occurring at maximum load, indicating that you've overloaded your server with players, rather than it being some random bug or hacking attempt.

Also, it takes at least 40 minutes to restart the server (This may not apply to everyone, because everyone have different amount of users). We have over 76 000 registered users, so it takes a ton of time to load all the data files.

That issue will be addressed in version 5 when I convert everything to databases. In the mean time, I'd suggest you keep your own user database of players and then you can create/delete accounts on the fly as players log into and out of your web site. Otherwise Windows has to load 76K individual files into memory at startup. I suspect 90% of them are probably long abandoned anyway, right?