Site Freezing and Odd Behavior

[escorpius]

Crashed running the 7.10b1 release. Nothing in the ErrorLog. Thread was dead.

[Kent Briggs]

Crashed running the 7.10b1 release. Nothing in the ErrorLog. Thread was dead.

Just to be clear, you're checking for the .txt file in the data folder and not just looking at the Logs tab in the admin console, correct?

[escorpius]

That's correct. In this case, it was "...\Poker Mavens 7\Logs\ErrorLog2022-09-15.txt"

The file had lines 3-4 "invalid reconnect attempts" from earlier in the day and then the Unexpected Balance and Crash Recovery lines we see any time we have to force quit the server and restart it following a freeze.

[Kent Briggs]

That's correct. In this case, it was "...\Poker Mavens 7\Logs\ErrorLog2022-09-15.txt"

The only point in the code that would close the PM_Timer thread is manually clicking the Stop button in the admin console. Anything else is some kind of fatal crash of the server (as you are well aware) since the thread is enclosed in an exception catch block. No errors getting recorded gives me very little to go on. Have you seen any duplicate logins since running the beta? That may be a completely unrelated issue.

[escorpius]

No duplicate logins, whether actual or artifacts, but likely because we've implemented both automatic cleanup of residue connections when someone logs in and monitoring of the service that allows us to restart the service very quickly to avoid extended outages. Not great to crash and lose active hands, but better than then also being down for minutes at a time. However, an effect of both is to limit what we see when it does happen.

Any chance we could provide a second pair of eyes on the PM_Timer code? Do you have a code block that runs on thread exit, for example? If we had that, perhaps we could use it to dump state.

[Kent Briggs]

Any chance we could provide a second pair of eyes on the PM_Timer code? Do you have a code block that runs on thread exit, for example? If we had that, perhaps we could use it to dump state.

When you discover the PM_Timer thread stops or disappears, is any part of the software still responsive?

Here's pseudo code of the thread:

Code: Select all

while not(terminated) do
try
  send timer event callbacks
  record traffic report to event log
  start tournaments
  delete old logs
  kill expired player sessions
  calculate port stats
  update listing at pokermavens.net
except
  dump stack trace to error log
end

As I mentioned, the only point that sets terminated to true is when the Stop button is clicked. Anything else should cause an Error Log dump, unless the program is so frozen that it cannot do any file i/o.

I could create a beta version for you with code inserted between each command marking its execution but I'm not sure how to communicate that back to you.

[escorpius]

Thanks for sharing the pseudocode. Obviously a lot of details missing here, e.g., delays, error handling per step, etc., but interesting nonetheless. Unfortunately, much of the potential for failure lies in those details.

What is the "update listing at pokermavens.net" step? It's the only one I didn't readily recognize.

Loosely related question: The API call to "AccountsSessionKey" is only logged when a call is made to "AccountsSessionKey" via the API, correct? I know it probably seems like an odd question, but I'm seeing multiple log entries even when our site should only be making that API call once. So, curious if there are any other situations where that log entry could show up, e.g., chaining functions. Perhaps completely unrelated, but wanting to better understand the inner-workings of the server.

Having a beta version with a log entry between literally each command of the function would be interesting. I would also be interested in putting in a "finally" block that logs that it is entering the finally block and then also dumping the stack trace.

Not clear on the "communicate that back to you" comment. For the information, I would think just logging to the EventLog would be sufficient. If it's about actually getting the beta to us, then we would follow the same approach as before.

Let me know how we can help.

[Kent Briggs]

What is the "update listing at pokermavens.net" step? It's the only one I didn't readily recognize.

If you have "Directory listing" turned on, it keeps the login/table count numbers updated for your listing at pokermavens.net every 3 minutes.

Loosely related question: The API call to "AccountsSessionKey" is only logged when a call is made to "AccountsSessionKey" via the API, correct?

That's correct. I'd suggest you look at your API call and make your own log every time you call AccountsSessionKey. If you see multiple entries in your own log then there's an issue on your end.

Not clear on the "communicate that back to you" comment. For the information, I would think just logging to the EventLog would be sufficient.

I can do that, but it will make a lot of entries. A half dozen or so every second. Probably better to throw them in the error log.

[escorpius]

Given that we parse the EventLog in realtime, I definitely prefer dumping to the ErrorLog to keep our queue traffic at normal levels. That means approximately 1,000,000 entries in a 24-hour window ... not terrible. Ready to give it a test when you are.

[Kent Briggs]

Ready to give it a test when you are.

Link sent via email.