Site Freezing and Odd Behavior

Report bugs found in Poker Mavens
Post Reply
Kent Briggs
Site Admin
Posts: 5878
Joined: Wed Mar 19, 2008 8:47 pm

Re: Site Freezing and Odd Behavior

Post by Kent Briggs »

escorpius wrote: Fri Aug 19, 2022 12:25 am We also make extensive use of the API and parse the logs in realtime.
Is this something you could pause temporarily just to test if it's a factor or not?
BlackKnite69
Posts: 13
Joined: Mon Mar 16, 2020 2:19 pm

Re: Site Freezing and Odd Behavior

Post by BlackKnite69 »

It has frozen 3 times in 2 days. It froze during the tourney and we could not restore it after reboot. We presently have no game running. We will be without player very soon if this continues. Escorpius, runs the tech stuff for the site and can answer technical questions.

We always get duplicate logins showing in lobby before the crash. It may be for hours or minutes before. There are dead connections showing up in admin page. Oh, the duplicate logins on lobby do not show up in admin page or logs.

We are having it lock up with as few as 4 players now.
escorpius
Posts: 64
Joined: Wed Apr 29, 2020 12:47 am

Re: Site Freezing and Odd Behavior

Post by escorpius »

Kent Briggs wrote: Fri Aug 19, 2022 9:22 am
escorpius wrote: Fri Aug 19, 2022 12:25 am We also make extensive use of the API and parse the logs in realtime.
Is this something you could pause temporarily just to test if it's a factor or not?
In theory, yes. However, this functionality powers a lot of the features on the site, e.g., realtime awards for various promotions, maintaining a realtime hand log database, etc. It would damage the site in the near term, e.g., require pausing certain promotions, so not my call. We can discuss whether to do this and then how to avoid loss of data that will stop flowing, e.g., the hand logs.

On a related note, is it possible to change the logging level to perhaps begin spewing something closer to debug-level log entries?
Kent Briggs
Site Admin
Posts: 5878
Joined: Wed Mar 19, 2008 8:47 pm

Re: Site Freezing and Odd Behavior

Post by Kent Briggs »

escorpius wrote: Fri Aug 19, 2022 11:02 pm On a related note, is it possible to change the logging level to perhaps begin spewing something closer to debug-level log entries?
What do you mean by debug-level log entries?
escorpius
Posts: 64
Joined: Wed Apr 29, 2020 12:47 am

Re: Site Freezing and Odd Behavior

Post by escorpius »

Many services are designed with different logging levels, e.g., (arbitrarily labeled) "Info", "Warning", "Error", and "Debug", where a "Debug" level of logging would spew a large amount of information that the developer would enable when they are developing for debugging purposes and then disable in favor of one of the other levels for production release. Just curious if you followed such a design pattern and, if so, whether it is possible to either enable that in the production release through configs or whether you can provide a debug-enabled version for us to capture more detailed insights.

On the question of the API integration earlier, while we will discuss disabling and such, it is worth noting that we were running this way in v6 without issue. We may no changes to the integration patterns when we upgraded to v7 and the issues began. Were there changes to how the API connections are handled as part of the upgrade?

I'm also suspicious of the session key login management given the duplicate logins we're seeing, even sporadically and clearly erroneously. We may also look to roll that back, which we were also using prior to v7 without issue.
Kent Briggs
Site Admin
Posts: 5878
Joined: Wed Mar 19, 2008 8:47 pm

Re: Site Freezing and Odd Behavior

Post by Kent Briggs »

escorpius wrote: Sat Aug 20, 2022 9:58 am Many services are designed with different logging levels, e.g., (arbitrarily labeled) "Info", "Warning", "Error", and "Debug", where a "Debug" level of logging would spew a large amount of information that the developer
That's pretty much what the Log Settings group on the System tab does. And PM 7 uses a third-party debugging tool called MadExcept that automatically puts out a stack trace in the Error Log any time a code exception occurs.
Were there changes to how the API connections are handled as part of the upgrade?
No, it's the same HTTP request and response system as before.
setspike
Posts: 100
Joined: Fri Apr 03, 2020 6:32 pm

Re: Site Freezing and Odd Behavior

Post by setspike »

Using t3.medium

Doing some basic API integrations and some real time log parsing, but probably not anywhere near as extensive as what you're doing.
escorpius
Posts: 64
Joined: Wed Apr 29, 2020 12:47 am

Re: Site Freezing and Odd Behavior

Post by escorpius »

setspike wrote: Sat Aug 20, 2022 12:32 pm Using t3.medium

Doing some basic API integrations and some real time log parsing, but probably not anywhere near as extensive as what you're doing.
Thanks for the response. Yeah, we do some fairly complicated stuff to extend the functionality. Was never an issue until v7, and we changed nothing on our side.
escorpius
Posts: 64
Joined: Wed Apr 29, 2020 12:47 am

Re: Site Freezing and Odd Behavior

Post by escorpius »

Kent Briggs wrote: Sat Aug 20, 2022 10:12 am
escorpius wrote: Sat Aug 20, 2022 9:58 am Many services are designed with different logging levels, e.g., (arbitrarily labeled) "Info", "Warning", "Error", and "Debug", where a "Debug" level of logging would spew a large amount of information that the developer
That's pretty much what the Log Settings group on the System tab does. And PM 7 uses a third-party debugging tool called MadExcept that automatically puts out a stack trace in the Error Log any time a code exception occurs.
Were there changes to how the API connections are handled as part of the upgrade?
No, it's the same HTTP request and response system as before.
Ah. Well, we have all of those dials on that are in any way meaningful. We have made a change to our login process to do some connection scrubbing that the Mavens software seems to allow to linger. First night ran clean. If it continues to run cleanly (0 confidence, but hopeful), we will provide details in the hopes this begins to lead us to root cause.
escorpius
Posts: 64
Joined: Wed Apr 29, 2020 12:47 am

Re: Site Freezing and Odd Behavior

Post by escorpius »

Froze again.

As I mentioned earlier, we followed your advice and have the system message generating every 60 seconds. Ever 7 seconds after the hour, we saw an entry like this, which is the last before the freeze:
2022-08-22 20:11:07|System|Traffic - Seconds: 60, Bytes in: 3336, Bytes out: 203115, Total: 206451, Threads: 111, CPU: 0.0%, Memory: 41052 kb

Despite the system logging messages past the 20:12:07 mark, no additional System status messages are seen. At some point therafter, the freeze was brought to our attention and we restarted the server, as usual.

What could cause the system logging to stop but other logging to continue? Is this running on a separate thread? Curious if this is a symptom of the larger problem.
Post Reply