Site Freezing and Odd Behavior

Report bugs found in Poker Mavens
Post Reply
escorpius
Posts: 55
Joined: Wed Apr 29, 2020 12:47 am

Re: Site Freezing and Odd Behavior

Post by escorpius »

Player Load: If you mean # of players, it's a very normal, relatively small number, e.g., a single table worth of players the last time it happened.

Laggy Behavior: No, not really. It happens pretty abruptly, usually signaled by anomalies such as duplicate (but, not real) connection listings in the UI suddenly appearing for all players.

SpeedTest: We run the server on AWS, so that's not an issue. Last test was 4500+MBPS both up and down. [Report]

Error Log: The last time it occurred, there were memory access violation entries and a couple of stack traces dumped to the log:

Code: Select all

2022-06-29 21:04:38|ProcessPacket error: Access violation at address 00000000007B268B in module 'PMService.exe'. Read of address FFFFFFFFFFFFFFFF, ID=00000138, Response=Login
2022-06-29 21:04:38|ProcessPacket error: Access violation at address 000000000093CF16 in module 'PMService.exe'. Read of address 0000000000000000, ID=00000131, Response=Login
2022-06-29 21:04:38|ProcessPacket error: Access violation at address 00000000007B268B in module 'PMService.exe'. Read of address FFFFFFFFFFFFFFFF, ID=0000013E, Response=Login
2022-06-29 21:04:38|ProcessPacket error: Access violation at address 00000000007B268B in module 'PMService.exe'. Read of address FFFFFFFFFFFFFFFF, ID=0000013D, Response=Login
2022-06-29 21:04:38|PlayerExecute error: Access violation at address 00000000007B268B in module 'PMService.exe'. Read of address FFFFFFFFFFFFFFFF
2022-06-29 21:04:38|ProcessPacket error: Access violation at address 00000000007B268B in module 'PMService.exe'. Read of address FFFFFFFFFFFFFFFF, ID=00000134, Response=Login
2022-06-29 21:04:38|PlayerExecute error: Access violation at address 000000000093CF16 in module 'PMService.exe'. Read of address 0000000000000000
2022-06-29 21:04:38|WSServerExecute error: Access violation at address 00000000007B268B in module 'PMService.exe'. Read of address FFFFFFFFFFFFFFFF
2022-06-29 21:04:38|WSServerExecute error: Access violation at address 000000000093CF16 in module 'PMService.exe'. Read of address 0000000000000000
2022-06-29 21:04:38|ProcessPacket error: Access violation at address 00000000007B268B in module 'PMService.exe'. Read of address FFFFFFFFFFFFFFFF, ID=00000127, Response=Login
2022-06-29 21:04:38|WSServerExecute stack trace 7.07: 007b268b PMService.exe IdContext           218 TIdContext.GetBinding - 0093cf0e PMService.exe Players            1854 TPlayerSession.ProcessLogin - 00952e87 PMService.exe Players            4134 TPlayerSession.ProcessPacket - 00934699 PMService.exe Players             632 PlayerSessionExecute - 00ac480d PMService.exe Main               1204 WSServerEvents.OnExecute - 007d6682 PMService.exe IdCustomTCPServer   553 TIdCustomTCPServer.DoExecute - 007b25b7 PMService.exe IdContext           185 TIdContext.Run - 007b04e7 PMService.exe IdTask              136 TIdTask.DoRun - 007d20b8 PMService.exe IdThread            697 TIdThreadWithTask.Run - 007d1347 PMService.exe IdThread            428 TIdThread.Execute - 004f97a4 PMService.exe madExcept               HookedTThreadExecute - 005d5050 PMService.exe System.Classes    14561 ThreadProc - 0040eaf8 PMService.exe System            23354 ThreadWrapper - 004f9679 PMService.exe madExcept               ThreadExceptFrame - 7ffa3e88 KERNEL32.DLL                          BaseThreadInitThunk - 7ffa3f16 ntdll.dll                             RtlUserThreadStart
2022-06-29 21:04:38|WSServerExecute stack trace 7.07: 0093cf16 PMService.exe Players            1854 TPlayerSession.ProcessLogin - 00952e87 PMService.exe Players            4134 TPlayerSession.ProcessPacket - 00934699 PMService.exe Players             632 PlayerSessionExecute - 00ac480d PMService.exe Main               1204 WSServerEvents.OnExecute - 007d6682 PMService.exe IdCustomTCPServer   553 TIdCustomTCPServer.DoExecute - 007b25b7 PMService.exe IdContext           185 TIdContext.Run - 007b04e7 PMService.exe IdTask              136 TIdTask.DoRun - 007d20b8 PMService.exe IdThread            697 TIdThreadWithTask.Run - 007d1347 PMService.exe IdThread            428 TIdThread.Execute - 004f97a4 PMService.exe madExcept               HookedTThreadExecute - 005d5050 PMService.exe System.Classes    14561 ThreadProc - 0040eaf8 PMService.exe System            23354 ThreadWrapper - 004f9679 PMService.exe madExcept               ThreadExceptFrame - 7ffa3e88 KERNEL32.DLL                          BaseThreadInitThunk - 7ffa3f16 ntdll.dll                             RtlUserThreadStart
Kent Briggs
Site Admin
Posts: 5622
Joined: Wed Mar 19, 2008 8:47 pm

Re: Site Freezing and Odd Behavior

Post by Kent Briggs »

usually signaled by anomalies such as duplicate (but, not real) connection listings in the UI suddenly appearing for all players.
Try to grab a screenshot of that if it happens again.
and a couple of stack traces dumped to the log
Those traces point to a portion of code where a player attempts a new login while already logged in and access violation happens at the point where it tries to disconnect the existing login. Don't know why, though. This is isolated to a particular connection thread and should not affect the rest of the server. In fact it may just be an after-the-fact symptom of something else gone wrong rather than the cause. What was happening in the Event Log just before that 21:04:38 timestamp?
Kent Briggs - [email protected]
Briggs Softworks - http://www.briggsoft.com
escorpius
Posts: 55
Joined: Wed Apr 29, 2020 12:47 am

Re: Site Freezing and Odd Behavior

Post by escorpius »

We captured a screenshot when it happened previously. I am attaching a .PNG.

I have also attached a fully redacted and obfuscated EventLog from this event (June 29). Note that, while obfuscated, the IP addresses are still correctly matched, i.e., the obfuscated version of an address is the same everywhere in the file to allow for tracing of events.

Let me know if we can provide anything more.
Attachments
ClickItLogins.png
ClickItLogins.png (175.92 KiB) Viewed 272 times
EventLog2022-06-29.txt.zip
(30.48 KiB) Downloaded 11 times
Kent Briggs
Site Admin
Posts: 5622
Joined: Wed Mar 19, 2008 8:47 pm

Re: Site Freezing and Odd Behavior

Post by Kent Briggs »

escorpius wrote: Mon Jul 04, 2022 2:10 pm We captured a screenshot when it happened previously. I am attaching a .PNG.
Do you still have your May 4th logs? I'd like to see what was happening around that 06:11:29 pm timestamp (error and event). You can email them to me direct rather than posting them here.
Kent Briggs - [email protected]
Briggs Softworks - http://www.briggsoft.com
Kent Briggs
Site Admin
Posts: 5622
Joined: Wed Mar 19, 2008 8:47 pm

Re: Site Freezing and Odd Behavior

Post by Kent Briggs »

Looking at the June 29 logs, something bizarre happened at the 21:04:38 mark where there was a mass logout event. There were no adjacent API or Admin commands showing this was intentional. And many of the logouts were duplicates, which makes no sense. And then instantly there were mass login events (also full of duplicates) that had no corresponding connect events or even API calls to generate session keys. None of that makes any sense to me.
Kent Briggs - [email protected]
Briggs Softworks - http://www.briggsoft.com
escorpius
Posts: 55
Joined: Wed Apr 29, 2020 12:47 am

Re: Site Freezing and Odd Behavior

Post by escorpius »

I've emailed you directly the May 04 log. FWIW, we manage all logins through the API creating Session Keys -- no one logs in directly to the server. Also, it likely goes without saying, but the mass logouts were definitely not intentional.
BlackKnite69
Posts: 10
Joined: Mon Mar 16, 2020 2:19 pm

Re: Site Freezing and Odd Behavior

Post by BlackKnite69 »

Any update on this? Experiencing this and it is frustrating. Players are getting fed up.

I have noticed that I can terminate all connections and the site will usually let people sign back in shortly afterwards.
Kent Briggs
Site Admin
Posts: 5622
Joined: Wed Mar 19, 2008 8:47 pm

Re: Site Freezing and Odd Behavior

Post by Kent Briggs »

BlackKnite69 wrote: Thu Jul 14, 2022 12:06 am Any update on this? Experiencing this and it is frustrating. Players are getting fed up.
Are you getting duplicate logins showing in the lobby?
Kent Briggs - [email protected]
Briggs Softworks - http://www.briggsoft.com
BlackKnite69
Posts: 10
Joined: Mon Mar 16, 2020 2:19 pm

Re: Site Freezing and Odd Behavior

Post by BlackKnite69 »

Yes. sometimes multiple of logs of players.
ftpjesus
Posts: 150
Joined: Fri Dec 30, 2016 1:54 am

Re: Site Freezing and Odd Behavior

Post by ftpjesus »

While not directly involved I have to wonder if theres something going on with the login process on his server its like its creating multiple session keys for the same user id which shouldnt be happening since OP says they use Session Keys to control access. Im guessing this is less a PM issue and more on the front end they are using. Just my observation and .02.
Owner Operator of (TBA)
http://www.( TBA ) .com (Home Page)
http://www. (TBA) .net (Play Money)
http://www. (TBA) .eu (BTC Token tied currency and Stable Coins through Crypto quasi real money play)
Post Reply