Very, very urgent bug

Report bugs found in Poker Mavens

Very, very urgent bug

Postby vpalmer » Thu Dec 04, 2008 11:42 am

So we ran into a server failure when we had 300 simultaneous users connected on a dedicated NT dual-processor Xenon system.

a few questions

1) is it possible to run multiple instances of your client on the same server ... each one running on multiple processors

2) what can we do to make the client more efficient

3) we noticed a lot of users that just closed the browser and were still 'logged on' ... does this consume server resources or slow down the client?

Victor Palmer
CentSports.com
[email protected]
979-574-5133
vpalmer
 
Posts: 5
Joined: Thu Sep 04, 2008 1:05 am

Re: Very, very urgent bug

Postby Kent Briggs » Thu Dec 04, 2008 12:41 pm

vpalmer wrote:So we ran into a server failure when we had 300 simultaneous users connected on a dedicated NT dual-processor Xenon system.


Yeah, I logged into centpoker last night at one point when there were over 400 users. But it was running very slow. I was curious if you happened to check the CPU to see if it was pegged out near 100% or if the bottleneck was the bandwidth.

1) is it possible to run multiple instances of your client on the same server ... each one running on multiple processors


Each one would have to be under a different account with its own Windows registry. And they might have to use different ports, I'm not sure about that. Of course if that many users is maxing out the CPU, spreading it across multiple instances under the same CPU won't help anyway.

2) what can we do to make the client more efficient


You may have to balance the load across multiple servers. The system isn't currently designed to scale up like that however so you would have to use the API to design a front end that puts players into specific servers and move their account data with them.

3) we noticed a lot of users that just closed the browser and were still 'logged on' ... does this consume server resources or slow down the client?


First thing you should do is install the latest version (2.23) which has fixed some connectivity bugs. I've mentioned this numerous times to your administrator in email but it's just been ignored along with several other suggestions I made. :( But closing the browser should send a disconnect signal to the server and it should be removed from the connections list within 2 minutes. All bets are off if the system is overloaded, however.
Kent Briggs - [email protected]
Briggs Softworks - http://www.briggsoft.com
Kent Briggs
Site Admin
 
Posts: 3830
Joined: Wed Mar 19, 2008 8:47 pm

Re: Very, very urgent bug

Postby Kent Briggs » Thu Dec 04, 2008 2:13 pm

vpalmer wrote:2) what can we do to make the client more efficient


Oh by the way, those Flash video ads you run on the left side are sucking 50 to 60% of my CPU usage all by themselves in the web browser. On the client side, that's a big part of your performance problem right there.
Kent Briggs - [email protected]
Briggs Softworks - http://www.briggsoft.com
Kent Briggs
Site Admin
 
Posts: 3830
Joined: Wed Mar 19, 2008 8:47 pm

Re: Very, very urgent bug

Postby vpalmer » Thu Dec 04, 2008 2:59 pm

I ran perfmon.msc and CPU usage and memory usage were both very low, but there were a very high number of context switches and interrupts. we also noticed 100s of reads and writes per second to the registry when running sysinternals tools.
vpalmer
 
Posts: 5
Joined: Thu Sep 04, 2008 1:05 am

Re: Very, very urgent bug

Postby Kent Briggs » Thu Dec 04, 2008 5:31 pm

vpalmer wrote:I ran perfmon.msc and CPU usage and memory usage were both very low, but there were a very high number of context switches and interrupts. we also noticed 100s of reads and writes per second to the registry when running sysinternals tools.


The registry should only be updated once per hand to record the last hand number value and any rake collected. There are probably lots of reads on the registry, though, because that's were all the system variables are stored. Do you have any tools that measure the bandwidth?
Kent Briggs - kbriggs[email protected]
Briggs Softworks - http://www.briggsoft.com
Kent Briggs
Site Admin
 
Posts: 3830
Joined: Wed Mar 19, 2008 8:47 pm

Re: Very, very urgent bug

Postby Kent Briggs » Thu Dec 04, 2008 6:10 pm

In perfmon.msc I see you can select "Network Interface" for the object and "Bytes Received/sec" and "Bytes Sent/sec" as counters. I'd be curious to know what numbers you get there at various login loads.
Kent Briggs - [email protected]
Briggs Softworks - http://www.briggsoft.com
Kent Briggs
Site Admin
 
Posts: 3830
Joined: Wed Mar 19, 2008 8:47 pm

Re: Very, very urgent bug

Postby lp69 » Wed Dec 10, 2008 2:03 pm

Hi
I'm interested in this thread as we might run the poker server on a very large scale one day (500+ users, maybe much more).
We used to run a server with other games, with 3000+ simultaneous users, on a 2x2 GHZ, 4 GB RAM server. Cpu hardly reaches 40% usage with this game server.
Bottlenecks could be CPU, memory, bandwith, but also disk usage, especially as poker mavens uses file I/O to do the 'databasing'. You should put a performance counter on disk usage.

If CPU usage is involved it is certainly due to a non optimized server code. We did profile our game server code (written in VB6) and could make it work about 5x faster by just making obvious modifications.
We also wrote a small app that simulates 5000 users logging in and doing various random things to exercize our server and see how it behaved. Very instructive. Might be a good idea for you Kent.

You talked about a failure. What kind of failure was it ? Did you look in the event viewer to see what Windows had logged about this ? (type 'eventvwr' in a cmd box)

Laurent
lp69
 
Posts: 9
Joined: Sat Sep 20, 2008 2:49 am

Re: Very, very urgent bug

Postby lp69 » Wed Dec 10, 2008 2:05 pm

BTW, registry I/O is cached on Windows system so it's not likely to be the bottleneck
lp69
 
Posts: 9
Joined: Sat Sep 20, 2008 2:49 am

Re: Very, very urgent bug

Postby Kent Briggs » Wed Dec 10, 2008 2:39 pm

lp69 wrote:Bottlenecks could be CPU, memory, bandwith, but also disk usage, especially as poker mavens uses file I/O to do the 'databasing'. You should put a performance counter on disk usage.


I suspect the first point of failure is bandwidth related. When I ran perfmon.msc on my end as a client connected to centpoker's site, I was receiving an average of about 8000 bytes per second from the server with 1 table open and about 300 logins. Add on top of that the 700K client module that is being pushed to players who are connecting for the first time (or had cleared their browser cache since their last visit) and the numbers pile up. By far the biggest packet is the list of players logged in. Although it doesn't go out more than once per 2.5 seconds, the whole list goes with every change. So that's a big optimization I can make right there just by only sending changes. And if necessary I could dump the list and just implement a player search feature instead.

We also wrote a small app that simulates 5000 users logging in and doing various random things to exercize our server and see how it behaved. Very instructive. Might be a good idea for you Kent.


Yeah, that's the weak point in my testing, it's hard to simulate lots of logged in users. Even with a custom app like you described, I can't simulate the client end without having 100's of PCs to spread the load over.
Kent Briggs - [email protected]
Briggs Softworks - http://www.briggsoft.com
Kent Briggs
Site Admin
 
Posts: 3830
Joined: Wed Mar 19, 2008 8:47 pm

Re: Very, very urgent bug

Postby lp69 » Wed Dec 10, 2008 4:12 pm

Although it doesn't go out more than once per 2.5 seconds, the whole list goes with every change. So that's a big optimization I can make right there just by only sending changes. And if necessary I could dump the list and just implement a player search feature instead.


sure !! You don't even need to think of something else. At this time I have 2481 players at my own site and it is pushing out "only" 619 Kb/s for all of them (they're not all in the same room though).
The full list is only sent once at log in, and the client receives player in/out info only

Yeah, that's the weak point in my testing, it's hard to simulate lots of logged in users. Even with a custom app like you described, I can't simulate the client end without having 100's of PCs to spread the load over.


I'm using TCP sockets for client/server communications.
My app opens 5000 sockets to the server and sends info simulating a dummy player for each socket. So there is no need to run a full client for each dummy players.

LAurent
lp69
 
Posts: 9
Joined: Sat Sep 20, 2008 2:49 am

Next

Return to Bugs

Who is online

Users browsing this forum: No registered users and 6 guests

cron