Hi Guys, I'm providing a pooling duplex service to pottentionaly a lot of clients, and it seems like I'm hitting barries when it comes to number of allowed threads in ASP.Net Thread pool. Is it possible to increase that? Thanks, Stevo
What exactly is the error message you are getting and do you have some instructions for us to follow to replicate it on our end?
Hi Ray, Before the datails, I think its important to note that I only have one .aspx page hosting a silverlight app. I'm hosting a pooling duplex service on a pub/sub model, where my clients (in Silverlight) consume that service by subscibing to the server. Once subscribed, the server is pumping messages to clients (one every second, each message approx. 5-10kb). From my web config: Code: <system.serviceModel> <extensions> <bindingExtensions> <add name="pollingDuplexBinding" type="System.ServiceModel.Configuration.PollingDuplexHttpBindingCollectionElement, System.ServiceModel.PollingDuplex" /> </bindingExtensions> <bindingElementExtensions> <add name="pollingDuplex" type="System.ServiceModel.Configuration.PollingDuplexElement, System.ServiceModel.PollingDuplex" /> </bindingElementExtensions> </extensions> <behaviors> <serviceBehaviors> <behavior name="PoolingDuplexBehavior"> <serviceMetadata httpGetEnabled="true" /> <serviceDebug includeExceptionDetailInFaults="true" /> <serviceThrottling maxConcurrentSessions="200" maxConcurrentCalls="50" /> </behavior> </serviceBehaviors> </behaviors> <bindings> <pollingDuplexBinding> <binding name="PubSubChunkedBinary" maxOutputDelay="00:00:00" useTextEncoding="false" duplexMode="MultipleMessagesPerPoll" /> </pollingDuplexBinding> </bindings> <services> <service name="WebLiveTiming.Web.Services.LiveDataService" behaviorConfiguration="PoolingDuplexBehavior"> <endpoint address="chunked-binary" binding="pollingDuplexBinding" bindingConfiguration="PubSubChunkedBinary" contract="WebLiveTiming.Web.Services.ILiveDataService" /> <endpoint address="mex" binding="mexHttpBinding" contract="IMetadataExchange" /> </service> </services> <serviceHostingEnvironment aspNetCompatibilityEnabled="true" multipleSiteBindingsEnabled="true" /> </system.serviceModel> With small number of clients, the service runs fine. However with reaching about 50/100 clients, the Subscribe request from clients stop getting answered because of a bottleneck somewhere. Here is what errors I get from client side: Code: w3wp.exe Error: 0 : System.ServiceModel.CommunicationException: [SFxServerDidNotReply] Arguments: Debugging resource strings are unavailable. Often the key and arguments provide sufficient information to diagnose the problem. See http://go.microsoft.com/fwlink/?linkid=106663&Version=4.0.51204.0&File=System.ServiceModel.dll&Key=SFxServerDidNotReply at System.ServiceModel.AsyncResult.End[TAsyncResult](IAsyncResult result) at System.ServiceModel.Channels.ServiceChannel.EndCall(String action, Object[] outs, IAsyncResult result) at System.ServiceModel.ClientBase`1.ChannelBase`1.EndInvoke(String methodName, Object[] args, IAsyncResult result) at WebLiveTiming.LiveDataServiceProxy.LiveDataServiceClient.LiveDataServiceClientChannel.EndSubscribe(IAsyncResult result) at WebLiveTiming.LiveDataServiceProxy.LiveDataServiceClient.WebLiveTiming.LiveDataServiceProxy.ILiveDataService.EndSubscribe(IAsyncResult result) at WebLiveTiming.LiveDataServiceProxy.LiveDataServiceClient.OnEndSubscribe(IAsyncResult result) at System.ServiceModel.ClientBase`1.OnAsyncCallCompleted(IAsyncResult result) Code: w3wp.exe Error: 0 : System.TimeoutException: This request operation sent to http://www.liveracers.com/Services/LiveDataService.svc/chunked-binary did not receive a reply within the configured timeout (00:01:00). The time allotted to this operation may have been a portion of a longer timeout. This may be because the service is still processing the operation or because the service was unable to send a reply message. Please consider increasing the operation timeout (by casting the channel/proxy to IContextChannel and setting the OperationTimeout property) and ensure that the service is able to connect to the client. at System.ServiceModel.AsyncResult.End[TAsyncResult](IAsyncResult result) at System.ServiceModel.Channels.ServiceChannel.EndCall(String action, Object[] outs, IAsyncResult result) at System.ServiceModel.ClientBase`1.ChannelBase`1.EndInvoke(String methodName, Object[] args, IAsyncResult result) at WebLiveTiming.LiveDataServiceProxy.LiveDataServiceClient.LiveDataServiceClientChannel.EndSubscribe(IAsyncResult result) at WebLiveTiming.LiveDataServiceProxy.LiveDataServiceClient.WebLiveTiming.LiveDataServiceProxy.ILiveDataService.EndSubscribe(IAsyncResult result) at WebLiveTiming.LiveDataServiceProxy.LiveDataServiceClient.OnEndSubscribe(IAsyncResult result) at System.ServiceModel.ClientBase`1.OnAsyncCallCompleted(IAsyncResult result) Code: w3wp.exe Error: 0 : System.ServiceModel.CommunicationException: [HttpWebRequest_WebException_RemoteServer] Arguments: NotFound Debugging resource strings are unavailable. Often the key and arguments provide sufficient information to diagnose the problem. See http://go.microsoft.com/fwlink/?linkid=106663&Version=4.0.51204.0&File=System.Windows.dll&Key=HttpWebRequest_WebException_RemoteServer ---> System.Net.WebException: [HttpWebRequest_WebException_RemoteServer] Arguments: NotFound Debugging resource strings are unavailable. Often the key and arguments provide sufficient information to diagnose the problem. See http://go.microsoft.com/fwlink/?linkid=106663&Version=4.0.51204.0&File=System.Windows.dll&Key=HttpWebRequest_WebException_RemoteServer ---> System.Net.WebException: [HttpWebRequest_WebException_RemoteServer] Arguments: NotFound Debugging resource strings are unavailable. Often the key and arguments provide sufficient information to diagnose the problem. See http://go.microsoft.com/fwlink/?linkid=106663&Version=4.0.51204.0&File=System.Windows.dll&Key=HttpWebRequest_WebException_RemoteServer at System.Net.Browser.BrowserHttpWebRequest.InternalEndGetResponse(IAsyncResult asyncResult) at System.Net.Browser.BrowserHttpWebRequest.<>c__DisplayClass5.<EndGetResponse>b__4(Object sendState) at System.Net.Browser.AsyncHelper.<>c__DisplayClass2.<BeginOnUI>b__0(Object sendState) --- End of inner exception stack trace --- at System.Net.Browser.AsyncHelper.BeginOnUI(SendOrPostCallback beginMethod, Object state) at System.Net.Browser.BrowserHttpWebRequest.EndGetResponse(IAsyncResult asyncResult) at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelAsyncRequest.CompleteGetResponse(IAsyncResult result) --- End of inner exception stack trace --- at System.ServiceModel.AsyncResult.End[TAsyncResult](IAsyncResult result) at System.ServiceModel.Channels.ServiceChannel.EndCall(String action, Object[] outs, IAsyncResult result) at System.ServiceModel.ClientBase`1.ChannelBase`1.EndInvoke(String methodName, Object[] args, IAsyncResult result) at WebLiveTiming.LiveDataServiceProxy.LiveDataServiceClient.LiveDataServiceClientChannel.EndSubscribe(IAsyncResult result) at WebLiveTiming.LiveDataServiceProxy.LiveDataServiceClient.WebLiveTiming.LiveDataServiceProxy.ILiveDataService.EndSubscribe(IAsyncResult result) at WebLiveTiming.LiveDataServiceProxy.LiveDataServiceClient.OnEndSubscribe(IAsyncResult result) at System.ServiceModel.ClientBase`1.OnAsyncCallCompleted(IAsyncResult result) I'm thinking maybe there is not enough threads available to serve the incomming calls, as to every connected client I'm sending a message on a separate thread every 1 second. I also have other processing which takes some available threads: I'm using RIA Services to trasfer data between the client and the server. That appears to be working correctly even under heavy load. I'm connected to other servers that are sending data to my database approx 1 request every 500 ms, at most that's about 5-10 servers (connections) - means another 5-10 threads permanently working. This all seems to work fine, it's just the Polling Duplex that's not beeing able to even reply to the Subscribe request when the number of clients start reaching 50-100 ...
I still will need the error message and if possible instructions on how we can replicate it on our end.
The server doesnt seem to produce an error message, just does not reply... The only error messages I got are the ones above from client. Its not a trivial task to reproduce this kind of behavior, that's why I'm asking about the threads limit... there might be another reason though, my app reaching the 200MB memory limit and thus beeing recycled. Could you help me with what process do I need to watch to see if it's exceeding the memory limits when debugging at home - is it the w3wp.exe ASP.NET v4.0 IIS Worker Process - process ? or the DefaultAppPool one? Or is it better to use the built-in Visual Studio Web server to monitor the memory usage of my web app - what is the best way to do that? Thank, Stevo
Here are the conditions we set on the servers that can cause your application pool to be recycled. 1) More than 20 minutes of idle time (no http request in 20 minutes) 2) The application uses more than 100 MB memory for Basic plan and 200 MB for Max/Ultimate plan. 3) The application uses more than 75% of CPU time I'm not quite sure if there is a specific limit set to the thread pool for ASP.Net if there is it should be the servers default setting. I'll have to check with our systems administrator about that.
Hi Ray, ani luck finding out about threads? Anyway I have a more interesting question about the memory limits in regards to .NET garbage collection. According to msdn garbace collection happens when: The system has low physical memory. The memory that is used by allocated objects on the managed heap surpasses an acceptable threshold. This means that a threshold of acceptable memory usage has been exceeded on the managed heap. This threshold is continuously adjusted as the process runs. The GC.Collect method is called. In almost all cases, you do not have to call this method, because the garbage collector runs continuously. This method is primarily used for unique situations and testing. I dont know much about your backend and please correct me if I'm wrong, but this is what I image can be happening: You have X-number of servers, each having lets say 12Gb of RAM. On each of those boxes, you can host up to 12Gb/100Mb applications (obviosly you need some RAM to allow for OS and other stuff, but to keep the example simple lets not include that) On those boxes you host our websites. So technically each website has 12Gb of physical RAM space available to it. However by some monitoring mechanism you watch all the applications in the application pool and if any exceeds the 100Mb Ram space, it gets recycled. If that is true, then the first and most important condition for the GC to kick in will never happen - system will never be low on actuall physicall memory. So when my app is reaching 100Mb, the GC looking after it is not worried and happy to allocate more memory, however that way I go over the limit. Please help me to understand how you guys work so I can make my app work better in your environment. Thanks, Stevo
I just verified with our SA's and there is really no hard limit set on the threads an ASPNet IUSR can initiate. However, there the system is set to monitor the servers resources and if there are any evidence of abuse, the SA's will get a text message from the server. The SA's will then contact the user of that account and notify them of the infraction. If you did not get any email from our systems administrator I highly doubt that your web application is crashing due to thread pool issues.
Thanks for that, I just figured out a way how to monitor that myself so I can watch that... Any comments to my question abour RAM limit?
Well I'm a little confused on exactly what you are referring to concerning the memory threshold we have and how ASP.Net Garbage Collection works. First the memory threshold we have is on the server level and not on the application level therefore GC really does not know what the system is monitoring or whether it was recycled. If your application pool does reach the 100 or 200 MB memory threshold, depending on what plan you have, the system will automatically recycle the application pool. What ever objects you already have committed into memory will be wiped clean, kind of like cleaning your room, thus the memory usage of your application will go back to 0 MB until the next call is made to it and all the objects are once again loaded. Plus GC is not only used to maintain memory allocation it is used to keep memory clean by discarding unused or no longer referenced objects, however what if those objects are used. Lets run an example. You have a web application and it connects to a SQL 2008 data source. When you call on a page, the page connects to the database (the connection itself is an object) then you query the database (that too becomes an object). You get 5 results of that record set. Within that record set it carries 5 objects, first name, last name, address, and phone number, but in your web UI you only display the first name and last name. GC will not go through and discard the other 3 SQL objects because in its eyes it is all being utilized. The bottom line is that GC helps to manage memory but it is not the only method to incorporate an efficient and optimized web application. You may have GC enabled on your application and maybe it is set to run of a certain time frame, but if your application reaches the memory threshold dictated by our system, all objects committed into memory, and this includes the GC object, will be wiped clean from memory thus throwing your application pool memory usage back to 0. I hope I understood your question, if not, try rephrasing it again.
Hi Ray, that's fine I understand how GC works. I wanted to point out that the first reason for GC to kick in is when system is running low on RAM. And because the system has much more RAM than what your treshold for recycling an app is (I can imagine usually few Gb of available RAM even when an application gets recycled) , the GC doesnt have a good reason to kick in. Or to put it simply - you're telling GC that there is plenty available RAM however once the app reaches a magic treshold of 100/200Mb it gets killed, without acutally giving GC a chance to do it's job. Anyway I'll try to make my app more memory efficient using memory pooling and some other tricks... Thanks, it's a bit more clear now how the whole thing hangs together. Best Regards, Stevo
Yeah I don't think the GC actually knows the memory threshold for your application pool. Like I said the threshold is more on the server level. The GC may monitor the memory as a whole which are several GB's but we have another monitoring system that monitors the RAM as a whole and makes sure that is is within a certain percentage usage, otherwise it will warn our SA's. So with that aspect the GC will always see the overall RAM as being within limits.
