So, yesterday we came to the decision that we were happy with the way WSUS was doing updates for the Middle School.  It was decided that it was time to expand the roll out to more of the district.  The 4th/5th Grade building is getting new desktops this year, and so is half the High School, so that means they’re not super high on the priority list.  That really leaves the three K-3 buildings.

So, we added them to WSUS via GPO and bad things started to happen…

First, I got the following error:

The WSUS administration console was unable to connect to the WSUS Server via the remote API. 

Verify that the Update Services service, IIS and SQL are running on the server. If the problem persists, try restarting IIS, SQL, and the Update Services Service.

The WSUS administration console has encountered an unexpected error. This may be a transient error; try restarting the administration console. If this error persists, 

Try removing the persisted preferences for the console by deleting the wsus file under %appdata%\Microsoft\MMC\.

System.IO.IOException -- The handshake failed due to an unexpected packet format.

Source
System

Stack Trace:
 at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
 at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
 at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
 at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)
 at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
 at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
 at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
 at System.Net.TlsStream.ProcessAuthentication(LazyAsyncResult result)
 at System.Net.TlsStream.Write(Byte[] buffer, Int32 offset, Int32 size)
 at System.Net.ConnectStream.WriteHeaders(Boolean async)
** this exception was nested inside of the following exception **

System.Net.WebException -- The underlying connection was closed: An unexpected error occurred on a send.

Source
Microsoft.UpdateServices.Administration

Stack Trace:
 at Microsoft.UpdateServices.Administration.AdminProxy.CreateUpdateServer(Object[] args)
 at Microsoft.UpdateServices.UI.SnapIn.Scope.ServerSummaryScopeNode.GetUpdateServer(PersistedServerSettings settings)
 at Microsoft.UpdateServices.UI.SnapIn.Scope.ServerSummaryScopeNode.ConnectToServer()
 at Microsoft.UpdateServices.UI.SnapIn.Scope.ServerSummaryScopeNode.get_ServerTools()

Well, that’s marvelous.  The suggested fixes for this involved resetting IIS and that seemed dangerous to me.

So I did what any person would do when something misbehaves: I rebooted the server.

BUM BUM BUMMMMMM!

…

Just kidding, it came back up properly after the reboot and things worked fine.

Until 2:30pm, when we got a different error:

The WSUS administration console was unable to connect to the WSUS Server via the remote API. 

Verify that the Update Services service, IIS and SQL are running on the server. If the problem persists, try restarting IIS, SQL, and the Update Services Service.

System.Net.WebException -- The request failed with HTTP status 503: Service Unavailable.

Source
System.Web.Services

Stack Trace:
 at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall)
 at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters)
 at Microsoft.UpdateServices.Internal.ApiRemoting.ExecuteSPSearchComputers(String computerTargetScopeXml)
 at Microsoft.UpdateServices.Internal.DatabaseAccess.AdminDataAccessProxy.ExecuteSPSearchComputers(String computerTargetScopeXml)
 at Microsoft.UpdateServices.Internal.BaseApi.ComputerTarget.SearchComputerTargets(ComputerTargetScope searchScope, UpdateServer updateServer)
 at Microsoft.UpdateServices.UI.AdminApiAccess.ComputerTargetManager.GetComputerTargets(ComputerTargetScope searchScope)
 at Microsoft.UpdateServices.UI.AdminApiAccess.BulkComputerPropertiesCache.GetAndCacheComputers(ExtendedUpdateScope updateScope, ComputerTargetScope computerTargetScope)
 at Microsoft.UpdateServices.UI.SnapIn.Pages.ComputersListPage.GetListRows()

Crap.

This is 2 distinct errors in a row, following a major configuration change.

This calls for some research, methinks.

Upon doing some investigation, I found that the WSUS IIS pool was not running properly anymore.  It seemed like a simple fix.

Open up IIS, find the Application Pools, restart WSUS.

It failed and immediately entered the “Not Running” state again.

Crappppppppp.

Doing more digging, and someone else had the problem too: the WSUS IIS pool needs more memory.

So I give it more memory, and it starts back up.  It stays running…. It hasn’t stopped.  It’s happy again.

So let this be a lesson to you:

If you add a bunch of machines to WSUS and it suddenly seems VERY unhappy, it’s time to bump up the available memory for the WSUS Application Pool.

You can do this by:

1. Open IIS Manager from Start Menu.
2. Open the server view.
3. Open the Application Pools from the server view.
4. Right click on WsusPool and stop it if it is running.
5. Right click on WsusPool again and go to Advanced Settings.
6. Scroll to the bottom of the list and look for the Private Memory Limit (KB). It’s default value is 1.7GB.  I have increased it to 6GB.  Some people have reported as high as 10GB for 25,000 clients.
7. Click OK.
8. Right click on WsusPool a final time and start it.

There’s a good write up as to why this happens here.  The long and short of it is that WSUS keeps track of each update for each machine.  That means your DB grows crazily fast as the number of updates you add to WSUS increases AND as you increase the number of machines that WSUS manages.  For example, our WSUS instance has 3485 updates in it’s database.  It currently has 469 computers in it.  That means WSUS has 1.63 million records to keep track of (as a guesstimate0.  Adding 100 updates means almost 47 thousand more rows.  Yikes.

It’s also a great case for keeping your updates managed well by declining and removing superceeded updates.

I am learning a lot.

This is good.

-M, out.