503 error using Java Hosting Package

[Hoobajoob Profile]

So, I followed the directions for setting up a Java WAR application on my Windows 2008 R2 server running IIS 7 as described here:

http://www.helicontech.com/articles/dep ... -with-iis/

1) I installed the Java Hosting Package
1.1) I ran iisreset
2) I created a directory c:\sites\sample
3) I copied the Apache "Hello World" sample jar into c:\sites\sample (sample jar available here: http://tomcat.apache.org/tomcat-6.0-doc/appdev/sample/)
4) I edited the security properties on c:\sites to give the DefaultAppPool identity, IUSR, and IIS_IUSRS all full read/write privileges just to cover all the bases
5) I created a Web.config file in the same directory as the sample jar (see attachment) and used the example from the above article
6) In the IIS manager, I created a new web app under the default site (which is bound to port 8080 on my machine), giving it the name "sample" with the physical directory c:\sites\sample and the DefaultAppPool
7) I pointed my browser at http://<the-server>:8080/sample (and http://<the-server>:8080/sample/sample in case the web app synonym was adding another level of indirection)
8) The browser comes back with a 503 error "the service is unavailable" and the following response header:

Code: Select all

Cache-Control:must-revalidate,no-cache,no-store
Content-Length:27
Content-Type:text/html
Date:Tue, 29 Oct 2013 23:47:21 GMT
Server:Jetty(7.6.7.v20120910)
X-Powered-By:ASP.NET

At least Jetty appears to be processing the request, but it clearly is not functioning properly.

I looked in the sample application directory (c:\sites\sample) and found a file named "ZooError.log" with the following contents:

Code: Select all

[10/29/2013 16:24:23] v3.0.98.472   BUG: Unexpected exit from worker. ActiveConnectionsCount()!=0 Please send report to [email protected]


I tried stopping the site, running iisreset, and restarting the site, but the results are the same (except there are no new entries appearing in ZooError.log).

How can I troubleshoot this problem?

[Ruslan]

Hello!

First, thank you for detailed description of problem.

It is looks like we have fixed this issue in beta version of Helicon Zoo Module.

Please use beta feed
http://www.helicontech.com/zoo/feed-beta/
to install new version of Zoo module from Web Platform Installer
or direct link
http://www.helicontech.com/zoo/feed-bet ... 73_x86.msi

Please report us about results.

Thank you!

[Hoobajoob Profile]

Awesome! Thank you for the quick response. I'll pull in the update and let you know the results.

[Hoobajoob Profile]

Unfortunately, the update appears to have created new problems.

Here is what I did:

1) uninstalled the current helicon zoo module using the control panel's add/remove programs
2) recycled the server
3) ran the new module msi from the direct link you provided
4) ran iisreset

Now even the default site, instead of returning the default IIS 7 logo, returns a 503 service unavailable error, which was not happening with the previous package. The headers in the response look like this:

Code: Select all

HTTP/1.1 503 Service Unavailable
Content-Type: text/html; charset=us-ascii
Server: Microsoft-HTTPAPI/2.0
Date: Wed, 30 Oct 2013 16:39:58 GMT
Connection: close
Content-Length: 326

I haven't altered the default configuration under inetsrv\config that is set up by the installation, nor have I changed the Web.config file contents for the sample WAR.

There are no new errors showing up in ZooError.log.

I'm concerned that this installation has affected the operation of the default site - what can I do to find the root cause of this problem?

[Hoobajoob Profile]

I also found this error in the windows event logs:

Code: Select all

Application pool 'DefaultAppPool' is being automatically disabled due to a series of failures in the process(es) serving that application pool.

Note that there is only one application configured on this site - the sample application.

[Hoobajoob Profile]

There are also several warnings leading up to the error, presumably corresponding to each browser request that hit the site. Below is a copy of the message and data that is reported with those events:

Code: Select all

A listener channel for protocol 'http' in worker process '1700' serving application pool 'DefaultAppPool' reported a listener channel failure.  The data field contains the error number.

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
- <System>
  <Provider Name="Microsoft-Windows-WAS" Guid="{524B5D04-133C-4A62-8362-64E8EDB9CE40}" EventSourceName="WAS" />
  <EventID Qualifiers="32768">5139</EventID>
  <Version>0</Version>
  <Level>3</Level>
  <Task>0</Task>
  <Opcode>0</Opcode>
  <Keywords>0x80000000000000</Keywords>
  <TimeCreated SystemTime="2013-10-30T16:36:21.000000000Z" />
  <EventRecordID>8465</EventRecordID>
  <Correlation />
  <Execution ProcessID="0" ThreadID="0" />
  <Channel>System</Channel>
  <Computer>jm2-shatner</Computer>
  <Security />
  </System>
- <EventData>
  <Data Name="AppPoolID">DefaultAppPool</Data>
  <Data Name="ProcessID">1700</Data>
  <Data Name="param3">0</Data>
  <Data Name="ProtocolID">http</Data>
  <Binary>7E000780</Binary>
  </EventData>
  </Event>


[Hoobajoob Profile]

There are also 10 errors in the application event logs, one every second for 10 seconds, following an iisreset. Each one indicates that the 64-bit zoo module dll is failing to load. Below is the error message and data:

Code: Select all

The Module DLL C:\Zoo\NativeModule\HeliconZoo_x64.dll failed to load.  The data is the error.


- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
- <System>
  <Provider Name="Microsoft-Windows-IIS-W3SVC-WP" Guid="{670080D9-742A-4187-8D16-41143D1290BD}" EventSourceName="W3SVC-WP" />
  <EventID Qualifiers="49152">2280</EventID>
  <Version>0</Version>
  <Level>2</Level>
  <Task>0</Task>
  <Opcode>0</Opcode>
  <Keywords>0x80000000000000</Keywords>
  <TimeCreated SystemTime="2013-10-30T17:31:29.000000000Z" />
  <EventRecordID>5391</EventRecordID>
  <Correlation />
  <Execution ProcessID="0" ThreadID="0" />
  <Channel>Application</Channel>
  <Computer>jm2-shatner</Computer>
  <Security />
  </System>
- <EventData>
  <Data Name="ModuleDll">C:\Zoo\NativeModule\HeliconZoo_x64.dll</Data>
  <Binary>7E000000</Binary>
  </EventData>
  </Event>

[Hoobajoob Profile]

It looks like there is no HeliconZoo_x64.dll file in the c:\Zoo\NativeModule folder. In that directory there is only the x86 dll.

Shouldn't the 64-bit dll be provided by the installer?

The default application pool has "Enable 32-Bit Applications" set to "False".

[Hoobajoob Profile]

Doh! I guess the link you provided was for an x86 msi. Found the link to the x64 msi (http://www.helicontech.com/zoo/feed-bet ... 73_x64.msi), and trying that now.

[Hoobajoob Profile]

Ok, now the default site is working, but I'm still getting the same 503 error from jetty:

Code: Select all

HTTP/1.1 503 Service Unavailable
Cache-Control: must-revalidate,no-cache,no-store
Content-Type: text/html
Server: Jetty(7.6.7.v20120910)
X-Powered-By: ASP.NET
Date: Wed, 30 Oct 2013 18:11:15 GMT
Content-Length: 27

...do I need to remove and re-install the Java Hosting Package using the beta feed? If so, I can't figure out how to remove the original Java Hosting Package - is there a procedure for this?

[Hoobajoob Profile]

I am also noticing after running for a while that the IIS worker process that presumably hosts the zoo modules appears to have a memory leak, as it is consuming all of the available memory on the system - 3+ GB of ram in one w3wp process on my system.

Running iisreset drops it back down, but then it appears to pick up an additional 1-3 MB of RAM every second.

This growth appears to stop (does not grow or decrease) when I go to the IIS manager and stop the web site.

If I start the web site again, the memory usage starts growing again at the same rate.

[Ruslan]

1. Is there is a new messages in ZooError.log under site?
2. Do you see java/jetty process in process explorer when you start the site and make first request?

Thank you.

[Ruslan]

Could you please make direct request to jetty without iis/zoo. See jetty running port (it is random for every jetty start) via process explorer and request http://localhost: port>/

[Hoobajoob Profile]

No, there are no errors written to ZooError.log.

I'm working on other tasks at the moment, so it might be a few days before I can get back to this and try your troubleshooting suggestions. Thank you for your help!

[Ruslan]

Looks like something goes wrong in Jetty.
It would be great to see jetty stout/stderr output.

To log Jetty output:
1. Create 'log' folder and set full permission to IIS_IUSRS group (or your iis app pool user)
2. Add enviroment variable 'ERROR_LOG_DIR' to zoo configuration in web.config:

Code: Select all

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
  <system.webServer>
    <heliconZoo>
      <clear />
      <application name="jetty.project" >
        <environmentVariables>
          ...
          <add name="WAR_FILE" value="example.war" />
     
          <!-- logging -->
          <add name="ERROR_LOG_DIR"  value="log" />
        </environmentVariables>
      </application>
    </heliconZoo>

    ...

  </system.webServer>
</configuration>


3. iisreset and make request to web site.
4. Send us files located in logs folder.

Thank you!