JBOSS APR Error

Problem: 

Application reports APR exception as below

javax.servlet.ServletException: Not in a valid Comet configuration (use an APR or NIO connector)

at org.granite.gravity.jbossweb.AbstractHttpEventServlet.service(AbstractHttpEventServlet.java:217)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.ja

dsda

Solution:

Install JBOSS native libraries and set LD_LIBRARY_PATH

1) Download  jboss-eap-native-5.0.1-RHEL5-i386.zip for Linux.

2) Extract the zip file to temporary location and move the native folder to jboss-eap-5.0.1 directory (one level above jboss home directory).

The structure looks like this.

jboss-eap-5.0.1

   |__jboss

   |__native

3) Update install account's profile (/home/jbossadm/.bash_profile)

Add this line to the .bash_profile 

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$JBOSS_HOME/../native/lib

4) Exit from Unix shell 

5) Restart JBOSS server

Port forwarding

IP tables port forwarding can be used to direct requests from one port to another. It is extremely helpful in situation where you need to run your application as non-root but still need to serve the app on port 80. This will also eliminate the need for root/sudo privileges.

iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j REDIRECT --to-port 8000

To save the changes permanently, execute the save command.
service iptables save

To look at the saved configuration,
more /etc/sysconfig/iptables

You can also execute the stop and start conmands as required.
service iptables stop
service iptables start

JBOSS Database Connection Leak

If you suspect database connection leak code issues, you can apply this fix.
From JBOSS deploy directory, edit Cached Connection Manager section in jca-jboss-beans.xml.

<bean name="CachedConnectionManager" class="org.jboss.resource.connectionmanager.CachedConnectionManager">
<annotation>@org.jboss.aop.microcontainer.aspects.jmx.JMX(name="jboss.jca:service=CachedConnectionManager", exposedInterface=org.jboss.resource.connectionmanager.CachedConnectionManagerMBean.class)</annotation>
<!-- Whether to track unclosed connections and close them -->
<property name="debug">true</property>
<!-- Whether to throw an error for unclosed connections (true) or just log a warning (false) -->
<property name="error">true</property>
  
<!-- The transaction manager -->
<property name="transactionManager"><inject bean="TransactionManager" property="transactionManager"/></property>
</bean>
--------------------------------------------------------------------------
The following message will be reported in the logs when unclosed connections are detected and closed.

ERROR [org.apache.catalina.connector.CoyoteAdapter] (http-172.22.85.83-8080-2) An exception or error occurred in the container during the request processing
javax.servlet.ServletException: Error invoking cached connection manager
at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:174)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:330)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:829)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:598)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
at java.lang.Thread.run(Thread.java:619)
Caused by: javax.resource.ResourceException: Some connections were not closed, see the log for the allocation stacktraces
at org.jboss.resource.connectionmanager.CachedConnectionManager.popMetaAwareObject(CachedConnectionManager.java:251)
at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:164)

Splunk Errors and Solutions

Problem
Splunk reports error "Your maximum disk usage quota has been reached. The search was not run" when doing searches.

Solution
 Login to Splunk.
 Click on Jobs link on top right corner.
 Delete the jobs and run search again.

Tealeaf Report to track Apache and JBOSS servers

You can build a Tealeaf report template to identify which web and app servers are hit by the users. This can save your troubleshooting time and get to the root cause quickly.

I) Setup an Event
1) Launch RealiTeaPro viewer.
2) Navigate to Edit -> Event Editor -> Attributes
3) Create an Attribute jk-attr-webserver with Attribute Type as Text.

4) Navigate to Edit -> Event Editor -> Categories
5) Create a Category jk-ctg-webserver
 Flag: Active selected
 Match Type: 0-String pattern
 Case: Insensitive
 Encoding: No Translation
 Buffer: Request
 Start Tag: \njk-webserver-req-set-field=
 End Tag: \r

6) Navigate to Edit -> Event Editor -> Events
7) Create an Event jk-evt-webserver
Group: SysOps
Value Tyoe: Default
Match Type: 16-Data is NOT null
Buffer: Filtered by Category
Flag: Interesting Event selected
Event Result Type: Text
Attribute Name: jk-attr-webserver
Category: jk-ctg-webserver

8) Save and Commit the changes.

II) Setup Privacy Filter
1) From the browser, login to Tealeaf portal.
http://{tealeaf-server}/portal/TMS.aspx
2) Navigate to WorldView -> Transport Service -> Privacy Filter configuration -> View/Edit Raw

3) Create or Edit one of the rules
[Rule3]
Enabled=true
Actions=IndexRemote_Addr, IndexRequest_Method, ReqSetTLTURL, ReqSetjk-action-webserver

4) Add the action 

[ReqSetjk-action-WebServer]

Section=cookies

Action=ReqSet

Field=BIGipServer{cookienameforwebserver}

Inclusive=true

ReqSetField=jk-webserver-req-set-field

ReqSetSection=appdata

BIGipServer{cookienameforwebserver}. You can find this field/cookie name from Request data when replaying the Tealeaf session. In my case, BigIP LTM injects a cookie with prefix BIGipServer. This can be any cookie injected by the server.

5) Save the config
6) Click on Transport Service and restart.

III) Build a Tealeaf Report Template
1) From the browser, login to Tealeaf portal.
http://{tealeaf-server}/portal/SearchTemplateConfig.aspx
2) Create a new template 'Operations Template' or add the relevant columns to the existing template.

3) Add the WebServer column
Title: WebServer
Field: Session Attribute Value
Attribute: jk-attr-webserver
Operation: Display Field Value
4) Save.

Your report now displays the column titled WebServer with cookie value. With this cookie value, you can find out which server is being hit. The same procedure can be repeated to track JBOSS appserver by reading the corresponding cookie.

If you are using BigIP LTM, you can track the server using cookie. Please refer
 http://techwaver.blogspot.com/2008/12/decode-bigip-cookie-to-identify-pool.html

JBOSS LDAP Password Encryption

Please make sure your JBOSS LDAP connection works fine with clear password before proceeding with encryption.

1) Create a mbean file named encrypt-service.xml and place it in the deploy folder.
encrypt-service.xml
------------------------------------------------------------------------------
   <mbean code="org.jboss.security.plugins.JaasSecurityDomain"
      name="jboss.security:service=JaasSecurityDomain,domain=jk-ldap-security">
      <constructor>
         <arg type="java.lang.String" value="jk-ldap-security"></arg>
      </constructor>
      <attribute name="KeyStorePass">rchitect</attribute>
      <attribute name="Salt">rchitect</attribute>
      <attribute name="IterationCount">66</attribute>
   </mbean>
------------------------------------------------------------------------------
Note: The Salt value should be 8 bytes long. More than 8 bytes is not accepted at the moment.

2) Restart the server if required.

3) Login to jmx-console http://{ip-address}:8080/jmx-console/

4) From the left hand side navigation Object Name Filter, select jboss.security and click on the link domain=jk-ldap-security,service=JaasSecurityDomain

5) Go to Operation -> encode64 -> Type your LDAP Bind Password and click Invoke.

6) The encrypted password will be displayed on the screen. Please save this.

7) Update your login-config.xml 

Replace

<module-option name="bindCredential">clear-text-password</module-option>

with 

<module-option name="bindCredential">{encrypted-password-from-above}</module-option>

<module-option name="jaasSecurityDomain">jboss.security:service=JaasSecurityDomain,domain=jk-ldap-security</module-option>

8) Restart the server.

Your ldap bind password is now encrypted!

9) As you can notice above, KeyStorePass is still in clear text form. In order to encrypt Keystore pass, create a file server.password in conf directory using the command below substituting with proper parameters.
java -cp common/lib/jbosssx.jar org.jboss.security.plugins.FilePassword $saltvalue $iterationcountvalue $password $JBOSS_SERVER_HOME/conf/server.password
e.g.
java -cp common/lib/jbosssx.jar org.jboss.security.plugins.FilePassword rchitect 66 rchitect $JBOSS_SERVER_HOME/conf/server.password

10)  Replace the clear text KeyStorePass with the folllowing in encrypt-service.xml
<attribute name="KeyStorePass">{CLASS}org.jboss.security.plugins.FilePassword:${jboss.server.home.dir}/conf/server.password</attribute>

If you prefer not to use mbean for encryption, you can also use the following command to encrypt the ldap bind password.

java -cp common/lib/jbosssx.jar org.jboss.security.plugins.PBEUtils $saltvalue $iterationcountvalue $password $encryptpassword

e.g.

java -cp common/lib/jbosssx.jar org.jboss.security.plugins.PBEUtils rchitect 66 rchitect  ldap-bind-clear-password

Splunk Dashboard

Splunk search dashboard summary displays sources, sourcetypes and hosts. These hosts will list your actual server name. If you would like to have user friendly name for your hosts, follow these steps.
1) Navigate to http://{splunk_agent_host}:8000 and login to Splunk agent on target host.
2) Manager -> System settings -> General settings -> Index settings.
3) Update the Default host name (optional) field to have user friendly name
4) SSH into splunk agent and run these commands
 $SPLUNK_HOME/bin/splunk stop
 $SPLUNK_HOME/bin/splunk clear all
 $SPLUNK_HOME/bin/splunk start
5) Login to Splunk server and check the dashboard for discovery of user friendly host.
Note: The password may get reset due to clear all command.
------------------------------------------------------------------------------

By default, Splunk dashboard lists 10 hosts. When you have large no. of hosts, navigating 10 hosts at a time might be cumbersome. To increase the size of hosts displayed,
1) Navigate to Manager -> User interface -> Views -> Select 'search' as app context from dropdown -> dashboard.
2) Update this block in the section after  <!-- The list of hosts -->
        <module name="Paginator">
            <param name="count">25</param>
            <param name="entityName">settings</param>
            <param name="maxPages">25</param>
            <module name="SearchLinkLister">
3) Restart splunk server