Advanced SSL configuration on IBM Http Server – Restrict unused HTTP methods and Verbose HTTP headers

Restricting unused HTTP methods

The HTTP method is supplied in the request line and specifies the operation that the client has requested. Browsers will generally just use two methods to access and interact with web sites; GET for queries that can be safely repeated and POST for operations that may have side effects. This means, we need to disable unused http methods. some of them are:(PUT|DELETE|TRACE|TRACK|COPY|MOVE|LOCK|UNLOCK|PROPFIND|PROPPATCH|SEARCH|MKCOL). Check with the application teams, if they need any of these methods for the application to work, before disabling them.

Testing before limiting http methods:

telnet josephamrithraj.mp 80
Trying xx.xx.xx.xx…
Connected to josephamrithraj.mp.
Escape character is ‘^]’.
OPTIONS / HTTP/1.1
Host: josephamrithraj.mp

HTTP/1.1 200 OK
Date: Thu, 14 Sep 2010 00:11:57 GMT
Server: Apache Web Server
Content-Length: 0
Allow: GET, HEAD, POST, PUT, DELETE, CONNECT, OPTIONS, PATCH, PROPFIND, PROPPATCH, MKCOL, COPY, MOVE, LOCK, UNLOCK, TRACE

Connection closed by foreign host.

your IBM http servers configuration file [httpd.conf] has 2 sections named main and virtualhost sections. you need to add the following code at both the places. I am explaining this task using mod_rewrite module. So, first make sure that… mod_rewrite is enabled. then, add the following lines to your http.conf files main and virtualhost sections.

RewriteEngine On
RewriteCond %{REQUEST_METHOD} ^(PUT|DELETE|TRACE|TRACK|COPY|MOVE|LOCK|UNLOCK|PROPFIND|PROPPATCH|SEARCH|MKCOL)
RewriteRule .* – [F]

Restart the web server after adding the above lines.


Now, when someone tried to use one of these http methods, they will get forbidden response since we specified [F] in the rewrite rule.

Testing after adding and restarting web server

telnet josephamrithraj.mp 80
Trying xx.xx.xx.xx...
Connected to josephamrithraj.mp.
Escape character is '^]'.
OPTIONS / HTTP/1.1
Host: josephamrithraj.mp

HTTP/1.1 200 OK
Date: Thu, 14 Sep 2010 00:15:44 GMT
Server: Apache Web Server
Content-Length: 0
Allow: GET, POST
Connection closed by foreign host.
Testing TRACE methods

telnet josephamrithraj.mp 80
Trying xx.xx.xx.xx...
Connected josephamrithraj.mp
Escape character is '^]'.
TRACE / HTTP/1.0
Host: josephamrithraj.mp
testing... <- ENTER twice HTTP/1.1 403 Forbidden Date: Thu, 14 Sep 2010 00:18:31 GMT Server: Apache Web Server Content-Length: 320 Connection: close Content-Type: text/html; charset=iso-8859-1

Forbidden

You don't have permission to access / on this server.

Connection closed by foreign host.
Disable verbose HTTP headers:


you might have seen this … when the web server [apache or ibm http server] throws errors page, sometimes it might show the information related to its version, build, modules etc. This is a security issue since you are giving away the details about your web server. for example, take a look at this:

Server: Apache/2.0.53 (Ubuntu) PHP/4.3.10-10ubuntu4 Server at xx.xx.xx.xx Port 80
The line in the server header expose important version and variant information about the Linux operating system and Apache software used on the machine, indirectly expose the possible security holes that are existed to the hackers, or at least make malicious attackers easier to identify your system for available attack points.
To ensure that the Apache HTTP web server does not broadcast this message to the whole world publicly and fix possible security issue, modify these two directives ServerTokes and ServerSignature in httpd.conf configuration file.

ServerTokens

This directive configures what you return as the Server HTTP response Header. The built-in default is ‘Full’ which sends information about the OS-type and compiled in modules. The recommended value is ‘Prod’ which sends the least information.

Options: Full | OS | Minor | Minimal | Major | Prod

“ServerTokens Prod”

This configures Apache to return only Apache as product in the server response header on very page request, suppressing OS, major and minor version info.

ServerSignature

This directive lets you add a line containing the server version and virtual host name to server-generated pages. It is recommended to set it to OFF and Set to "EMail" to also include a mailto: link to the ServerAdmin.

Options: On | Off | EMail

“ServerSignature Off”

This instructs Apache not to display a trailing footer line under server-generated documents, which displays server version number, ServerName of the serving virtual host, email setting etc..


Courtesy:http://josephamrithraj.wordpress.com/2010/09/16/advanced-ssl-configuration-on-ibm-http-server-restrict-unused-http-methods-and-verbose-http-headers/

More about → Advanced SSL configuration on IBM Http Server – Restrict unused HTTP methods and Verbose HTTP headers

Advanced SSL configuration on IBM Http Server – Client Authentication and Ciphers

The Advanced SSL Configuration settings are

Client Authentication
Setting Ciphers
SSL for multiple IP virtual Hosts
Client Authentication:

If you enable client authentication, the server validates clients by checking for trusted certificate authority, Known as CA root certificates in the local key database. To enable client authentication, you need to use SSLClientAuth directive. The options to use with this stanza are:

None – The server requests no client certificate from the client.
Optional – The server requests, but does not require, a client certificate. If presented, the client certificate must prove valid.
Required – The server requires a valid certificate from all clients and returns a 403 status code if no certificate is present.
Required_reset – The server requires a valid certificate from all clients, and if no certificate is available, the server sends an SSL alert to the client. This enables the client to understand that the SSL failure is client-certificate related, and will cause browsers to re-prompt for client certificate information on subsequent access. make sure you have GSKit version 7.0.4.19 or later when you choose this option.
For example, If i want all the clients to be authenticated, then i need to add the following stanza
SSLClientAuth required

Ciphers

We set the cipher specification to use during secure transactions. The specified cipher specifications validate against the level of the Global Security Kit (GSK) toolkit that is installed on your system. Invalid cipher specifications cause an error to log in the error log. If the client issuing the request does not support the ciphers specified, the request fails and the connection closes to the client. IBM HTTP Server has a built-in list of cipher specifications to use for communicating with clients over Secure Sockets Layer (SSL). The actual cipher specification that is used for a particular client connection is selected from those which are supported by both IBM HTTP Server and the client.

Some cipher specifications provide a weaker level of security than others, and might need to be avoided for security reasons. Some of the stronger cipher specifications are more computationally intensive than weaker cipher specifications and might be avoided if required for performance reasons. When an SSL connection is established, the client (web browser) and the web server negotiate the cipher to use for the connection. The web server has an ordered list of ciphers, and the first cipher in that list which is supported by the client will be selected.

IBM HTTP Server supports the following SSL ciphers: SSLv3 and TLS and SSLv2

IBM recommends the following setting, keeping in mind both strong security and performance

## SSLv3 128 bit Ciphers
SSLCipherSpec SSL_RSA_WITH_RC4_128_MD5
SSLCipherSpec SSL_RSA_WITH_RC4_128_SHA

## FIPS approved SSLV3 and TLSv1 128 bit AES Cipher
SSLCipherSpec TLS_RSA_WITH_AES_128_CBC_SHA

## FIPS approved SSLV3 and TLSv1 256 bit AES Cipher
SSLCipherSpec TLS_RSA_WITH_AES_256_CBC_SHA

## Triple DES 168 bit Ciphers
## These can still be used, but only if the client does
## not support any of the ciphers listed above.
SSLCipherSpec SSL_RSA_WITH_3DES_EDE_CBC_SHA

## The following block enables SSLv2. Excluding it in the presence of
## the SSLv3 configuration above disables SSLv2 support.

## Uncomment to enable SSLv2 (with 128 bit Ciphers)
#SSLCipherSpec SSL_RC4_128_WITH_MD5
#SSLCipherSpec SSL_RC4_128_WITH_SHA
#SSLCipherSpec SSL_DES_192_EDE3_CBC_WITH_MD5
View the Ciphers which the server uses for Secure transactions

Set the LogLevel to info in the configuration file. Look in the error log for messages in this format: TimeStamp info_message mod_ibm_ssl: Using Version 2/3 Cipher: longname|shortname. The order that the cipher specifications are displayed in the error log from top to bottom represents the attempted order of the cipher specifications.

View the Ciphers were used for negotiating a connection

You can use the following LogFormat directive to view and log the SSL cipher negotiated for each connection:

LogFormat “%h %l %u %t \”%r\” %>s %b \”SSL=%{HTTPS}e\” \”%{HTTPS_CIPHER}e\” \”%{HTTPS_KEYSIZE}e\” \”%{HTTPS_SECRETKEYSIZE}e\”" ssl_common

CustomLog logs/ssl_cipher.log ssl_common

This logformat will produce an output to the ssl_cipher.log that looks something like this:

127.0.0.1 – - [01/Sep/2010:00:02:05 -0800] “GET / HTTP/1.1″ 200 1582 “SSL=ON” “SSL_RSA_WITH_RC4_128_MD5″ “128″ “128″

SSL for multiple IP virtual hosts

When you do not define an SSL directive on a virtual host, the server uses the directive default. You can define different (SSL) options for various virtual hosts. To enable SSL:

Specify the SSLEnable directive on the virtual host stanza in the configuration file, to enable SSL for a virtual host.
Specify a Keyfile directive and
Any SSL directives you want enabled for that particular virtual host.
Restart the server.
With all the above security options enabled, your virtual host may look like this:



SSLEnable

Keyfile keyfile.kdb

SSLCientAuth required

## SSLv3 128 bit Ciphers

SSLCipherSpec SSL_RSA_WITH_RC4_128_MD5

SSLCipherSpec SSL_RSA_WITH_RC4_128_SHA

## FIPS approved SSLV3 and TLSv1 128 bit AES Cipher

SSLCipherSpec TLS_RSA_WITH_AES_128_CBC_SHA

## FIPS approved SSLV3 and TLSv1 256 bit AES Cipher

SSLCipherSpec TLS_RSA_WITH_AES_256_CBC_SHA

## Triple DES 168 bit Ciphers

## These can still be used, but only if the client does not support any of the ciphers listed above.

SSLCipherSpec SSL_RSA_WITH_3DES_EDE_CBC_SHA

## The following block enables SSLv2.
## Excluding it in the presence of the SSLv3 configuration above disables SSLv2 support.

## Uncomment to enable SSLv2 (with 128 bit Ciphers)

#SSLCipherSpec SSL_RC4_128_WITH_MD5

#SSLCipherSpec SSL_RC4_128_WITH_SHA

#SSLCipherSpec SSL_DES_192_EDE3_CBC_WITH_MD5



Courtesy:http://josephamrithraj.wordpress.com/2010/09/04/advanced-ssl-configuration-on-ibm-http-server-client-authentication-and-ciphers/

More about → Advanced SSL configuration on IBM Http Server – Client Authentication and Ciphers

How to collect performance data on Linux

Collect the following information when high CPU consumption is with IBM Java process:
Enable garbage collection trace to see whether Java garbage collection is thrashing if possible. If you want to enable Java garbage collection trace on IBM WebSphere Application Server, please refer to the following document: Enabling verbose garbage collection (verbosegc) in WebSphere application Server


Run the following command:

top -d delaytime -c -b > top.log

Where delaytime is the number of seconds to delay. This must be 60 seconds or greater, depending on how soon the failure is expected.


Create a script file, vmstat.sh with the following content:

#vmstat.sh
#output file name
VMSTAT_LOG=$1
LIMIT=288
#sleep for 5 miniutes
SLEEP_TIME=300
while true
do
i=0
echo >$VMSTAT_LOG
while [ $i -le "$LIMIT" ];
do
date >> $VMSTAT_LOG;
vmstat 5 12 >> $VMSTAT_LOG;
i=`expr $i + 1`;
sleep $SLEEP_TIME;
done
done

Create a script, ps.sh with the following content:

#ps.sh
#output file name
PS_LOG=$1
LIMIT=288
#sleep for 5 miniutes
SLEEP_TIME=300
while true
do
i=0
echo >$PS_LOG
while [ $i -le "$LIMIT" ];
do
date >> $PS_LOG;
ps -eLf >> $PS_LOG;
i=`expr $i + 1`;
sleep $SLEEP_TIME;
done
done

Run the scripts:

./ps.sh ps_eLf.log
./vmstat.sh vmstat.log

Notes: . The scripts ps.sh and vmstat.sh, as provided, roll over every 24 hours. . You might need to modify the scripts to meet your needs. . The preceding scripts will run forever. After the error condition is reached, you will have to terminate them.


When high CPU consumption occurs, collect the following logs:

netstat -an > netstat1.out


If the Web server is remote, run the following on the Web server system:

netstat -an > netstatwebserver1.out


Run the following:

kill -3 [PID_of_problem_JVM]


The kill -3 commands create javacore*.txt files

Note: If you are not able to determine which JVM process is experiencing the high CPU usage then you should issue the kill -3 PID for each of the JVM processes.



Wait two minutes.


Run the following:

kill -3 [PID_of_problem_JVM]


Wait two minutes.


Run the following:

kill -3 [PID_of_problem_JVM]


Wait two minutes.


Run the following:

netstat -an > netstat2.out



If the Web server is remote, run the following on the Web server system:

netstat -an > netstatwebserver2.out



If you are unable to generate javacore files, then perform the following:

kill -11 [PID_of_problem_JVM]

WARNING: kill -11 will terminate the JVM process, produce a core file, and possibly a javacore.


Review all output files and collect the following files for IBM Performance Analysis Tool for Java for Linux


ps_eLf.log
javacore*.txt files

Courtesy:http://wasissues.blogspot.com/

More about → How to collect performance data on Linux

Migrating from Websphere 6.0 to Websphere 7.0

The application I manage at work is a client/server application written entirely in java. My company for years has been an IBM shop, so we have a large Websphere presence which is where the server is deployed. The application had been running on Websphere 5.1 for a few years and was fairly recently migrated to Websphere 6.0 to remain on a supported version of Websphere. Because the end of life for Websphere 6.0 is September 2010, we’re starting to plan for another upgrade now (we have major releases in January and June, so we’re targeting the June ’10 release for the upgrade). Websphere 6.1, if IBM holds to it’s pattern of every 2 years or so will remain supported until September 2012, however there’s no current end of life date documented yet (link). To get the longest life possible, I’m looking at Websphere 7.0 as the target platform for our June ’10 upgrade.

We have RAD 7.5 in house and a couple members of my team have installed it. I’ve been working on getting a local WAS 7.0 server up and running and getting our app deployed on it. There are some major differences between Websphere 5.1/6.0 and 7.0. I won’t go into details as those are readily available on IBM’s website, but would like to share some of my observations and pain so far.

Application Background: The server side of our application is effectively broken down into 2 pieces. One component is what we call a provision server that is essentially a cache of configuration data read from DB2. This configuration information contains rules which drive how the second component operates. The second component is the main workhorse app which receives a request, and creates a response based on configuration data retrieved from the provision server (if necessary) and data retrieved from any number of other applications we interface with. We have 2 provision server jvms for load balancing and fail over and roughly 20 app engine jvms spread across 2 data centers (the app engine hosts roughly 3500 end users and we target 200 users per jvm…roughly).

Unsolved Problem 1 – Remote EJB calls across separate local jvm/profiles doesn’t work: Now that you have a high level view of our applications architecture, here’s my first dilema which I haven’t found a solution to. Websphere now has the concept of profiles. Basically a profile equates to a jvm instance. It’s a little more than that, but that’s a good enough understanding for now. So if you want 2 distinct/separate JVMs configured differently, you would need to create 2 profiles and create servers associated with each profile. In all our lifecycles, we have distinct jvms setup for provision and app engine – we don’t cluster the app engine with the provision server because we want our dev/test lifecycles to mirror production, and production is separate because we don’t want a 1:1 correlation of provision server to app engine as the provision server is memory intensive and 2 jvms can handle the entire app engine load very effectively. So I want to replicate that with my RAD 7.5 setup – 1 app engine jvm and 1 provision server jvm running locally within RAD 7.5. That requires 2 separate profiles to be created, then a server defined and associated for each one. No problem. Where I run into problems is at runtime. The app engine makes remote (cross-jvm) EJB calls to the provision server. That requires a JNDI lookup of an EJB remote home object. For some reason, jvm 1 cannot see any JNDI objects that are stored in jvm 2. When I do an initial context and dump out the contents, all I ever get are the local JVM’s name server items. But if I point the local server at one of our test lifecycle provision servers, it sees those just fine. I have no idea why 1 local jvm can’t access another local jvm’s name server. I’m not sure if it’s because of the base version of Websphere that’s running, or some other limitation of the development environment, but that is one hurdle I can’t get over. So my workaround is to just deploy the provision server and app engine in the same jvm as local ejb calls work just fine.

Solved Problem 2 – creating a secure socket for an outbound ssl SOAP request: The app engine is a portal of sorts. It will call any number of external systems to retrieve data and aggregate that data as needed based on the request. There are several system we currently interface and several protocols we use to do so…SOAP over SSL, EJB, JDBD for example. We use apache soap (old, but still works) to call several external systems, one of which is the main system we interface with. In Websphere 5.1 and 6.0, we set our own JKS truststore for the request using the javax.net.ssl.truststore property. This truststore contains the SSL certificates of our target URL. It just worked. Now we move to Websphere 7.0 and the same requests which work in a local WAS 6.o server no longer work. After much digging and reading of documentation, it turns out WAS 6.1 (and 7.0) changed how SSL security was handled. Long story short, when WAS sees a secure socket being created, it assumes responsibility for securing that connection (Big Brother?) instead of letting you do your own thing. Now, there are ways around it, but the point is it is NOT backwards compatible. The quick fix for this was to put the SSL certs in Websphere’s default truststore (go to the admin console, under security and then ssl configuration and you can find a whole bunch of related config). There are several articles on this and I highly recommend reading the Websphere Application Server V7.0 Security Guide for background on this. It is extremely helpful.

More Here

Courtesy:http://bairdblog.wordpress.com/

More about → Migrating from Websphere 6.0 to Websphere 7.0