Creating and Managing Threads in Java
Threading in the early days of Java use to be a pain. Since Java 1.5 and the introduction of the ExecutorService, it is much easier to start up and manage them. Code that you want executed in a separate thread must be encapsulated in a class that implements Runnable. An instance of that class can then be passed to an ExecutorService that will handle its execution. Below is a trivial example but it demonstrates how to use the ExecutorService and thread pools.
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
import org.apache.log4j.Logger;
public class ExecutorServiceExample {
public static void main (String args[]) {
final Logger logger = Logger.getLogger(ExecutorServiceExample.class);
ExecutorService threadExecutor = Executors.newFixedThreadPool( 10 );
for (int i = 0; i < 20; i++) {
threadExecutor.execute(new Runnable() {
@Override
public void run() {
try {
logger.info("This thread is going to sleep");
Thread.sleep((long)(Math.random() * 10000));
} catch (InterruptedException e) {
logger.error("Oh noes... interruptted thread?!",e);
}
}
});
}
threadExecutor.shutdown();
try {
threadExecutor.awaitTermination(5, TimeUnit.MINUTES);
logger.info("All threads done");
} catch (InterruptedException e) {
logger.error("Oh noes... interruptted thread?!",e);
}
}
}
In the example above, I created an ExecutorService that manages a pool of 10 threads. It then executes 20 anonymous classes which log an info message and sleep between 0 and 10 seconds. After that, it waits for the threads to finish executing and then logs another info message.
As you can see from the output below, the 10 threads in the pool execute one after the other. After that though, as different threads complete execution before others, they begin executing out of order due to their availability.
Output: 2011-09-13 22:07:19,511 [pool-1-thread-1] INFO ExecutorServiceExample - This thread is going to sleep
2011-09-13 22:07:19,512 [pool-1-thread-3] INFO ExecutorServiceExample - This thread is going to sleep
2011-09-13 22:07:19,511 [pool-1-thread-2] INFO ExecutorServiceExample - This thread is going to sleep
2011-09-13 22:07:19,512 [pool-1-thread-4] INFO ExecutorServiceExample - This thread is going to sleep
2011-09-13 22:07:19,512 [pool-1-thread-5] INFO ExecutorServiceExample - This thread is going to sleep
2011-09-13 22:07:19,512 [pool-1-thread-6] INFO ExecutorServiceExample - This thread is going to sleep
2011-09-13 22:07:19,512 [pool-1-thread-7] INFO ExecutorServiceExample - This thread is going to sleep
2011-09-13 22:07:19,513 [pool-1-thread-8] INFO ExecutorServiceExample - This thread is going to sleep
2011-09-13 22:07:19,513 [pool-1-thread-9] INFO ExecutorServiceExample - This thread is going to sleep
2011-09-13 22:07:19,513 [pool-1-thread-10] INFO ExecutorServiceExample - This thread is going to sleep
2011-09-13 22:07:19,619 [pool-1-thread-1] INFO ExecutorServiceExample - This thread is going to sleep
2011-09-13 22:07:21,424 [pool-1-thread-6] INFO ExecutorServiceExample - This thread is going to sleep
2011-09-13 22:07:21,815 [pool-1-thread-2] INFO ExecutorServiceExample - This thread is going to sleep
2011-09-13 22:07:22,567 [pool-1-thread-10] INFO ExecutorServiceExample - This thread is going to sleep
2011-09-13 22:07:23,474 [pool-1-thread-10] INFO ExecutorServiceExample - This thread is going to sleep
2011-09-13 22:07:23,863 [pool-1-thread-9] INFO ExecutorServiceExample - This thread is going to sleep
2011-09-13 22:07:24,473 [pool-1-thread-8] INFO ExecutorServiceExample - This thread is going to sleep
2011-09-13 22:07:24,569 [pool-1-thread-7] INFO ExecutorServiceExample - This thread is going to sleep
2011-09-13 22:07:25,506 [pool-1-thread-5] INFO ExecutorServiceExample - This thread is going to sleep
2011-09-13 22:07:27,492 [pool-1-thread-4] INFO ExecutorServiceExample - This thread is going to sleep
2011-09-13 22:07:34,308 [main] INFO ExecutorServiceExample - All threads done
Java SFTP
Believe it or not, there are very few options currently when it comes to implementing SFTP in Java. I believe that the Apache Mina project will eventually provide support for it (http://mina.apache.org/sshd/index.html) however, the project is still at version 0.5 and I have not found any good documentation for it. jCraft’s JSch package is the most straightforward library that I have used so far and in this article, I am going to provide a sample implementation.
JSch: http://www.jcraft.com/jsch/
JSch download: http://sourceforge.net/projects/jsch/files/jsch.jar/0.1.44/jsch-0.1.44.jar/download
Something to keep in mind here is that SFTP is not FTPS. SFTP is SSH FTP hence why I am hoping that Apache Mina will eventually support it. FTPS is FTP SSL which you can interact with using Apache FtpServer (http://mina.apache.org/ftpserver/). Until there is a mature Apache project, I recommend JSch. Actually, I recommend that you first try using SSH Fuse (http://fuse.sourceforge.net/sshfs.html) to mount your SFTP account on your file system and if you cannot set it up, then use JSch. import com.jcraft.jsch.ChannelSftp;
import com.jcraft.jsch.JSch;
import com.jcraft.jsch.Session;
import com.jcraft.jsch.SftpProgressMonitor;
public class SFTPExample {
public static void main(String args[]) throws Exception {
String user = "enter user name";
String password = "enter password";
String host = "enter host";
int port = 22;
String knownHostsFilename = "/path/to/.ssh/known_hosts";
String sourcePath = "/path/to/test_file";
String destPath = "test_file";
JSch jsch = new JSch();
jsch.setKnownHosts(knownHostsFilename);
Session session = jsch.getSession(user, host, port);
session.setPassword(password);
session.connect();
ChannelSftp sftpChannel = (ChannelSftp) session.openChannel("sftp");
sftpChannel.connect();
System.out.println("Uploading test file");
sftpChannel.put(sourcePath, destPath, new ExampleProgressMonitor());
System.out.println("Downloading test file");
sftpChannel.get(destPath, sourcePath + ".new", new ExampleProgressMonitor());
System.out.println("Remove test file");
sftpChannel.rm(destPath);
sftpChannel.exit();
session.disconnect();
}
}
class ExampleProgressMonitor implements SftpProgressMonitor {
private double count;
private double max;
private String src;
private int percent;
private int lastDisplayedPercent;
ExampleProgressMonitor() {
count = 0;
max = 0;
percent = 0;
lastDisplayedPercent = 0;
}
public void init(int op, String src, String dest, long max) {
this.max = max;
this.src = src;
count = 0;
percent = 0;
lastDisplayedPercent = 0;
status();
}
public boolean count(long count) {
this.count += count;
percent = (int) ((this.count / max) * 100.0);
status();
return true;
}
public void end() {
percent = (int) ((count / max) * 100.0);
status();
}
private void status() {
if (lastDisplayedPercent <= percent - 10) {
System.out.println(src + ": " + percent + "% " + ((long) count) + "/" + ((long) max));
lastDisplayedPercent = percent;
}
}
}
Using Dependency Injection To Incorporate A/B Testing Into Your Applications
I posted this article to my company’s tech blog a few weeks back. I am reposting it here because… well… I wrote it :-)
http://tech.gilt.com/post/8391205906/using-dependency-injection-to-incorporate-a-b-testing
If you work at an e-commerce company, chances are you’ve probably come across the term “A/B testing”. We all know that it has something to do with testing out new features on users and seeing which ones are “better”. A/B testing is really the practice of comparing the effect that a feature (“treatment” in statistical terms) has on different groups of users. For instance, you might be developing a new navigation bar for your Website. You can see if it improves how long users stay on your site by testing it on a group of users (A) and comparing them to users in another group (B) that are using the original navigation bar. Again in statistical lingo, group B would be considered your “control group”. It’s important to note that you can test a variety of navigation bars here and not violate the principles behind A/B testing since your treatment is the type of bar you are exposing.
Now you are probably wondering how dependency injection (DI) factors into A/B testing. DI is the design principle where your objects are assigned references to their dependencies at runtime. Generally speaking, objects using DI refer to their dependencies through interfaces or super classes so they are indifferent to how those dependencies are actually implemented and instantiated. There are a few frameworks in Java that leverage DI. For the work that we do at Gilt, we use Java Spring. The example in this post is in Spring but it’s general enough that it can be applied to other frameworks.
So now that we know what DI is, let’s apply it to make A/B testing a part of your applications. In A/B testing, groups of users are exposed to a given treatment. That means we need to have a mapping between groups of users and the treatment that they are receiving. Java Spring makes that very easy; in your Spring configuration, you can create such a mapping using the map tag. For example:
<util:map id="ABTestMapping">
<entry>
<key>
<value>group 1</value>
</key>
<ref bean="baseline_nav" />
</entry>
<entry>
<key>
<value>group 2</value>
</key>
<ref bean="test_nav1" />
</entry>
<entry>
<key>
<value>group 3</value>
</key>
<ref bean="test_nav2" />
</entry>
</util:map>
<bean id="baseline_nav" class="com.gilt.examples.BaselineNavBar"/>
<bean id="test_nav1" class="com.gilt.examples.NewNavBar1"/>
<bean id="test_nav2" class="com.gilt.examples.NewNavBar2"/>
So in this map, our keys are different test groups of users and their corresponding values are the different navigation bar we will expose to them. This example is a bit contrived but it illustrates the point quite well. Instead of having to code A/B testing into your application, you can externalize it in a configuration. Java Spring instantiates this map and your application can use it to determine which navigation bar a user should see. A nifty trick in Spring is to use the import tag in your configuration.
<import resource="file:test_buckets.xml"/>
This tag allows you to put your A/B testing configuration into a separate file which is very powerful. Doing so allows you to change which groups receive various treatments without having to recompile or redeploy your code.
Simple. Fast. Very Useful.
Using the ClassLoader to access files in your classpath
Here is a quick tid-bit about how to access files in Java that are in your classpath. For example, suppose you have a text file that you want to parse and it is packaged in your jar. You can access it through the ClassLoader using either the method getResource(), that returns an instance of URL, or getResourceAsStream, that returns an instance of InputStream. Below is a simple coding example:
import java.util.Scanner;
public class ExampleFileLoader {
public static void main (String args[]) {
Scanner scan = new Scanner (ExampleFileLoader.class.getClassLoader().getResourceAsStream("test_file.txt"));
while (scan.hasNextLine())
System.out.println(scan.nextLine());
}
}
JDBC fetch size and Postgresql
Every now and then, you need to pull a massive amount of data from a database, more than can fit into memory reasonably. To this end, you can set the fetch size for your statement so that the database driver will pull back more manageable chunks of data. For example, if you set the fetch size to 100, the driver should pull back 100 row chunks. A new chunk is pulled when needed as you iterator over the corresponding result set.
Now setting the fetch size for most databases should be sufficient however, not for Postgresql. If you want to enable fetching, you also have to turn auto commit off as well. The reason has to deal with how Postgresql fetches chunks of data; it requires a transaction block and if auto commit is on, it cannot get one. You would think that the Postgresql JDBC driver would be smart enough to turn auto commit off if the fetch size is set to anything other than 0. Anyhow, as per usual, some example code is below:
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.Properties;
public class BigQueryExample {
public static void main (String args[]) {
Properties connectionProps = new Properties();
connectionProps.put("user", "enter username");
connectionProps.put("password", "enter password");
Connection conn = null;
Statement statement = null;
ResultSet rs = null;
try {
conn = DriverManager.getConnection("enter url", connectionProps);
conn.setAutoCommit(false);
statement = conn.createStatement(ResultSet.TYPE_FORWARD_ONLY, ResultSet.CONCUR_READ_ONLY);
statement.setFetchSize(100);
rs = statement.executeQuery("SELECT name FROM giant_table_of_users");
while (rs.next()) {
System.out.println("Hi " + rs.getString("name"));
}
}
catch (SQLException se) {
System.err.println("some sort of jdbc error encountered");
throw new RuntimeException(("some sort of jdbc error encountered", se);
}
finally {
rs.close();
statement.close();
conn.close();
}
}
}
External Spring Config Files
Most of the time when you are creating a Spring app, you end up packaging the XML config files with the war/jar. Sometimes though, it is quite beneficial to have a configuration file external to your built package. That allows you to configure your Spring app without having to rebuild or redeploy it; trust me, your system engineers/admins will love you for that. Using an external spring config is quite easy. You can use the import tag.
For example:
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd">
<import resource="file:/path/to/external/config.xml"/>
</beans>
The above Spring config will import /path/to/external/config.xml. Having an external file will allow you to configure beans without having to rebuilding your main war/jar. Furthermore, an external Spring config allows you to create map configurations, something that you cannot do in traditional unix style property files. Below is an example map instance you can create in your external config:
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:util="http://www.springframework.org/schema/util"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util-3.0.xsd">
<util:map id="sampleMap">
<entry>
<key>
<value>Hello</value>
</key>
<value>World</value>
</entry>
</util:map>
</beans>
Programatically logging a user out in Spring Security
So I use Spring Security to handle user authentication in most of my Web applications. Every now and then, you need to log a user out programmatically. For example, users perform some sort of operation that redirects to a success page and logs them out. Logging a user out is quite simple. You need use the logout method for the relevant LogoutHandlers in your application. You are always going to have to use the SecurityContextLogoutHandler. I generally use the “remember me token” so I also have to use the PersistentTokenBasedRememberMeServices. Below is a sample method that you could have in your controller.
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import org.springframework.security.Authentication;
import org.springframework.security.context.SecurityContextHolder;
import org.springframework.security.ui.logout.SecurityContextLogoutHandler;
import org.springframework.security.ui.rememberme.PersistentTokenBasedRememberMeServices; import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.RequestMapping;
@Controller
public class SampleController {
@RequestMapping(value="/some_page.htm")
public String somePage (HttpServletRequest request, HttpServletResponse response) {
/* some business logic code */
Authentication auth = SecurityContextHolder.getContext().getAuthentication();
if (auth != null){
new SecurityContextLogoutHandler().logout(request, response, auth);
new PersistentTokenBasedRememberMeServices().logout(request, response, auth);
}
}
}
Rotating log files in log4j
In Java, it is pretty standard to use log4j to handle your logging. In a production environment, you rarely want to all your log data to be kept in a single file. If you did, eventually that file would become very large and difficult to use especially when you are looking for specific error messages.
A good practice is to employ rotating log files. Old log data gets moved into other files and really old log data gets removed automatically. It is pretty easy to setup this style of logging up in log4j. You need to use a RollingFileAppender.
log4j.rootCategory=WARN, stdout, rollFile
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d %p [%c] - <%m>%n
log4j.appender.rollFile=org.apache.log4j.RollingFileAppender
log4j.appender.rollFile.File=/web/logs/sample.log
log4j.appender.rollFile.MaxFileSize=10MB
log4j.appender.rollFile.MaxBackupIndex=5
log4j.appender.rollFile.layout=org.apache.log4j.PatternLayout
log4j.appender.rollFile.layout.ConversionPattern=%d %p [%c] - <%m>%n
In the above example, I send log data to both the console and a rotation log file.
Adding authentication to an Apache CXF Web Service Client
In my last post, I provided steps for creating an Apache CXF Web Service Client. Now that client only works with Web Services that do not require you to authenticate. In this post, I will provide an example of how add authentication.
Step 1. Create a CXF Web Service client
Step 2. Create a CallbackHandler. The code below is actually from a previous post pertaining to authenticating an Axis 1.4 Web Service Client:
import java.io.IOException;
import javax.security.auth.callback.Callback;
import javax.security.auth.callback.CallbackHandler;
import javax.security.auth.callback.UnsupportedCallbackException;
import org.apache.commons.configuration.Configuration;
import org.apache.commons.configuration.ConfigurationException;
import org.apache.commons.configuration.PropertiesConfiguration;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.ws.security.WSPasswordCallback;
public class SamplePWCallback implements CallbackHandler {
private Configuration _config;
private static Log _logger = LogFactory.getLog(SamplePWCallback.class);
public SamplePWCallback () {
super();
_logger.info("Initializing Sample PW Callback");
_config = null;
try {
_config = new PropertiesConfiguration("sample.properties");
}
catch (ConfigurationException ce) {
_logger.error("Error loading configuration file", ce);
}
}
public void handle(Callback[] callbacks) throws IOException,
UnsupportedCallbackException {
for (int i = 0; i < callbacks.length; i++) {
if (callbacks[i] instanceof WSPasswordCallback) {
WSPasswordCallback pc = (WSPasswordCallback)callbacks[i];
// set the password given a username
if (_config.getString("username").equals(pc.getIdentifier())) {
pc.setPassword(_config.getString("password"));
}
}
else {
throw new UnsupportedCallbackException(callbacks[i], "Unrecognized Callback");
}
}
}
}
Step 3. Update your client to use your CallbackHandler. For the purposes of demonstration, let’s assume the global weather Web Service that I used in the previous blog post required authentication. The client would have to be updated to look similar to the sample one below:
import java.net.MalformedURLException;
import java.net.URL;
import java.util.HashMap;
import java.util.Map;
import net.webservicex.*;
import org.apache.cxf.binding.soap.saaj.SAAJOutInterceptor;
import org.apache.cxf.endpoint.Client;
import org.apache.cxf.endpoint.Endpoint;
import org.apache.cxf.frontend.ClientProxy;
import org.apache.cxf.interceptor.LoggingInInterceptor;
import org.apache.cxf.interceptor.LoggingOutInterceptor;
import org.apache.cxf.ws.security.wss4j.WSS4JOutInterceptor;
import org.apache.ws.security.handler.WSHandlerConstants;
import org.apache.ws.security.WSConstants;
public class HelloCXF {
public static void main (String args[]) {
try {
GlobalWeather service = new GlobalWeather(new URL("file:globalweather.wsdl"));
GlobalWeatherSoap weatherSoap = service.getGlobalWeatherSoap();
Client client = ClientProxy.getClient( weatherSoap );
Endpoint endPoint = client.getEndpoint();
Map<String, Object> outProperties = new HashMap<String, Object>();
outProperties.put( WSHandlerConstants.ACTION,
WSHandlerConstants.USERNAME_TOKEN );
outProperties.put( WSHandlerConstants.USER, _config.getString("username"));
outProperties.put( WSHandlerConstants.PASSWORD_TYPE, WSConstants.PW_TEXT );
outProperties.put( WSHandlerConstants.PW_CALLBACK_CLASS,
SamplePWCallback.class.getName() );
WSS4JOutInterceptor wssOut = new WSS4JOutInterceptor( outProperties );
endPoint.getOutInterceptors().add( wssOut );
endPoint.getOutInterceptors().add( new SAAJOutInterceptor() );
endPoint.getInInterceptors().add( new LoggingInInterceptor() );
endPoint.getOutInterceptors().add( new LoggingOutInterceptor() );
System.out.println("Weather for New York");
System.out.println(weatherSoap.getWeather("New York", "United States"));
}
catch (MalformedURLException mue) {
System.err.println("problem with the wsdl url");
System.exit(1);
}
}
}
Using the Spring Expression Language with static properties
One of the things that I love about Spring 3 is the Spring Expression Language (SPEL). At first glance, SPEL does not appear to give you that much; with it, you can execute Java expressions in your Spring config files. This capability, however, is awesome when have properties that take values defined by static variables. For example, how days of the week are defined in the Calendar class:
Calendar.MONDAY
Calendar.TUESDAY
Calendar.WEDNESDAY
Calendar.THURSDAY
Calendar.FRIDAY
Calendar.SATURDAY
Calendar.SUNDAY
Prior to SPEL, if you wanted to set an integer property to Calendar.MONDAY, you had to know and specify the actual integer value for it. Using SPEL, can simply use the following snippet instead; you need to add the org.springframework.expression-3.X.X.jar to your classpath as well.
<property name="dayOfWeek" value="<entry key="#{T(java.util.Calendar).MONDAY}" />
