Java 9: JShell, a Read-Eval-Print Loop (REPL)

What’s a REPL?

A Read-Evaluate-Print-Loop, or, REPL, is a command line interface for interacting with a programming language. In the words of JEP 222, the new Java REPL (JShell) is to “provide an interactive tool to evaluate expressions of the Java programming language, together with an API so that other applications can leverage this functionality.”

Why Does Java Need This?

According to JEP 222: JShell, “The number one reason schools cite for moving away from Java as a teaching language is that other languages have a ‘REPL’ and have far lower bars to an initial ‘Hello, world!’ program.” If we want Java to have a bright future (and don’t we all?) then Java has to compete with other languages (Scala, Groovy, Clojure, Python, Ruby, Haskell…) in academia that have REPL’s and a low barrier to entry.

The REPL is a good feature for students, but what about professional developers? Ask a professional Scala developer, and it becomes clear that the REPL is an iterative development tool with even faster feedback than TDD. Additionally it lets us experiment with third-party libraries with less ceremony than setting up a new project. Finally, it allows us to execute arbitrary Java code from the command line, bringing Java much closer in usability to a scripting language. So at a professional level, the REPL aims to make us more productive in a very direct way.

What Features Does It Have?

JShell maintains history, has a built-in editor, has tab-completion, saves and loads code, and has automatic addition of semicolons. It has forward references (except in method signatures) so we can reference a variable or method before it’s defined. To see everything it can do, call jshell -help from the command line, or /help from within jshell.

How Else Can I Use It?

JShell is pretty flexible and has an API, so people are likely to come up with creative uses for it.

People may use it like a Java oriented shell for doing regular work like we do with the bash shell right now. I can imagine Java moving at least partially into the scripting space, so people have another language at their disposal to work from the command line with files and data, connect to servers or databases, retrieve and manipulate data, etc.

The fact that JShell includes an API means we are likely to see this included with IDE’s, providing a Java “scratch pad” for our daily work. The API also means it could potentially integrate with build tools and increase the flexibility of build scripts.

Personally I would like to see a Github Gist browser and downloader so it’s easy to experiment with other people’s ideas!

Example: Try A Library

We can load jars or classes to try them out in JShell, just specify a jar file when calling jshell –class-path from the command line, or call /classpath from within the shell.

To try out a library like Google Guava, call this from the command line:

wget http://central.maven.org/maven2/com/google/guava/guava/19.0/guava-19.0.jar
jshell --class-path guava-19.0.jar 

then inside jshell, we can type in some code like this:

import com.google.common.base.Optional;
Optional b =  Optional.of(new Integer(10));
b.get()

// java 8's Optional has .ofNullable()
Optional a = Optional.fromNullable(null);
a.or(5);

Example: Pretend Java Is A Scripting Language

I think .jsh makes a good file extension for Java snippets intended for use with JShell. Let’s put this in a file called hello.jsh:

System.out.println("hello world!")
/exit

Then from the command line

jshell hello.jsh

Yay, a compiled language that runs like a script!

Example: Load Code At Startup

What if we wanted Linux bash shell-like capabilities inside JShell? We could write code to accomplish that capability, and have that code loaded at startup time.

Let’s put this code in a file called bash.jsh

// a startup file completely replaces the default
// so need to bring in the default imports yourself

import java.util.*
import java.io.*
import java.math.*
import java.net.*
import java.util.concurrent.*
import java.util.prefs.*
import java.util.regex.*

void ls() {
  File cur = new File(".");
  for(String s : cur.list())
    System.out.println(s);
}

Then from the command line:

jshell --startup bash.jsh

Now when we type “ls()” from within jshell we get the file listing of the current directory. It would probably be a bit of work, but in theory we could replicate a complete shell environment. But, you know, we already have a shell.🙂 What we will see in practice is more likely to be freely available snippets for working with data and code at a higher level.

Conclusion

JShell has a lot of potential to change the way we learn and use Java. I have high hopes for its expanded use in Java 9!

How will you use JShell?

Leave a comment

Filed under Java 9

The Business Case For Zero Downtime Deployments

This post describes the business advantages of zero-downtime, (a.k.a. blue-green) deployments. If you are not familiar with blue-green deployments, you can read the excellent Martin Fowler article about blue-green deployments. But in a nutshell this amounts to the idea of releasing to production by deploying a release to a separate production environment and behind the scenes gradually redirecting user activity to the new environment from the old one.

Why Do I Want This?

  • Eliminate downtime: Done correctly, your users will literally see zero downtime. Many organizations do deployments at midnight local time so that they can takenthe entire service offline “when nobody’s using it.” But if your software is successful in a global environment (or if your target demographic’s usage time is split between midnight and mid-day), there is no time when “nobody’s using it” and the best time for one group of users may be the worst time for another.
  • Increase support: Production deployment is potentially the single most dangerous activity your team does. Instead of midnight deployments when everybody is exhausted or asleep, deploy during business hours when the entire team is available and alert.
  • Reduce risk: allows easy and safe rollback, if something unexpected happens with your release, you can immediately and safely roll back to the last version by simply directing user traffic back to the previous environment
  • Provide for staging: when the new environment is active, the previous environment becomes the staging environment for the next deployment. If you didn’t have a staging environment, you should probably have one anyway.
  • Hot backup for disaster recovery: after deploying and we’re satisfied that it is stable, we can deploy the new release to the previous environment too. This gives us a hot backup in case of disaster.

What’s The Catch?

Zero-downtime deployment isn’t free: it requires a certain diligence and maturity of process and devops, but not any more than you really need to play with the big kids in the world of software engineering anyway. Details at the engineering level to follow in a future post!

Conclusion

The benefits of this kind of deployment are actually huge. If you can deploy with zero downtime once per month, why not once per sprint? Why not every day? Why not as you finish features and fix bugs so that your customers see their bug fixes and feature requests in a fraction of the time?

If you can do deployments this way, your users with thank you. And your boss will thank you.🙂

Leave a comment

Filed under Software Engineering

Java 9: The New HttpClient

With JEP-110: HTTP/2 Client, Java 9 will “Define a new HTTP client API that implements HTTP/2 and WebSocket, and can replace the legacy HttpURLConnection API.”

The Problem

Java’s existing HttpURLConnection has some problems. It is hard to use, in part from being abstracted to work with multiple protocols (like FTP and Gopher). It turns out 20 years later that HTTP was the big winner in protocol land. But the biggest problem is that HttpURLConnection works only in blocking mode so there is only one thread per request/response.

There are other HTTP clients out there that solve some of these problems, such as Jetty and the Apache HttpClient. Jetty even has a non-blocking mode. However, both of these are sizable libraries to bring in, and neither take advantage of lambdas.

The Solution

Introducing: Java 9’s HttpClient!

The new API works with HTTP/2, deals with multiple responses (server push), can stream the request and/or response bodies, and can work with synchronous or asynchronous requests or responses. So there is a lot of flexibility to taylor the behavior of your client to the problem you’re trying to solve. If that was not enough, the API makes it very easy to work with your request and response bodies as strings, bytes, input streams, even directly as files.

The HttpRequest Javadocs are thorough and provide a number of examples demonstrating various use cases. After reading the Javadocs, you’ll find that you’ll really understand the API more fully when you try to use it yourself. The rest of this post will explore a specific use case that I wanted to accomplish as I learned the API myself.

An Example: Stream It

I had in mind a specific problem: What if I wanted to download a very large document in a response body that I didn’t want to save to disk, but instead wanted to parse as it was downloading? One could make the argument that downloading to a file and processing after the download is a better separation of concerns, and that it allows you to retry the download or retry the processing if something goes wrong in either step. But for sake of argument let’s say the response body is unreasonably huge, the client is on a device with limited space, etc.

To get started, we can obtain an HttpClient with the builder from HttpClient.create() which provides a fluent API for creating a client. If you are ok with the defaults you can get started a little more quickly with the HttpRequest.Builder obtained from HttpRequest.create(), which is what happens in the code below.

The important things to learn from this example are that:

  1. The possible arguments to HttpRequest.Builder.body() allow for reading the request body from a file or or other source. Using RequestBuilder.body(fromInputStream(…)) allows you to stream the request body.
  2. The corresponding .body() and .bodyAsync() of the HttpResponse allow us to optionally stream the response body, again into a String, file, etc
  3. The HttpRequest’s .response() and .responseAsync() allow us to optionally block on the response itself

The code shows this in action with a blocking non-streaming request and a non-blocking streaming response (although we can choose to block with .join()).

import java.io.*;
import java.net.http.*;
import java.net.URI;
import java.util.*;
import java.util.concurrent.*;

import static java.net.http.HttpRequest.*;
import static java.net.http.HttpResponse.*;

public class Main {

   public static void main(String[] args) throws Exception {

      requestStreaming("http://stackoverflow.com");

      System.out.println("Program done.");
      System.exit(0);
   }

   public static void requestStreaming(String url) throws Exception {

      HttpRequest request = HttpRequest.create(new URI(url))
                                       .body(noBody())
                                       .GET();

      request.response()
             .bodyAsync(HttpResponse.asInputStream())
             .thenAccept( s -> readBody(s))
             .join();
   }

   public static void readBody(InputStream stream) {

      try(BufferedReader br = new BufferedReader(new InputStreamReader(stream, "UTF-8"))) {
         String line = br.readLine();
         while(line != null) {
            processLine(line);
            line = br.readLine();
         }
      }
      catch(Exception e) {
         e.printStackTrace();
      }
      System.out.println("Stream processing Done!");
   }

   public static void processLine(String line) throws Exception {
      System.out.print(".");
   }
}

Conclusion

Java 9’s new HttpClient provides a lot of flexibility and incorporates the latest language features. I expect we will see lots of creative uses!

Leave a comment

Filed under Java 9

Joining the JCP and Working with JSR’s

Recently I was inspired to join the Java Community Process (JCP) and try to support a Java Specification Request (JSR). In some ways it was easy, but in other ways there were parts of the process that were confusing. This post is intended to help anybody who wants to join the JCP and adopt a JSR.

Prologue: Working With Your JUG

If all you want to do is help out a JSR that you’re excited about, and your home JUG has a JCP membership, then stop here. The fact is, you don’t personally need to join the JCP or sign any paperwork to participate with a JSR. If you work in the context of your home JUG, your efforts and those of your fellow JUG members can be submitted to a JSR working group as a contribution of the JUG, so it is covered under the JUG’s JCP membership.

However, if you wish to join the JCP program in your own right as an Associate Member and be listed individually as a Contributor on a JSR, you can join by following the instructions for individual membership on JCP.org

The above link has detailed steps for joining. It is a lot of text and it looks intimidating, but it’s really easier than it looks!

The rest of this blog post is about joining the JCP as an individual. Essentially you create an Oracle Account if you don’t already have one, fill out a form and submit it, they send you an agreement to sign, you sign it and you’re done!

Step 1: Your Oracle Account

Once you create an Oracle Account and verify your email address, you will need to create your JCP account. On your first login to jcp.org you will see

“Your SSO login is not yet associated with any JCP account. So click the “Create New JCP Account” button below to create your account. If you are trying to link your SSO login to an existing JCP account, please contact the PMO: admin@jcp.org. In the message, describe both the e-mail address for your SSO login and the JCP account. If you are trying to create a new jcp.org account, please click this “Create New JCP Account” button.”

Click the button to create your account and you’re done!

One caveat is that your SSO login can only be associated with a single JCP membership. This is described on the the membership description page but I didn’t see it at first because it was in small print at the bottom of the page. This tripped me up because I tried to use a single login both to create a JCP account for a JUG and then for myself as an individual. Each Membership requires the use of a distinct user account.

Step 2: Legalese

They will send you the Associate Membership Agreement (AMA) form to sign, which really is just two pages of legalese and a couple clicks. It takes just a few minutes, and when you submit it, it then goes back to them for approval.

Approval can take a week or more, this is not an automated process. So don’t worry if you don’t hear anything for a little while. But if multiple weeks go by, try pinging the admin@jcp.org.

Step 3: Adopt A JSR

Ok, here is where we need some more explanation. The JCP / JSR relationship is a little confusing at first.

Let’s say you sign the AMA and log in to jcp.org. If you go to the Java SE 9 JSR to join it, there is a “join” link in a tiny font halfway down the page which is easy to miss. And if you attempt to join and sign up as a Contributor, they will respond that you can’t be a Contributor because you haven’t contributed anything. Also if you go to adoptajsr.org, that’s a different login (for java.net) and it doesn’t mention the JCP at all. So how do you “join” or “adopt” a JSR and start contributing?

The JCP is an organization and joining it gives you  certain abilities like voting rights, and the legal agreement you sign when you join covers the legal ownership of any contributions to a JSR. But except for login and joining/voting, you don’t do much on jcp.org. The JCP doesn’t host any repositories or code, and it doesn’t host the actual JSR home pages. The JSR on the JCP site (Such as Java EE8, JSR 366) is not the JSR’s home page (such as the Java EE JSR Home). So to join a JSR, you will have to go to the individual JSR’s home page and get involved from there. Each JCP page has a link called “Public Project Page” to take you to that JSR’s actual home page, but if you didn’t know that that’s what you’re looking for, it might be hard to find.

Each JSR hosts code itself and has its own set of rules. Joining a JSR means different things depending on the JSR because the individual JSR’s are managed by different people and different organizations, so each one could be run differently. For instance, OpenJDK is on java.net, some independent JSR’s like JSR 107 (Java Caching) are on github, and Redhat and IBM each use their own repos. You’ll need to create separate logins to do anything if you want to work on multiple JSR’s.

In the case of OpenJDK (for the Java SE JSR) it operates more like an open source project. There is no “I hit the Join button and now I’m a member, or now I’ve adopted it.” It’s more like “I’ve been monitoring the email list, picked out a favorite bug, submitted patches, and gradually become part of the community of people working on this thing.” The JDK contribution page outlines how to get involved with OpenJDK. Getting started would involve as little as getting familiar with the conversations going on in the mailing lists, such as the Java 9 observer mailing list.

Conclusion

For more information, go to to the Adopt A JSR Home Page
Joining the JCP and working with a JSR is not difficult, but getting started can be unclear for the uninitiated. Hopefully this post clears the way for you yourself to become active in the Java Community!

Leave a comment

Filed under Software Engineering

Java 9: Why Modules?

What Are Modules?

The Java Module System, a.k.a. Project Jigsaw, is an effort to introduce a higher-level structural concept into Java. In a nutshell: Classes assemble into packages, and packages assemble into modules. Modules declare on which packages they depend, and in turn export packages for others to depend on.

Modules are a fundamental new concept going deep into the JDK, and have been under development for years. Jigsaw was originally planned for Java 7. As work began it became clear that this was a huge undertaking reaching deep into the internals of the JVM and the Java language itself. Yet, despite being slipped to Java 8 and then to Java 9, the need for it has only increased. Why do we need modules in Java? Why is Project Jigsaw such a big deal?

Why Modules?

Mark Reinhold, the Chief Architect of the Java Platform Group at Oracle, gave a keynote speech at Devoxx Belgium where he described the purpose of a module system and why it is so important.

There are many use cases that modules solve, but they all boil down to two primary pain points:

  1. Brittle, error-prone classpath (AKA JAR Hell)
  2. Monolithic JDK

Let’s look more at each of these.

Brittle Classpath

Have you ever seen a ClassNotFoundException? How about NoClassDefFoundError, NoSuchFieldError, or NoSuchMethodError? Maybe even a ClassCastException: “Cannot cast com.foo.Bar to com.foo.Bar”? All of these are the results of a brittle classpath. What that means is that ClassLoaders don’t have a good mechanism for distinguishing one loaded class from another class of the same name, or for isolating classes loaded by one ClassLoader from classes loaded by another.

From the Wikipedia entry on JAR Hell, problems from a brittle classpath arise from one of these situations:

  • Accidental presence of two different versions of a library installed on a system. This will not be considered an error by the system. Rather, the system will load classes from one or the other library. Adding the new library to the list of available libraries instead of replacing it may result in the application still behaving as though the old library is in use, which it may well be.
  • Multiple libraries or applications require different versions of library foo. If versions of library foo use the same class names, there is no way to load the versions of library foo with the same classloader.
  • The most complex JAR hell problems arise in circumstances that take advantage of the full complexity of the classloading system. A Java program is not required to use only a single “flat” classloader, but instead may be composed of several (potentially very many) nested, cooperating classloaders. Classes loaded by different classloaders may interact in complex ways not fully comprehended by a developer, leading to errors or bugs that are difficult to analyze, explain, and resolve

In some cases, build tools may help you detect some of these situations (such as incompatible dependency graphs). But the fact remains that tools will not prevent you from doing dangerous things, and they do not protect you from all of the situations described above.

Java Modules will provide reliable configuration and strong encapsulation. If you have incompatibilities, you will discover these at build time instead of some indeterminate time after your application has started running in production.

Monolithic JDK

The JDK has been big for years. At this point in 2016, the JDK is too big to fit comfortably on small devices, and we would really like it to be easy to use embedded devices and other devices on the Internet of Things.

Now you might say “But I run my software on a large server, we are not constrained by space.” However, even on large servers we want to run lots of VMs and optimize memory use. If every instance you have requires an extra 200MB of memory that it doesn’t need, that adds up across hundreds of instances, and those resources are something you pay for every second of every day. Additionally, with a monolithic JDK startup performance is slower because of loading unnecessary things. We would like to measure our startup time in milliseconds, not thousands of milliseconds.

Java Modules will allow us to break up the JDK into modules so that we only use what we need. We will be able to deploy modular runtime images that take less space and that start up faster.

Conclusion

The changes for Java 9 to support modules are far reaching, and it will take some work because we will have new language features and syntax to learn. But I think this will be worth it to make Java applications stronger, faster, and more reliable. In a subsequent post we will look more at the details of how modules work and how we can start to incorporate them into our own software.

What do you think? Have you had the problems that Jigsaw is trying to solve? Or are these non-issues for you?

Leave a comment

Filed under Java 9

Java 9: Multi-Release JAR Files, Part II

In Part I on multi release JAR Files, we learned what it is and why it is useful. In this part we will learn how to actually construct such a JAR and test it out.

For this demonstration we will be using just the command line as opposed to Maven or Gradle. Tooling support for Java 9 is still ramping up🙂

in a src folder, we have two classes

public class Application {
   public static void main(String[] args) throws IOException {
      Generator gen = new Generator();
      System.out.println("Generated strings: " + gen.createStrings());
   }
}

and

public class Generator {
   public Set<String> createStrings() {
      Set strings = new HashSet();
      strings.add("Java");
      strings.add("7");
      return strings;
   }
}

Then in another src folder we have another implementation of the class using Java 9 features:

public class Generator {
   public Set<String> createStrings() {
      return Set.of("Java", "9");
   }
}

Finally, we need to specify a file containing a new manifest attribute to indicate that the JAR is a multi-release JAR. The manifest needs to include:

Multi-Release: true

And now, to compile! From the command line:

javac -d build -release 7 src/*.java
javac -d build/META-INF/versions/9 -release 9 src9/*.java
jar --create --file mrjar.jar --manifest MANIFEST.MF --main-class=Application -C build .

There are a couple things going on here: We need to compile in multiple phases like this because we literally need to generate different versions of bytecode and put it in different places. Also note that we are compiling with the new -release argument instead of -source and -target. Refer to JEP-247 for more details, but essentially it guarantees the class to be compatible with a given version of the JVM in a way that -source and -target can’t.

Finally, if we set our path to use the appropriate JVM, when running on Java 9 we see

java -jar mrjar.jar 
Generated strings: [Java, 9]

But running the same JAR on Java 7 we see

java -jar mrjar.jar 
Generated strings: [Java, 7]

Proving that a different class was used for each version of Java that was running it!

Admittedly this is still a bit crude, with the non-standard file locations and dropping to the command line to do everything. We will build up to a Gradle example soon!

Leave a comment

Filed under Java 9

Java 9: Multi-Release JAR files, Part I

Is your software directly installed by your end users? Do you write libraries that could be used on any version of the JVM?

In either of these cases, you don’t have control over which JVM your code is running on, so you need to target your code to the lowest version of the JVM that you can support. In some cases, this could even be Java 6. Even in early 2016, 10% of commercial JVM installations ran on Java 6. Not a lot, but… you know… Enough for you to have to consider it.

This is a problem

I have lived with this situation myself. Besides just keeping us from using the latest language features, supporting older versions allows the possibility that bugs that our users complain about in our software might actually be attributable to underlying JVM bugs. For example, there are known issues with Java.io.File that are fixed with NIO2, older JVM’s generally have lower performance, and finally older JVM’s lack certain security features and other bug fixes. Running on older JVM’s can even expose issues in our code that are not JVM specific but that would be easier to fix on later versions, such as hard-to-find multithreading bugs for parallelism that would be moot if we could just use parallel streams.

Wouldn’t it be nice if we could start using a later version of Java while still supporting older JVM’s for your users? We can!

Multi-Release JAR Files

Allow me to introduce you to JEP-238: Multi-Release JAR Files.

The idea behind Multi-Release JAR Files is that within a single JAR we can include multiple versions of class file for a single class. A version of the class can be selected and loaded that is specific to the version of the JVM running it. Each class file could have a different version of byte code, so later versions of the class will be ignored by older JVM’s. This allows you to begin migration efforts earlier and fix version specific bugs as outlined above, all while still only delivering a single JAR file as usual.

How does this work?

Java 9 defines an extension of the Jar file format. See the diagram below, a picture is worth a thousand words:

jar root
   - A.class
   - B.class
   - C.class
   - D.class
   - META-INF
      - versions
         - 8
            - A.class
            - B.class
         - 9
            - A.class

As we can see, the normal class files are present right where we’d expect them. But now there are additional class files that represent the same class when run on a specific version of the JVM. So if you run this JAR on Java 6 or 7, you will get the “normal” version of the class. If you run it on Java 8, you will get those versions of class A and B only, and if you run on Java 9, you will get the Java 8 version of B and the Java 9 version of A.

Keep in mind: at the time of this writing, Java 8 is not MRJAR-aware. So the Java 8 portion of the example described in the JEP and shown in this diagram is not actually possible because the Java 8 classes can never be recognized and loaded on Java 8. But they will be loaded on Java 9 and there’s nothing to prevent future versions of the Java 8 JVM from becoming MRJAR aware.

In Part II we will look at an actual implementation that you can do with the early access release of Java 9!

Leave a comment

Filed under Java 9