Pavel Fatin

In this article I enumerate reasons why typical approach to painting in AWT / Swing can result in substantial visual lags, provide examples that demonstrate the problem and propose methods to significantly reduce the drawing latency.

Despite the focus on Java platform, the key ideas can be extended to most modern operation systems and GUI frameworks.

Contents:

1. Problem statement
2. Typical implementation
3. Asynchronous painting
   3.1. Queue delay
   3.2. Request skip
   3.3. Region extension
   3.4. Component reordering
4. Synchronous painting
   4.1. Component opacity
   4.2. Buffering overhead
   4.3. Showing delay
   4.4. Buffer reuse
5. Active rendering
   5.1. Incremental painting
   5.2. Pipeline flush
6. Summary

All key points are accompanied by short, shelf contained, compilable examples that clearly demonstrate the theoretical concepts in practice. The code is also accessible as a GitHub repository.

1. Problem statement

Painting latency is delay between a drawing request and a corresponding screen (framebuffer) update.

In practice, both AWT and Swing subsystems tend to exhibit non-optimal painting latencies. That happens not because those architectures are not optimized, on the contrary — they are highly optimized, but with a different goal in mind. The primary purpose of AWT / Swing is to efficiently render complex hierarchies of GUI components, which are either static or insensitive to moderate painting lags.

If application is not a GUI form and, by nature, requires instantaneous response (for example, a video game), it makes sense to consider using Java’s Fullscreen API, or OpenGL bindings (like JOGL). But what if we need both — the GUI framework, and low-latency painting for some interactive process that are highly sensitive to delays (like typing in IDE)? It turns out that, with the help of a few special tricks, we can eat our cake and have it.

Before reading further, I recommend you to look through a great official introduction to Painting in AWT and Swing so you can refresh your knowledge of the basics and understand what I’m going to explain next more easily.

2. Typical implementation

Here’s how painting is usually implemented in Swing (and in AWT, with some minor differences):

class MyComponent extends JComponent {
  @Override
  protected void paintComponent(Graphics g) {
    super.paintComponent(g);
    // draw on the graphics
  }

  private void onSomeAction() {
    repaint();
  }
}

The key point here is inversion of control — the drawing code is not invoked directly, we can only request painting, but it’s up to AWT / Swing subsystem to decide how and when to proceed.

Why use such an indirect way? The primary reason is how stacking window managers work — most operating systems don’t retain application window content, and if some window is overlapped, a part of the window becomes “dirty” and has to be restored afterwards by the application itself. A similar technique is used by Swing internally to render a hierarchy of lightweight components. So, there must be a way for OS / framework to invoke our rendering code “from outside”.

Another reason for this indirection is to provide a way for the frameworks to “inject” drawing optimizations, like clipping, buffering, merging, etc. That is why even the internal painting requests are made indirect.

While the above code is quite typical, it’s sub-optimal even in an obvious way, so let’s rewrite it like this:

class MyComponent extends JComponent {
  @Override
  public void paint(Graphics g) {
    Rectangle r = g.getClipBounds();
    // draw on the graphics in the clip rectangle
  }

  private void onSomeAction() {
    repaint(x, y, width, height);
  }
}

What’s improved:

• The painting area is minimized by drawing only what’s needed. As painting bounds are not provided via an argument and we need to obtain them separately, this important detail is often overlooked.
• The paint method is overridden directly (instead of the paintComponent) to skip code that draws component border and child components.
• Drawing is not delegated to the superclass as our component draws everything on its own.

Now it looks better, almost… perfect, isn’t it? What could possibly go wrong here? The short answer is “everything”! Let’s move to the less obvious stuff.

3. Asynchronous painting

Inversion of control is only a part of the whole story. Another important part is asynchronicity — paint operations are requested asynchronously and delivered via the event dispatching thread (EDT) queue.

While such an approach can be useful for some class of optimizations, it might play false when it comes to latency.

3.1 Queue delay

A first pitfall of the asynchronous painting is a strong possibility of substantial lags in request delivery. Here’s an example:

private void onSomeAction() {
  SwingUtilities.invokeLater(() -> pause(500));
  repaint(x, y, width, height);
  pause(500);
}

The actual handling of the repaint request in the code above will be performed after at least 1 second after the request. The first part of the delay is due to the pause at the end of the action handling method itself (which runs in Swing thread). The second part of the delay is caused by the invokeLater request, which places a long-running event before the repaint request in the AWT queue.

Demo: QueueDelay.java

Please note, that although we inserted clear pauses in this example, in real applications we don’t need to do such a thing explicitly — any code that runs in the event dispatching thread will delay our painting requests. Because in AWT / Swing most tasks (like input processing, action handling, component updates, etc.) must run in EDT (Swing is essentially single-threaded), there is no shortage of delays. In complex applications (like an IDE), the event queue might become highly polluted, which consequently give rise to serious lags in asynchronous painting.

How can we mitigate the problem? We should always strive to minimize time our code spends in the even dispatching thread. However such a measure is only partial, because some inherent framework processing that runs in EDT is outside of our control. More reliable and more straightforward solution is to employ synchronous painting (see below).

3.2 Request skip

What happens when a new painting request is issued before a previous one is handled? For example, let’s generate 10 painting requests with 50 ms interval, but spend 100 ms on each painting:

class MyComponent extends JComponent {
  @Override
  public void paint(Graphics g) {
    pause(100);
  }

  private void inAnotherThread() {
    for (int i = 0; i < 10; i++) {
      component.repaint(x, y, width, height);
      pause(50);
    }
  }
}

The scope of processing delay is not necessarily limited by the drawing method, any activity in EDT counts.

Demo: RequestSkip.java

If you run the demo application, you will find out that our 10 painting requests result only in 6 painting callbacks. This happens because there’s a special optimization for such cases that combines multiple unprocessed requests into a single one.

In Swing, the corresponding function is performed by RepaintManager (which can be customized). In AWT “the algorithm for determining when multiple requests should be collapsed is implementation-dependent” (and thus unpredictable).

On the one hand, the underlaying assumption sounds reasonable — indeed, why bother with intermediate requests if we need to repaint the component again real soon, anyway. On the other hand, if what we render is not a typical GUI form, but some dynamic process (like user typing), the loss of visual fluidity might be a big deal. For example, when user types 5 characters one by one, he / she expects them to appear one after the other (even with some delays), not “nothing, nothing… and then all 5 at once”.

Is it possible to avoid the skipping of painting requests? In Swing, we can create a custom implementation of RepaintManager for our component. In AWT there’s no way to redefine the behavior. Just like with the queue delays, synchronous painting is an effective solution (see below).

3.3 Region extension

The next peculiarity follows from the previously described questionable optimization: one does not simply ignore an inconvenient painting request, the next drawing bounds must be extended to encompass all the “dirty” regions.

If those areas are consecutive (or, at least, approximate) then all is well, but what if, for instance, in our text editor we want to draw a newly typed character in the top-left corner of the screen and simultaneously update caret position in the bottom right corner?

private void onNewCharacter() {
  repaint(0, 0, 10, 10); // new character
  repaint(width - 10, height - 10, 10, 10); // caret position
}

Good luck re-drawing a major portion of the screen on each inserted symbol! 4K screens are becoming popular, you know, so only ~10M pixels instead of just a few hundred, not a big deal (sarcasm, apparently).

Demo: RegionExtension.java

To control the extension of “dirty” regions, we can assign a dedicated repaint manager to our component (but only in Swing, not in AWT). Yet, synchronous painting is probably a better solution, because it give us even more control over the process, without a need to tinker with the custom RepaintManager implementation.

3.4 Component reordering

The next pitfall is a totally unexpected one — the order of component updates is not guaranteed.

Suppose we would like to update a primary component that is sensitive to drawing latency (like a text editor) and, additionally, a secondary component that is latency-insensitive (like project view). We naturally prefer the primary component to be painted first:

private void onNewCharacter() {
  editor.repaint(x, y, width, height);
  project.repaint(x, y, width, height);
}

Can we be sure that the editor is updated before the project view? It turns out, that we can’t — component order is totally unpredictable. For example, in Windows arbitrary order is preselected during initialization, while in Linux the order seems to be rigorously reversed. Obviously, when drawing of irrelevant components meddles in, it takes time and increases visual latency.

Demo: ComponentOrder.java

A possible workaround is to impose order by performing each subsequent repaint request withing a nested invokeLater call:

private void onNewCharacter() {
  editor.repaint(x, y, width, height);
  SwingUtilities.invokeLater(() ->
    project.repaint(x, y, width, height));
}

However, this approach works only for isolated requests, not when requests are merged by the RepaintManager / AWT (as we’ve seen previously), and it increases the possibility or request delaying. Once again, synchronous painting is the real answer.

4. Synchronous painting

So, when it comes to latency, asynchronicity is a problem. Hopefully, there’s an easy solution: it’s possible to circumvent the RepaintManager and invoke paintImmediately directly to perform so-called “synchronous painting”:

class MyComponent extends JComponent {
  @Override
  public void paint(Graphics g) {
    Rectangle r = g.getClipBounds();
    // draw on the graphics in the clip rectangle
  }

  private void onSomeAction() {
    paintImmediately(x, y, width, height);
  }
}

In this way the actual drawing will be performed synchronously — during the call to paintImmediately, consequently:

• painting cannot be delayed by other activity in EDT,
• painting cannot be skipped,
• painting region cannot be unpredictably extended,
• painting cannot be interleaved with irrelevant components.

The official documentation states that “programs should not invoke this method directly unless there is a valid need for real-time painting”. I bet now we know why such a need may arise.

There is an important rule for using paintImmediately: unlike repaint, this method must always be invoked from the event dispatching thread (because it still goes through a lot of Swing machinery).

With one leap, we’ve solved all the previously described problems. Now it has to be perfect, after all, there’s an “immediately” part, right?! Not so fast… Asynchronicity is gone, but inversion of control still remains.

4.1 Component opacity

Because our drawing code is still invoked indirectly, Swing can still do a lot of things behind the curtain, and some of them might be rather surprising.

To demonstrate the point, let’s simplify our drawing method to the limit:

@Override
public void paint(Graphics g) {
  // draw nothing
}

Are you sure that nothing will be drawn when paintImmediately is called? If you have doubts, then rightly so — feel free to run the demo.

Demo: ComponentOpacity.java

In reality, because by default Swing components are considered transparent, painting of the parent component will be performed before painting of the component itself (this is also true for the asynchronous painting, by the way). The additional drawing is usually rather light (like filling a rectangle), but none the less, unnecessary.

As paint method often fills its clip bounds by itself, the duplicate painting is hard to notice, so don’t forget to invoke setOpaque(true) on your custom components (unless you indeed want them to be transparent). Another solution is to completely forgo the Swing callback and resort to active rendering (see below).

4.2 Buffering overhead

OK, let’s dig deeper to find out how deep the rabbit hole goes what else Swing injects during the callbacks.

Let’s assume that we still draw nothing in the paint method and our component is now marked as “opaque”. How much time does it take to paint nothing?

private void onSomeAction() {
  long before = System.nanoTime();
  paintImmediately(x, y, width, height);
  long elapsed = System.nanoTime() - before;
}

Demo: BufferingOverhead.java

The numbers may vary significantly depending on hardware, OS, video driver, JVM, etc. On my (moderately powerful) machine painting nothing in 10 x 10 area takes about 0.5 ms, while painting nothing in 1000 x 1000 area takes ~5 ms (I leave it to you to check 4K screens). Not too much, but… for nothing? There must be something rather than nothing.

That something is an enforced double buffering and, as you might guess, it’s not for free (this is also true for the asynchronous painting). Behind the curtain, Swing substitutes window graphics for back buffer graphics and after our method returns, the content of the back buffer is copied to the framebuffer (via bit blit, page flip, or by other means), which takes additional time.

As documentation says “If your performance metric is simply the speed at which double-buffering or page-flipping occurs versus direct rendering, you may be disappointed. You may find that your numbers for direct rendering far exceed those for double-buffering and that those numbers far exceed those for page-flipping”. So, apart from possible v-sync (which we cannot control, see below) double buffering is only good for making complex drawing to appear instantaneously.

If your visual content is so complex that drawing process is clearly visible, then double buffering might help you to improve perceived performance. However, if you drawing is already close to instantaneous, double buffering only adds additional, unneeded delay and increases visual latency.

Can we suppress the forced double buffering? Yes, we can:

private void onSomeAction() {
  RepaintManager rm = RepaintManager.currentManager(this);
  boolean b = rm.isDoubleBufferingEnabled();
  rm.setDoubleBufferingEnabled(false);

  paintImmediately(x, y, width, height);

  rm.setDoubleBufferingEnabled(b);
}

If you modify the demo code in such a way, you will observe that painting time is reduced (~0.1 ms on my machine) and doesn’t depend on the region size anymore. Yet minimal constant delay still remains, because there is simply quite a lot of intermediate code involved (namely JComponent._paintImmediately, RepaintManager, BufferStrategyPaintManager).

Note that we need to revert the double buffering property to some previous, platform dependent state, because repaint manager is shared between different components (JComponent declares setDoubleBuffered method, but it only matters for JRootPane). You may also consider using a try-finally block for more safety.

If you choose to avoid double buffering, than there are even less reasons to meddle with the Swing callback — active rendering is a much cleaner solution (which also makes possible to use sub-region double buffering and only when needed).

4.3 Showing delay

Considering how double buffering works, the next point might be self-evident, nevertheless there are some details that are worth mentioning.

Let’s “reuse” conditions from the region-extension case: we want to draw a newly typed character in the top-left corner of the screen and simultaneously update caret position in the bottom right corner. With synchronous painting we can reliably separate drawing of those two areas, but let’s assume that, because such a separation requires some additional checks in the drawing method, we decided to update the screen in a single painting.

We might attempt to draw key elements — symbol and caret position first, and then draw remaining text in a second step (because it takes a long time, and that text is unchanged anyway):

@Override
public void paint(Graphics g) {
  // draw the new character
  // draw the caret position

  // draw the remaining text (time consuming)
}

Demo: ShowingDelay.java

Will the new character appear any faster? Of course not! Because of the double buffering, all the drawing will be performed in the back buffer and only then will be shown on the screen, simultaneously.

Additional, less obvious delay is a possible waiting for vertical synchronization. While v-sync might be useful to avoid tearing, in Java there’s no way to control vertical synchronization, so there might be no v-sync when it’s required, or enforced v-sync (and thus an additional delay) when it’s not needed at all. Because v-sync always produces additional visual lag, a few alternative technologies (like Adaptive-Sync, FreeSync and G-Sync) emerged recently, that aim to reduce screen tearing and simultaneously reduce visual latency.

A possible (but clumsy) solution is to insert additional checks in the drawing method and then paint in multiple passes:

@Override
public void paint(Graphics g) {
  Rectangle r = g.getClipBounds();

  if (r.intersects(characterRegion)) {
    // draw the character
  }

  if (r.intersects(caretPositionRegion)) {
    // draw the caret position
  }

  if (r.intersects(remainingTextRegion)) {
    // draw the remaining text (time consuming)
  }
}

private void onNewCharacter() {
  paintImmediately(characterRegion);
  paintImmediately(caretPositionRegion);
  paintImmediately(remainingTextRegion);
}

Needless to say, such an approach is rather inconvenient. Moreover, we perform multiple buffer flips, while v-sync might be enabled. The cause of this complication is reliance on the Swing callback, so we cannot control painting directly. Active painting is the solution (see below).

4.4 Buffer reuse

So, if it’s hard to paint in multiple passes via the Swing callback, can we simply ignore the time-consuming part which is not changed, and re-draw only changed regions? It’s worth trying:

@Override
public void paint(Graphics g) {
  // draw the new character
  // draw the caret position
}

private void onNewCharacter() {
  paintImmediately(0, 0, getWidth(), getHeight());
}

Demo: BufferReuse.java

As you can see, we have requested painting of the whole component area (to encompass the two subregions), but updated only the changed parts. Will it work? Surprisingly, yes, it might work. But that can be very misleading, because the whole list of possible outcomes is:

• new context will be merged with the existing context, as expected;
• new content will appear on a solid background;
• new content will be displayed on top of image of a previously painted component;
• new content will be displayed on top of some visual noise.

The exact result depends on OS, window manager, video driver, JVM, etc. For example, in Windows 7 with Classic desktop theme everything works “as expected”, but when Aero is enabled, a solid background is displayed in place of the untouched image.

It’s just not possible to reuse Swing back buffer, the whole area must always be redrawn completely … OK, technically, in some cases, it is possible to reuse back buffer content. To do that, we need:

• OpenJDK or Oracle JRE,
• system property swing.bufferPerWindow=true (default value),
• RepaintManager with BufferStrategyPaintManager inside (by default),
• no prior invocations of Component.getGraphics() inside JRootPane (we may call safelyGetGraphics() via reflection instead, if needed).

If all these conditions are met, we can expect that back buffer mirrors frame buffer. However, keep in mind, that this functionality is not a part of API and it’s JRE-specific. If we want a more reliable way to draw image incrementally, active rendering is required.

5. Active rendering

It’s time to become more independent. Previously we had to wait for AWT / Swing to invoke our drawing code (which is sometimes referred to as “passive rendering”). Now we’re going to use so called “active rendering” and draw directly to the screen, as we like.

We can completely abandon the Swing machinery:

private void onSomeAction() {
  Graphics g = getGraphics();
  if (g != null) {
    // draw on the graphics
    g.dispose();
  }
}

Because now there’s only a single method with no callbacks, we are at the helm during drawing, it’s up to us where to draw and how to draw.

There are several rules for using this approach:

• You must always check that component graphics exists, because components that are not yet shown return null instead of the Graphics object. If the code runs under OpenJDK or Oracle JRE, you may prefer to invoke safelyGetGraphics() via reflection to preserve so-called “true double buffering” (more info).

• The drawing must be performed in the event dispatching thread. While in principle we can do this kind of drawing from any thread (as there’s no Swing code involved) we need to coexists with Swing in terms of visual image and draw only on top of existing picture (unless you component occupies the whole window, but then you probably don’t need to use Swing at all).

• You should call dispose() on the graphics instance after use to release platform resources. This method is called on object finalization, but there’s no much sense in retaining resources longer than needed.

• The paint callback must be ready to repeat the drawing on framework’s demand. Although we don’t request repaints via the repaint method anymore, OS / framework may still call our code to restore window content when needed. If we choose to preserve “true double buffering”, we may need to repaint corresponding back buffer content to synchronize it with the frame buffer’s one.

What if we want to use double-buffering for some part of the rendering? When our content covers the top-level window completely, we can rely on BufferStrategy to provide double buffering (more info). However, such kind of buffering cannot be used for sub-regions and, thus, for separate components (unless we resort to platform-specific methods, like in BufferStrategyPaintManager with SubRegionShowable). We may employ a different technique for that purpose, which is based on VolatileImage (and which is also employed by generic PaintManager). Alternatively, you may consider mixing heavyweight and lightweight components, though the result might be less predictable.

Although now the code really looks ideal (there’s simply nothing more to subtract), we can add a thing or two to reduce drawing latency even further.

5.1 Incremental painting

With the fine-grained control over painting, it’s easy to re-draw only changed areas at arbitrary locations. So we can, for example, draw a newly typed character in the top-left corner of screen and simultaneously update caret position in the bottom-right corner, without having to re-draw unchanged text in between. That can save us quite a lot of time and substantially reduce latency of screen updates.

Another effective tool that is now at our disposal is copyArea method. Imagine that user typed a new character at the beginning of a long string in our text editor. Normally we would redraw the whole line, because the inserted character shifts all the remaining text. However, there’s a better way:

private void onNewCharacter() {
  Graphics g = getGraphics();
  g.copyArea(x, y, tailWidth, lineHeight, charWidth, 0);
  g.draw(c, x, y);
}

Demo: IncrementalPainting.java

We can reuse the previously drawn image of string by shifting it to the right, so we don’t have to actually draw the string again. The copying is most likely be performed via a bit blit inside video memory (extremely fast, without CPU involvement).

To improve visual latency, always seek to re-draw only changed regions. Consider prioritizing updates (as order of drawing now matters). Additionally, if your image has shifting parts consider using the copyArea method (instead of re-drawing) to maximize performance.

5.2 Pipeline flush

The following example is like a Zen Koan, and likewise, it may provide a valuable insight:

private void onSomeAction() {
  Graphics g = getGraphics();
  g.fillRect(x, y, width, height);
}

Will the rectangle appear on the screen immediately after fillRect is invoked? Feel free to try.

Demo: PipelineFlush.java

Because this demo queries actual screen content, it works better with stacking window manager, so switch to Classic theme in Windows / use something like Openbox in Linux to achieve greater measurement accuracy.

Internally, rendering in Java goes through a so-called pipeline (implemented on top of OpenGL, GDI, D3D, X11, XRender, etc) and that pipeline might delay actual drawing. Moreover, OS graphics subsystem and video driver might do some internal batching and buffering which can also delay drawing (for example, see how submitting a command buffer in Windows works). Depending on hardware / OS / pipeline / video driver combo, those delays might vary from non-existent to rather substantial.

There’s a simple (yet little-known) solution — it’s possible to “flush” rendering pipeline explicitly via Toolkit.sync:

private void onSomeAction() {
  g.fillRect(x, y, width, height);
  Toolkit.getDefaultToolkit().sync();
}

That’s how we can guarantee that rendering pipeline, OS graphics subsystem and video driver perform on-screen drawing immediately (and that can be a game changer).

6. Summary

Let’s summarize all the key points:

• Painting in AWT / Swing tend to exhibit non-optimal latencies.
• Synchronous painting & active rendering can drastically reduce the drawing latency.
• It’s possible to coexist with Swing by painting in EDT and respecting framework callbacks.
• We should re-draw only changed regions and prioritize order of updates.
• We can reuse existing image on the screen to draw incrementally.
• We can use copying to shift / clone regions.
• Buffering can be done via volatile images and should be used only when needed.
• We can explicitly flush graphics pipeline to trigger actual drawing.

Each of the optimizations improves latency to some degree so that cumulative improvement is synergistic. To see the approach in real application you may check a corresponding mechanism in ToyIDE.

See also:

• Typing with pleasure — Human- and machine aspects of typing latency, experimental data on latency of popular text / code editors.
• Typometer — Tool to measure and analyze visual latency of text / code editors.

Tags: awt, delay, drawing, java, lag, latency, painting, swing

This entry was posted on Sunday, December 20th, 2015 at 10:35 am and is filed under Programming. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.