Typing with pleasure

Keyboard with a smileIn this article I examine human- and machine aspects of typing latency (“typing lag”) and present experimental data on latency of popular text / code editors.

Latency recently became a hot topic in computer world — now we have low-latency keyboards, 144 Hz monitors, special technologies to reduce latency (like FreeSync or G-Sync), dedicated communities and whatnot… Sure, a part of the buzz is marketing, however the truth is that low latency became both feasible and desirable.

Apparently, gamers are the first who gain form those advancements. In some areas, like virtual reality latency turned out to be a crucial factor, where even a single millisecond matters. But what about programmers? Do we need “typing with pleasure” to “develop with pleasure”? Let’s find out.

The article is also (independently) translated into Russian.

See also: Scrolling with pleasure.

Contents:

  1. Human side
    1.1. Feedback
    1.2. Motor skill
    1.3. Internal model
    1.4. Multisensory integration
    1.5. Effects
  2. Machine side
    2.1. Input latency
    2.2. Processing latency
    2.3. Output latency
    2.4. Total latency
  3. Editor benchmarks
    3.1. Configuration
    3.2. Methodology
    3.3. Windows
    3.4. Linux
    3.5. VirtualBox
  4. Summary
  5. Links

1. Human side

Typing latency is a delay between the keystroke and corresponding screen update. Sounds simple, but make no mistake, its impact on typing process is rather complex, because typing is an amazing feat of our body and nervous system (at least, from engineering standpoint).

Let’s start with the basics — why bother with latency at all? We are not in such a hurry, so we can just type what we need and examine the result later. A few seconds doesn’t make a difference, right? Not actually…

1.1 Feedback

One does not “simply type” what he or she desires, we need feedback to perform that task, so that our senses form a so called closed control loop with our motions. Visual image is not only a result of typing, it’s an inherent part of the process.

The more feedback we receive during typing, the better. In principle, we can type with our eyes closed. We can continue typing even if we plug up our ears, relying only on tactile sensations from our fingers. However, if we block this last channel of feedback, typing becomes impossible. Because visual modality is dominant (vision tends to bias information from other senses), and it’s the only way to gain reliable data on typing errors, visual feedback is very important.

Less latency results in “shorter” feedback loop, so we can type more easily, with greater speed and accuracy. So far so good, but what is the reasonable bound? After all, human reaction time is ridiculously slow — “round trip” from senses to consciousness to muscles takes about 200 ms! Sure, it’s trainable and varies between individuals, but some comparatively small latency, like 100 ms looks like a safe bet, isn’t it? Not at all…

1.2 Motor skill

Remember your first attempts to use computer keyboard? Most probably, it took you minutes to press a proper key and then verify the result on screen, this process demanded all your conscious attention. However, as months and years went by, things improved up to the point where your typing became super-fast and semi-automatic — you don’t need to think about specific keys anymore, you probably don’t even need to look at the keyboard at all (touch typing). So, practice makes the difference. Such a process is called motor skill acquisition (or motor learning). When learning reaches autonomous phase the task can be completed “automatically” without us having to pay any attention to performing it.

Formally speaking, typing is a process of motor control exercised by human motor system. Because typing is a fine motor skill (it coordinates small muscle movements), it’s highly feedback-dependent. However, the feedback is handled below the threshold of our conscious perception, so we don’t necessarily need to be aware of latency to be severely affected by it. Human response time is only relevant for error correction.

But even semi-automatic processes are limited by vision delay (pdf), and any feedback is only good if we can use it. Human visual system needs about 40 ms to process its input. Again, this number depends on many factors, varies between individuals and can be improved by training. Yet, latency as small as 20 ms should be surely enough, right? Not entirely…

1.3 Internal model

First, let’s make it clear that any “external” delay is added on top of the vision delay, not “masked” by it, so any delay matters. But then we can argue that something like 20 ms is comparatively small anyway. Well… human motor system have some clever tricks to react faster.

To counteract input delays in sensory organs and nervous system, motor control relies on a special neural process, known as internal model. This process can simulate response of controlled system before receiving feedback data — by relying on a forward model of the process. A supplementary inverse model is used to predict response of controlled system basing on actual feedback.

The latency doesn’t influence feedback loop directly, but rather affects the internal model of typing process. Forward- and inverse models are used in combination, with some internal feedback loops, producing nonlinear system. Thus, even small delays in feedback data might result in much bigger disturbances in typing.

Apart from the internal model, there exists another process that is adversely affected by the visual latency, namely multisensory integration.

1.4 Multisensory integration

Visual image is not the only kind of feedback during keyboard typing: we can also hear key clicks, feel pressure on our fingers, know location of our hands and fingers (so called proprioception).

Each of these sensations is processed separately and nervous system is responsible for whether to integrate or segregate certain groups of signals. This task is known as binding problem, and it’s already a challenge by itself (check those multisensory illusions, for example). When brain receives stimuli from most sensory modalities immediately, but visual image is provided with some additional delay, it complicates things even further.

Nervous system can to some extent adapt (pdf) to constant visual feedback delays (though never perfectly), but any irregularities in delay durations (so called jitter) pose additional problem because of their inherent unpredictability.

There are a few illustrative examples from other areas where delayed feedback might be worse than no feedback at all, up to the point where primary activity becomes impossible: when you play on MIDI keyboard through computer, 20-30 ms of audio latency is enough to disrupt your performance (one paper (pdf) claims that even ~1 ms audio latency can matter); another good example is so-called SpeechJammer (pdf) which uses ~200 ms feedback delay to temporarily disrupt people’s ability to speak.

1.5 Effects

Let’s summarize the key points so far:

  • Typing is a complex, semi-automatic process that takes years to master.
  • In principle, typing can be affected by milliseconds delays in visual feedback.
  • One does not necessarily need to perceive latency consciously to be affected by it.
  • Perceptiveness and tolerance of latency vary greatly between individuals.

How exactly can latency affect typing? Here are a few possible effects:

  • Typing becomes slower.
  • More errors occur, more corrections needed.
  • Eye strain increases (as visual system is overloaded).
  • Muscle strain increases (as movement control is fuzzy).
  • Process demands more conscious attention.

The exact effect of latency depends on various factors, like particular delay distribution, your hardware (keyboard, monitor), editor content (text / code), primary activity (insertion / editing), typing method (ordinary / touch typing), specifics of your visual- and motor systems, personal preferences, etc.

To get a feel of how latency, in general, manifests itself, I recommend you to check a great video from Microsoft Research that demonstrates that (though in a slightly different context) and shows that even 1 ms matters. As for the personal side — please keep in mind, that human perception really differs — we should remember The dress and assume that other people might perceive / be affected by the typing latency in a completely different way than we do.

Needless to say, the possible impact of latency is hardly pleasant. The good news is that it works both ways, so by reducing the visual latency we can potentially improve a lot of things (it’s not only about typing speed per se).

The findings are generally agree with available research data. If you want to go deeper into the subject, you may check some books and scholarly articles that contain data on the effects of latency and the role of visual feedback in typing, for example:

One thing especially worth quoting (source):

Delay of visual feedback on a computer display have important effects on typist behavior and satisfaction.

Let’s now focus on the machine side of latency, to find out how we can make the improvement possible.

2. Machine side

What happens between the moment you press a key and a character appears on the screen? It turns out, many, many things, and all of them take some time. Can we simply buy a top-of-the-line computer to ensure smooth typing? This helps, but only to some degree, because many parts of the process are not CPU / GPU bound. Take the both pills, and I will show you how deep the rabbit hole goes (because even what follows is only a part of the whole truth).

The typing latency can be subdivided into the following components:

  • Input latency (keyboard).
  • Processing latency (software).
  • Output latency (monitor).

Let’s take a closer look at each part.

2.1 Input latency

First, let’s examine how typical, non-gaming, USB keyboards handle our typing. Those kind of keyboards are most widely used nowadays both for desktop computers and in laptops.

Keyboard scan

Typical keyboard contains more than 100 keys, which is quite a lot. To minimize the number of wires and control processor inputs, key switches are usually connected in a matrix circuit, so that rows and columns of wires cross. That’s how it’s possible to reduce the number of wires and inputs from x * y to x + y (for example, to 22 instead of 121). The drawback of such an approach is that control processor cannot read state of all the keys simultaneously — it has to scan switches repeatedly, with some constant frequency, which results in some latency. Typical matrix scan rate is 1000 Hz, so maximum scan-related latency is 1 ms (average delay is 0.5 ms).

Contact bounce

Regardless of keyboard type, key switches are mechanically imperfect and are subject to contact bounce — instead of a clean transition, the switch rapidly bounces between on and off states several times before settling. Bounce time depends on switch technology, for example, for Cherry MX switches bounce time is claimed to be less that 5 ms. Though exact probability distribution is unknown, basing on related empirical data, we can assume that average bounce time is about 1.5 ms.

Debouncing

Because sampling rate is fast enough to misinterpret contact bounce as keystrokes, keyboard control processor perform so-called debouncing of the signals by aggregating them across time to produce reliable output. Such a filtering introduces additional delay, which varies depending on microcontroller firmware. As manufacturers generally don’t disclose their firmware internals, let’s consider typical debouncing algorithms and assume that filtering adds ~7 ms delay, so that maximum total “debounce time” is about 12 ms, and average total debounce time is ~8.5 ms.

USB poll

As USB is a host-driven bus, keyboard has to wait for a host request before sending the data about registered keystroke (keyboard control processor places accumulated events in an output buffer). Though USB devices request desired polling rate on initialization (up to 1000 Hz), operating systems usually force 125 Hz polling frequency for low-speed / full-speed (USB1.x) devices, so maximum polling-related delay is 8 ms (average is 4 ms). Keep in mind, that it’s often possible to override the polling rate manually.

USB transmission

It also takes some time to actually perform data transmission. For low-speed / full-speed devices the shortest transaction time is 1 ms. For high speed devices (USB2) this time is much shorter – about 0.001 ms (by the way, there exists a great article on adequacy of USB for real-time systems (pdf)). To minimize data transfer times, it’s recommended to avoid connecting high-bandwidth USB devices (like storage devices, audio cards, webcams) to the same USB controller / hub where your keyboard is connected.

Let’s summarize the latency numbers for the typical keyboard:

Source Min, ms Max, ms Average, ms
Matrix scan 0 1 0.5
Debouncing 7 12 8.5
USB poll 0 8 4
USB transmission 1 1 1
Total 8 22 14

The findings are generally agree with some experimental attempts to measure keyboard lag.

Can we do better? Sure, a piece of cake! Professional keyboards employ:

  • switches with low bounce time (some are even optical),
  • high frequency matrix scan,
  • adaptive debouncing algorithms,
  • customizable firmwares,
  • USB2 for fast polling and low transaction delays.

All that can substantially reduce both maximum and average keyboard latencies. For example, when I bought Kinesis Advantage keyboard, I immediately noticed the difference with my previous “ordinary” keyboard.

Gaming keyboards advanced even further — if you want an uncompromising solution, you may consider something like Cherry’s RealKey technology that reads key switches via analog inputs (instead of digital ones) and offers close to zero latency.

2.2 Processing latency

Processing latency is a delay between receiving the keyboard input and generating an image of the typed character in a video frame. Put simply, it’s a delay inside computer itself.

Keep in mind that processing latency differs from performance as such — it’s theoretically possible to have a system that generates 300 FPS, yet exhibits 5 second “lag” behind the input. Particularly, batching and buffering is often an attempt to attain higher performance at the expense of latency.

While conceptually there are many distinct sources of processing latency, in practice, they are so tightly coupled with editor application that all the parts are usually measured together (see the next chapter for experimental data).

Operating system

After USB host (computer’s USB “port”) receives the data from keyboard, it generates a hardware interrupt so that software driver in operating system can read the received data via host controller interface (usually over PCI or PCIe). After that, the data is processed by Human Interface Device (HID) subsystem in the OS which eventually results in placing a “key pressed” event (like WM_KEYDOWN) in operating system’s message queue. Then the event is dispatched by event loop to active application window.

All these actions require some time, yet this time is negligible for our purpose (keyboard typing). What’s important is that major operating systems (namely, Windows, Mac OS, and Linux-based OSes) are not real-time ones, so there’s no guarantee of latency. Both hardware processes (network I/O, storage I/O, etc.) and software multitasking can increase processing times unpredictably (and unrestrictedly).

By the way, because GUI applications can impose tight restrictions on system latency, there were many successful attempts to optimize Linux kernel scheduling specifically for “desktop” use-case (particularly, via ulatencyd).

Another OS-related source of possible processing delays are programs that add system-wide keyboard hooks and process the events synchronously. For example, Workrave can noticeable increase the typing latency.

For simplicity, let’s assume that our computer is not under significant load during typing, so those kind of delays are below 1 ms.

Virtual machine

If text editor / IDE works on top of process virtual machine (like JVM for Java or CLR for .NET Framework), additional delays might occur because those runtime environments are also not real-time systems (with the exception of specialized implementations, like Real time Java).

Performance is usually not a problem per se, but when just-in time-compiler or garbage collector kicks in, some “lags” can be expected. JavaScript engines and Emacs Lisp interpreter also exhibit those kind of delays.

Mainstream VMs have greatly matured over the years, nevertheless some prior warm-up might still be required to guarantee stable latency.

Editor

This part is the most noticeable one, and rightly so. Editor application can give rise to the major part of typing latency. Unlike hardware-induced latencies, maximum delay time within editor is practically unlimited (we might easily stumble on 1-2 s lags).

Because editor is mostly CPU bound, we can usually improve editor latency by using a more powerful machine (good GPU also helps). But the reliance on raw power is fragile — it’s often not there when you need it the most (for instance, when you’re running a development server or processing some data in the background). Also keep in mind, that even top-of-the-line laptop can be much more sluggish in power saving mode (on battery), so this approach is effective only to a certain extent. Besides, it’s quite expensive, you know…

There’s a better way — in principle, we don’t need a supercomputer for smooth typing — all we need to do is to implement editor application with that goal in mind. Typing in editor is a relatively simple process, so even 286 PCs were able to provide a rather fluid typing experience. It’s perfectly possible to achieve close to zero typing latency even in a complex contemporary IDE, which is exactly what zero-latency typing mode in IntelliJ IDEA is aiming for. See my article on low-latency painting in AWT and Swing for technical details (despite the focus on Java platform, the key ideas can be extended to most modern operation systems and GUI frameworks).

Different editors / IDEs vary greatly in their typing latencies. Next chapter contains a great wealth of empirical information on that topic.

Rendering pipeline

CPU and GPU run in parallel and communicate through a command buffer, supplemented by an additional buffer in video driver. Most rendering commands are assumed to be asynchronous, so it’s perfectly legitimate for rendering to not even have started when drawing method returns. Drawing commands are placed in video driver’s buffer, then batched and pushed to GPU’s command buffer later (in an attempt to achieve higher frame rate at the expense of latency). Depending on hardware / OS / API / video driver combo, the resulting delay might vary from non-existent to rather substantial.

Typical desktop applications works on top of GUI frameworks, so they are isolated from underlaying rendering pipeline and thus, from rendering synchronization. That’s usually OK, because many desktop applications are latency-insensitive. However, some activities are highly sensitive to delays (like typing in IDE), so editors can explicitly “flush” rendering pipeline to force actual drawing and improve visual latency. For example, see how synchronization in OpenGL works. You may also check the pipeline flush demo.

Double buffering

To conceal intermediate drawing and present rendered image at once many applications resort to so called double buffering — the drawing is initially performed in an off-screen “back buffer” and when all drawing operations are complete, the updated image region is transferred to the framebuffer via bit blit or page flip. Many GUI frameworks (for example, Swing) force double buffering by default.

Needless to say, this additional step leads to additional delay. As Java’s documentation says “If your performance metric is simply the speed at which double-buffering or page-flipping occurs versus direct rendering, you may be disappointed. You may find that your numbers for direct rendering far exceed those for double-buffering and that those numbers far exceed those for page-flipping”.

Sometimes double buffering is absolute necessity, at other times it’s not actually required, so editors should use double buffering wisely to avoid redundant visual latency. Check the buffering overhead demo for more info.

Window manager

Today applications rarely run in full-screen mode, most desktop environments display each program in a separate window. This task is performed by so called window managers, which can be divided into several classes, depending on how windows are drawn and updated.

Stacking window managers orchestrate drawing of overlapped windows in such a way, that background windows are painted first. While this approach has some drawbacks (window content has to be restored explicitly), it introduces no additional delays because applications draw directly in the framebuffer. Examples of stacking window managers: Classic theme in Windows, Openbox in Linux.

Compositing window managers substitute the framebuffer with a dedicated off-screen buffer for each window, and then display all the windows together when (and how) they see fit. This separation inevitable leads to some latency increase. Examples of compositing managers: Aero in Windows, Compiz in Linux.

Corresponding empirical data is available in the next chapter.

V-Sync

Vertical synchronization or (V-Sync) is a method of avoiding screen tearing — a condition when a display shows information from multiple frames in a single screen draw. The resulting artifacts are mostly noticeable in horizontally-moving visuals with sharp vertical edges (i. e. not during typing).

V-Sync boils down to waiting for a new frame in generated video signal before transferring back buffer to framebuffer (so, V-Sync implies double buffering). Thus, we might expect additional delay before framebuffer update when vertical synchronization is turned on. Maximum delay is ~17 ms, average delay is ~8 ms (for 60 Hz refresh rate). Interestingly, this delay interplays with screen refresh, so average total delay is increased only by ~4 ms, because a part of the delay is unavoidable (depends on character vertical position). The maximum increase of total latency is still ~17 ms.

Desktop environments usually don’t allow to explicitly choose whether V-Sync is enabled. In Windows, V-Sync is enabled in Aero, but disabled in Classic theme. In Linux V-Sync seems to be disabled both with stacking- and compositing window managers.

In a sense, V-Sync forces GPU to adapt to monitor refresh rate, which is a legacy from CRT era. Today we have much better solutions (see below).

Let’s summarize the latency numbers for a typical computer (we assume that it’s not under significant I/O or computational load and uses stacking window manager without V-Sync):

Source Min, ms Max, ms Average, ms
Operating system 0 ?
Virtual machine 0 ?
Editor 0 ?
Rendering pipeline 0 ? ?
Double buffering 0 ? ?
Window manager 0 ? ?
V-Sync 0 0 0
Total 0 ?

Those delays are much less predictable than hardware-induced delays and they are highly-sensitive to computer configuration. In principle, the total processing latency can be as low as 0-1 ms, and as high as 10 s. That’s why theoretical musings are not enough and we need some hard numbers — see the next chapter for comprehensive empirical data.

How can we improve the processing latency? Here are a few ways:

  • Low-latency editor / IDE.
  • Powerful computer hardware (CPU, GPU).
  • Stacking window manager (Classic Windows theme, Openbox, etc).
  • No V-Sync.

Apparently, editor application is a single thing that matters the most.

2.3 Output latency

Let’s take a look at how typical, non-gaming LCD monitor displays generated picture.

Screen refresh

Computer screen picture is stored in the frame buffer inside graphics card memory (or inside shared graphics memory). Monitor cannot access the frame buffer directly, instead of this, video display controller generates video signal that is transmitted over a video interface (like VGA, DVI, HDMI, DisplayPort, etc.) so that monitor can continuously update its internal image buffer. This transmission is performed at a fixed frequency, known as refresh rate, which is equal to 60 Hz for ordinary LCD monitors. Thus, we can expect maximum refresh-related latency about 17 ms, and average latency ~8 ms (because we need to display only the newly typed character, we can ignore the time it takes to transmit the full frame).

Display lag

Unlike CRT monitors, which have to render incoming image immediately (because their only “memory” is phosphor persistence), LCD monitor contains its own buffer to hold the picture. In principle, this makes possible to delay showing of incoming image in order to resize, deinterlace or somehow “enhance” it (or just to receive a full frame before updating the pixels). The resulting delay is known as display lag (or “monitor input lag”). The phenomenon received much publicity recently, yet it’s still a rather controversial topic. Measurements of display lag are often incorrect, because it’s hard to separate true display lag from pixel response time (check an amazing Prad’s report on input lag to see how to do it properly). While worst-case latency can be as high as 2-3 frames (up to 50 ms for 60 Hz refresh rate), display lag is mostly present in HDTVs, not computer monitors, so we can assume that this latency is close to zero for typical monitors. Nonetheless, it’s a good idea to turn off all “image enhancers” in your monitor (surprisingly, including overdrive).

Pixel response

LCD pixels cannot react to controlling voltage changes immediately.The amount of time a pixel in a display takes to change is called pixel response time. This time depends mostly on LCD matrix technology (TN, IPS, VA, etc.). LCD monitors have come a long way in reducing pixel response time, so modern TN monitors (most widely used nowadays) exhibit average response times only about 4 ms.

Let’s summarize the latency numbers for the typical computer monitor:

Source Min, ms Max, ms Average, ms
Screen refresh 0 17 8
Display lag 0 ? 0
Pixel response ? ? 4
Total ? ? 12

Can we improve the output latency? Here are a few sure methods:

  • Gaming monitor that guarantees low display lag and provides faster pixel response.
  • Monitor with 144 Hz refresh rate (3.5 ms refresh-related latency).
  • Graphics card & monitor combo that supports Adaptive-Sync, FreeSync or G-Sync with dynamic refresh rates (up to 240 Hz) and synchronous updates.

Those measures can reduce total output latency to about 1-2 ms.

2.4 Total latency

Let’s summarize average delays in all the components of typing latency:

Source Typical, ms Ideal, ms
Input 14 1
Processing ? 0
Output 12 2
Total ? 3

Typical setup consists of a usual, non-gaming keyboard and an ordinary LCD monitor. Ideal configuration presumes a low-latency keyboard and a low-latency (gaming) monitor.

Both input and output components of the latency are precisely determined by periodical hardware processes that can be estimated with high accuracy. Those delays are independent of CPU / GPU performance, and have predictable upper bounds. Typical average latency of keyboard and monitor combined is about 26 ms, which is not much by itself, but when it adds up to processing delay it can make editor latency more noticeable. Even the I/O latency itself is already above the human perception threshold.

Unlike I/O latencies, processing latency is primarily caused by CPU / GPU computations, thus it depends on hardware and the maximum delay is practically unlimited. It’s impossible to predict processing latency, so we need to perform appropriate benchmarks to acquire the data. Theoretically, ideal processing latency can be close to 0-1 ms.

While every part of the chain matters, editor is the weakest link that can give rise to the major part of typing latency.

3. Editor benchmarks

To measure processing delays experimentally I created Typometer — a tool to determine and analyze visual latency of text editors (sources).

Typometer works by generating OS input events and using screen capture to measure the delay between a keystroke and a corresponding screen update. Hence, the measurement encompasses all the constituents of processing latency (i. e. OS queue, VM, editor, GPU pipeline, buffering, window manager and possible V-Sync). That is the right thing to do, because all those components are inherently intertwined with the editor, and in principle, editor application has influence on all the parts.

3.1 Configuration

Here are the details on hardware / software configuration I used for testing:

Hardware:

  • CPU: Intel Core i5 3427U, 1.8 / 2.8GHz, 3M cache
  • Graphics: Intel HD 4000 (driver v10.18.10.3958)
  • Memory: 4GB DDR3
  • Screen: 1920 x 1080, 32 bit, 59.94Hz

Software:

Editors:

Note that as I run benchmarks on the same hardware and all editor versions match, the results are directly comparable. Because results might vary depending on exact configuration, I used a rather typical setup for the benchmarking (the machine is neither too slow, nor too powerful) to make results more representative.

Only those editors that provide at least syntax highlighting were selected. Undoubtedly, Notepad is the most responsive editor, but it’s just not fare to compare it with any practically useful editor.

Additionally, I excluded cases where editor runs in a terminal emulator, because terminal emulator itself can significantly influence the typing latency. By the way, modern OSes don’t support real text mode anymore — both Windows console and Linux virtual console (via Alt+Fn) are simulated on top of the graphical framebuffer.

The analysis was performed in R. The charts were rendered via ggplot2.

3.2 Methodology

All the editors / IDEs were run with default settings, application windows were maximized.

Editor content was either an empty text file, or an XML file (namely, Maven schema). XML is chosen because it can be highlighted by all the editors and it doesn’t depend on other project files.

I used the following parameters in Typometer:

  • Character count: 200
  • Delay: 150 ms
  • Access: native
  • Mode: synchronous

The 150 ms delay is selected because that is an average time interval between key presses during my own relatively fast typing (minimum interval is about 40 ms). The exact number doesn’t matter much, since in synchronous mode Typometer waits for typed character to appear before making a pause and typing the next one.

I used Classic theme in Windows because, as I pointed out earlier, Aero’s compositing increases drawing latency and enforces V-Sync (and tests confirmed that Aero theme indeed makes all the editors less responsive). Here’s a comparison of GVim (one of the fastest editor overall) benchmarks in Classic theme and in Aero (time series with auxiliary lines at 16.68 and 33.37):

Effect of Windows Aero on drawing latency (GVim)

As we can see, Aero introduces at least one frame delay (~16.7 ms for 60 Hz refresh rate) and leads to time discretization. That conceals intrinsic editor performance and disrupts the benchmarking process. As I pointed out earlier, V-Sync-induced delays in video signal is slightly less than delays in framebuffer, but the average difference is only about 4 ms and the maximum added delay is still ~17 ms (for 60 Hz). As the effect is rather material, people often uncover the added delay “with naked eye” in human reaction time tests.

The graph also demonstrates that Typometer attains a very good accuracy in measuring the latency — the lower values lay so close to theoretical 16.68335, that it looks almost as mathematical precision.

As for Linux, Lubuntu was preferred to Ubuntu for the same reason — Compiz adds additional delay to application drawing. Here’s the graph (with median lines):

Effect of Linux Compiz on drawing latency (GVim)

The average introduced delay is about 8 ms, which is better than the Aero’s number, and there’s no synchronization-induced discretization (however there’s an increased jitter). As V-Sync is not used, delays in video signal are on a par with the delays in framebuffer. It’s preferable to use a stacking window manager (like Openbox) to achieve a better measuring accuracy.

Given that measured delays of fastest applications (like Notepad) is below 1 ms and stable, it seems safe to assume that the measuring tool provides enough precision and accuracy. Because the relationship between framebuffer and output video signal is deterministic, the results can be considered representative.

Despite the abundance of different observed behaviors, it’s possible to distinguish the following typical time series:

Typical editor latency time series

Simple editors usually exhibit very stable delay times, though the average delay might be relatively high.

Complex editors (and editors that run on top of virtual machine / interpreter) in addition to higher average latency often tend to show higher variability of the delay (jitter).

In-between behavior is also common, when periods of stable delay are periodically interrupted by occasional spikes. Sometimes values show a trend that reveals linear dependency of editor algorithms on character horizontal position.

3.3 Windows

As an introduction to analyzing latency, I recommend you to check a great talk on how not to measure latency, which explains why average values are better measure of latency than median values and why maximum values are very important.

Text file

Let’s start with a summary table (sorted by average latency). SD stands for standard deviation which is used as a measure of jitter. Keep in mind that it’s an “ideal” setup — a single empty file with no highlighting. Any other configuration possible would exhibit higher delays, by definition.

Editor latency in Windows (text file):

Editor Min, ms Max, ms Average, ms SD, ms
GVim 0.2 1.2 0.9 0.2
IDEA (zero-latency) 0.1 21.2 2.9 2.7
Notepad++ 0.1 5.9 4.3 0.8
Emacs 4.2 19.2 5.3 1.1
Sublime Text 6.2 35.2 8.2 2.0
Eclipse 0.1 20.8 10.1 1.6
Netbeans 7.3 31.6 11.8 3.9
IDEA 0.1 83.7 24.7 12.0
Atom 29.2 85.5 49.4 7.2

The summary table gives us only an approximate estimate of delay distribution so it’s good to complement the table by a corresponding fine-grained chart. Traditional histograms are not very suitable for comparing multiple sets of data. Box plots are too rough. Violin plots are better, but they looks quite strange for non-statisticians.

There’s a better way, which is also more good-looking — we can plot values on horizontal axis with some added random vertical shift to reveal distribution (more info on the approach). The resulting chart is both simple and descriptive, and it’s easy to compare the subsets:

Editor latency in Windows (text file)

Apparently, editors are not created equal (at least, from the standpoint of latency).

First, the average latency varies greatly — simpler editors tend to exhibit lower average latencies. GVim is a winner, yet IDEA with zero-latency mode enabled is at GVim’s heels. All JVM-based editors (including IDEA in default mode) are at the bottom, which is fairly predictable, surrendering the last place only to Atom, because Chrome runtime turned out to be even more sluggish.

Another noticeable difference is in jitter — “lightweight” editors demonstrate very stable delays, while complex editors, on the contrary, have a much wider delay spacing. IDEA in default mode shows the biggest spread of delays with high maximum value.

Here’s a histogram that demonstrates detailed effect of zero-latency mode in IntelliJ IDEA:

Effect of zero-latency mode in Windows (text file)

Zero-latency mode does a good job in both reducing the average delay and in eliminating the jitter (though not completely).

XML file

OK, the previous setup is too “synthetic”, after all, to edit empty files without any highlighting we have Notepad. Other editor are used for something more tangible. Let’s type in the relatively large XML file to see how the numbers will change:

Effect of content type on editor latency (Windows)

Wow! The difference is quite sensible, and also curious. Some editors are like Honey Badger — they “just don’t care”, namely GVim and IDEA in zero-latency mode. In most editors the delay is uniformly increased. Average delay in IDEA is slightly decreased (which is somewhat strange), yet the maximum delay is increased. The most impact is made on Emacs whose latency is skyrocketed.

Power-saving

Now we’re going to unplug our power cord and work on battery, in power saving mode.

Editor latency in Windows (XML file, power-saving):

Editor Min, ms Max, ms Average, ms SD, ms
GVim 0.6 2.9 1.4 0.2
IDEA (zero-latency) 1.5 58.7 4.3 7.4
Notepad++ 4.8 26.1 9.8 1.3
Sublime Text 10.4 19.3 12.6 0.7
Eclipse 13.9 60.4 19.0 5.0
IDEA 13.6 239.3 45.5 39.6
Emacs 46.1 77.4 50.5 4.3
Netbeans 11.9 138.1 59.0 21.2
Atom 48.9 104.0 60.4 7.0

Let’s examine how this affected the latencies in more detail:

Effect of power-saving mode on editor latency (Windows, XML file)

Most editors are not affected that much, but there are a few notable exceptions: IDEA and Netbeans. IDEA’s maximum delay is soared to ~240 ms, which is obviously above the threshold of “noticeable”.

Here’s how zero-latency mode does its job in those conditions:

Effect of zero-latency mode in Windows (XML file, power-saving)

Again, while it cannot eliminate the jitter completely, the effect is significant.

3.4 Linux

Let’s continue our observations in Linux, skipping the “synthetic” text part for simplicity.

Editor latency in Linux (XML file):

Editor Min, ms Max, ms Average, ms SD, ms
IDEA (zero-latency) 0.8 3.9 1.7 0.5
GVim 1.7 8.4 4.5 1.4
Gedit 5.0 24.2 12.4 3.0
Emacs 7.5 46.3 20.3 4.1
Sublime Text 9.4 35.4 23.1 5.2
Netbeans 8.4 92.7 24.5 12.9
Atom 16.0 59.6 32.5 8.4
Eclipse 23.6 87.1 46.5 14.0
IDEA 10.1 198.0 69.2 27.1

Complement chart with detailed delay distributions:

Editor latency in Linux (XML file)

Comparison with the previous results in Windows:

Effect of OS on editor latency (XML file)

The common trait is that jitter in Linux is noticeable higher for most editors (with the exception of zero-latency mode in IDEA, where jitter is actually reduced).

Emacs and Atom gains from Linux — their average latency is significantly lower.

GVim, Sublime Text, Eclipse and IDEA (in default mode), on the contrary, exhibit much more latency in Linux. IDEA is most affected — its maximum delay reaches ~200 ms even in usual power mode. Eclipse responsiveness is also severely impacted.

IDEA’s zero-latency typing is sure winner in Linux (it outperforms even GVim).

Effect of zero-latency mode in Linux (XML file)

Zero-latency mode in Linux demonstrates low- and exceptionally stable latency.

3.5 VirtualBox

Now let’s suppose, for example, that we use Windows primarily for gaming and do our work in Linux virtual machine (sure, why not?). How editor latencies will change in such a use case? Let’s see.

Editor latency in Linux (VirtualBox, XML file):

Editor Min, ms Max, ms Average, ms SD, ms
IDEA (zero-latency) 4.2 29.7 12.4 4.9
GVim 6.0 26.0 18.3 4.1
Gedit 19.5 47.6 30.1 4.8
Emacs 27.6 68.3 33.8 7.4
Sublime Text 31.5 58.2 46.1 5.2
Netbeans 12.6 159.9 46.7 19.5
Atom 29.1 89.4 62.6 10.2
Eclipse 45.3 131.4 79.4 13.7
IDEA 29.2 347.8 87.5 42.0

Here’s a comparison with the previous “native” results:

Effect of OS on editor latency (Linux, XML file)

Apart from delay discretization (caused either by V-Sync or by other kind of discrete buffer synchronization), we can observe steady latency increase for all the editors. Distributions are not altered much, which is easily explicable (underlying algorithms are not altered by visualization). IDEA’s maximum delay accrued to ~350 ms.

Power-saving

To make the picture complete let’s switch to battery. Power saving mode together with VirtualBox is probably the toughest combination when it comes to editor latency.

Editor latency in Linux (VirtualBox, XML file, power-saving):

Editor Min, ms Max, ms Average, ms SD, ms
IDEA (zero-latency) 4.3 69.6 11.9 8.9
GVim 5.0 35.8 19.0 3.8
Gedit 13.4 49.3 23.9 6.0
Emacs 19.6 79.5 44.7 6.9
Sublime Text 38.1 65.0 52.1 7.3
Atom 70.2 152.2 90.0 13.7
Netbeans 35.7 344.7 119.1 43.3
Eclipse 99.3 269.6 133.6 36.2
IDEA 119.4 544.6 198.8 63.1

Yeah, those delays are just ridiculous! The maximum delay is now more than half a second!

For illustrative purposes, let’s compare the data with the results from ideal conditions:

Effect on configuration on editor latency

Apparently, configuration plays an important role in resulting editor latency. However, editors are affected differently — performance of some editors can be severely degraded, while other editors are not affected that much.

The best resilience is demonstrated by IntelliJ IDEA with zero-latency mode enabled. Let’s see how it performs in the toughest setup:

Effect of zero-latency mode in VirtualBox (Linux, XML file, power-saving)

Interestingly, in zero-latency mode IntelliJ IDEA outperforms all the other editors (and also terminal-based editors, by the way).

4. Summary

Obviously, what you type is much more important than how you type it. Nevertheless, low-latency visual feedback can often make the process more efficient and more enjoyable.

  • Use a responsive editor (makes the most difference).
  • Use a low-latency keyboard, if possible.
  • Choose programs that add global keyboard hooks wisely.
  • Turn off unnecessary “image enhancers” in you monitor.
  • Enable stacking window manager in your OS (e.g. in Windows 7, 8).

Type with pleasure!

See also:

Interesting feedback:

Special thanks go to John Carmack and Miguel de Icaza for tweeting the article.

40 Comments

  1. Pavel says:

    Mechanical keyboards and Cherry switches are great, etc, but think about noise they produce. Not all of your co-workers will be happy.

  2. Pavel says:

    Good point, though mechanical keyboard / Cherry switches is just a specific example, proper keyboards can be completely silent as well.

    Besides, keyboard is only a nice addition, it’s editor that makes the significant improvement possible.

  3. Aleksey says:

    Pavel, thanks for your article. It’s a great work! When could be use this feature at IDEA by default?

  4. Pavel says:

    Refinement is still in progress. For a start, we’re considering enabling the feature by default in some recent IDEA EAP.

  5. stacktrace says:

    Pavel,

    Very useful work and something I have always believed that latency is negatively correlated iwth prodcutivity.

    Am I right in assuming you are comparing xemacs on Windows/Linux. The latency or xemacs is intolerable.

    I use emacs -nw in a terminal. Can you add that to your comparisons? My hunch is that its latency is comparable to vi/vim variants.

    Thanks!

  6. Pavel says:

    I tested the GUI version of Emacs in all the OSes.

    As for editors in terminal emulators, I excluded those cases because terminal emulators often produce a major part of latency (and results vary widely). Most terminal emulators seem to exhibit more latency than, for example, GVim.

  7. xah lee says:

    using xml is unfortunate for emacs, because the xml-mode in emacs 24.x does real-time xml validation on the fly, and is written by xml expert James Clark (who also wrote probably the most widely used xml parser lib in C)

    i think that using any of C, java, ruby, python files, emacs would do much better.

    PS most wonderful article. Thank you.

  8. […] Low-latency painting in AWT and Swing – Our own Pavel Fatin spent some time meticulously tracking sources of latency from keyboards to monitor pixels to bring you a better typing experience in IntelliJ IDEA. But moreover, you can apply the same techniques to AWT and Swing applications to better understand the sources of visual latency and improve your users’ experience. Pavel also published a benchmarking tool to help analyze end-to-end visual latency for text editors. Further data and analysis is available for your reading and typing pleasure. […]

  9. stingoh says:

    I would have liked to see the latency of a recent version of Visual Studio, especially on a really big project with intellisense or visual assist enabled. By default, both of those will preemptively try to name complete for the user, which can cause *severe* latency jitter on a large project. Easily addressed by letting the user trigger name completion manually. But I’m amazed by what people put up with.

  10. Ivan says:

    Pavel, it’s actually not a good point. Blue switches are loud due to their build in click in the keys’ mechanic, but other Cherry switches actually end up being less noisy than a rubber dome keyboard, because the mushy feeling on rubberdome keyboards makes people bottom out while typing, which is far noisier.

    Source: having worked with 40+ people in an open plan office.

  11. Mike says:

    Would love to see the metrics for VS Code!

  12. Pavel says:

    You can use Typometer to benchmark any editor. The tool is open source and cross-platform.

  13. Richard says:

    You ignore the cognitive parts of the system which are in the hgh 10’s to 100’s of milliseconds. Completely dominates the system.

  14. Leonid says:

    Can you please also test Akelpad text editor at same test machine:
    “vanilla” minimal (with or without plugins) – http://akelpad.sourceforge.net/en/download.php
    and some enhanced custom version pack, San_dr for example – http://forum.oszone.net/post-1889376-1375.html
    Don’t know if x64 versions have different latency – both links have 32 and 64 bit versions.

  15. Pavel says:

    Though I cannot benchmark all editors in the world, as AkelPad is fairly lightweight and portable, I’ve run several tests on the original test machine to give you a reference point so you can extrapolate numbers acquired in other desired configurations (by using Typometer).

    I employed the “XML” setup described in the “Methodology” part. The editor was from the default installer (without plugins). Optional highlighting and folding were provided by the “Coder” plugin.

    Plain editor:
    min: 6 | max: 15 | avg: 11 | SD: 1

    Highlighting enabled:
    min: 8 | max: 19 | avg: 12 | SD: 2

    Folding panel visible:
    min: 36 | max: 103 | avg: 79 | SD: 10

    Hence, AkelPad’s latency is on a par with Notepad++ and SublimeText, which is better than average, yet both GVim and IntelliJ IDEA (in zero-latency mode) can be faster than that.

    All the graphs reveal linear dependency of editor algorithms on character horizontal position. Moreover, there’s probably an excessive painting, as the latency is much lower when editor window is not maximized (even for an empty text file).

    The folding panel drastically degrades the performance (typing lag is clearly perceivable “with naked eye” when the panel is enabled).

  16. sshbio says:

    Maybe in the linux text console (TTY, Ctcl + Alt + F{1,2,3,4,5,6}), all text modes would be more responsive than graghical ones. Maybe I’m wrong, but my terminal in X11 is much slower than plain TTY. 🙂

  17. Pavel says:

    Modern OSes don’t use real text mode anymore – both Windows console and Linux virtual console (via Alt+Fn) are simulated on top of the graphical framebuffer.

    Nevertheless, it’s possible that some emulated console is more responsive that some X11 terminal emulator (and vice versa), however, in principle, they are equal because both are implemented via the framebuffer (as long as we avoid heavy-weight compositing layer).

    Thanks for mentioning the “plain TTY”, I’ve added the clarification to the article.

    Responsiveness of emulators varies greatly and the average latency usually exceeds GVim’s one. It would be interesting to compare terminal emulators as well, and everyone is welcome to do that 🙂

  18. Plugins can ruin latency. ESC no longer functions well with ConqueTerm as opposed to the rest of Vim.

  19. BSalita says:

    Awesome analysis. Thanks.

  20. oversky says:

    Very interesting benchmark.
    Are all the spell check on editors turned off?
    It may have some effect on the latency.

  21. Pavel says:

    As I stated in the “Methodology” section, all the editors / IDEs were run with default settings. That seemed to be a straightforward way to acquire baseline readings as there’s no need to argue indefinitely what features to enable / disable and what plugins to use for this or that. This is especially true considering that now we have an open source & cross-platform tool for any fine-grained testing.

    Undoubtedly, additional editor features might influence typing latency significantly (and I bet they do), however, they don’t necessarily have to. It should be perfectly possible to achieve close to zero latency even with highlighting, foldings, inspections, spellchecker and whatnot. I.e. it’s OK for a spellchecker to work slowly and to consume some CPU resources, but not at the expense of processing user typing.

    Detailed testing is useful to determine impact of each isolated feature on editor performance – in the case of AkelPad above, even a brief testing clearly revealed multiple performance “hot spots”.

  22. Pavel says:

    Here are a few tips on how you can use the tool to discover performance bottlenecks in text / code editors:

    * Check whether number of lines influences the latency. If so, typing might “lag” inside a large file.
    * Check whether editor window size influences the latency. If so, the editor is probably does excessive repainting instead of drawing only a newly inserted symbol.
    * Check whether latency depends on horizontal symbol position. Ideally, that correlation should be undetectable.
    * Try to enable highlighting / autocomplete / folding / spellchecker / etc. Those features should be processed asynchronously, without affecting the typing per se.
    * Try to run the test in power-saving mode. Ideally, typing should be handled decently even on less powerful hardware.

    I described various technical details in a separate article on low-latency painting in AWT and Swing (despite the focus on Java platform, the key ideas can be extended to most modern operation systems and GUI frameworks).

    You can use ToyIDE‘s code as a relatively simple example of how to implement some of the ideas.

  23. Pavel says:

    By the way, Raph Levien has just started Xi Editor project – “an attempt to build a high quality text editor, using modern software engineering techniques.”

    The first stated goal:
    Insanely high performance. All editing operations should commit and paint in under 16ms. The editor should never make you wait for anything.

    A related design decision:
    Asynchronous operations. The editor should never, ever block and prevent the user from getting their work done. For example, autosave will spawn a thread with a snapshot of the current editor buffer (the peristent rope data structure is copy-on-write so this operation is nearly free), which can then proceed to write out to disk at its leisure, while the buffer is still fully editable.

    It’s good to see that low latency is stated as the essentilal requirement. However, 16 ms (refresh period at 60 Hz) threshold is actually quite arbitrary. Rendering in a text / code editor significantly differs from rendering in a video game:
    * updates are not generated at a constant rate,
    * keystroke times are fundamentally unpredictable,
    * text line occupies only ~1/50 of the frame.

    Because of how all that interplays with screen refresh (and possible v-sync), there’s no formal threshold for editor painting latency – the lower, the better. We should aim for at least 1-5 ms, which is both technically possible and desirable.

  24. Anonymous says:

    > Debouncing

    There is a simple way to get zero-latency debouncing, both for actuations and deactuations. This does not solve “chatter at rest”, such as holding down a switch and having it deactuate due to mechanical noise. The method is to change actuation state in the firmware immediately on seeing the given switch’s state change, but to block further state changes on that specific switch for X ms (in my implementation, 7) after a confirmed actuation change. This makes it so that bounces back to deactuation are completely ignored for 7ms after the beginning of the actuation, but so that releases are not delayed. This also means that bounces back into actuation are not picked up.

    Another method, which has zero-latency actuation but not deactuation, that solves “chatter at rest”, is to report the switch as actuated if it was at any point actuated in the past X ms (typically something like 16). This removes all removable noise except chatter-at-rest while deactuated, which cannot be removed without increasing actuation latency, and isn’t an issue with the overwhelming majority of switch designs used in keyboards and mice.

    You can get smarter than these two methods if the switch has three states like the microswitches used in computer mice, but the overwhelming majority of mouse and keyboard firmware seems to be written by people that have no idea what they’re doing, so I don’t think you’ll see that outside of serious niche projects.

  25. Pavel says:

    That’s so. I also pondered on this paradox, because if you think about it, the potential improvements are rather obvious.

    I guess the naive approach came from microcontrollers, where either a counter or
    a shift register were used as a means of debouncing and everyone deemed that +10 ms don’t matter.

    But even 1 ms matters. By the way, here’s a great video from Microsoft Research that demonstrates this (though in a slightly different context).

  26. […] Zero-latency typing is now enabled by default […]

  27. […] Zero-latency typing is now enabled by default […]

  28. […] zero-latency typing previously available as an experimenting feature is now enabled by default. This change removes […]

  29. […] latency typing as a default. Pavel Fatin, a developer at JetBrains, has done extensive research into editor latency. With his improvements, typing in IDEA-based IDEs (like PyCharm) is now faster than it is in […]

  30. […] Zero-latency Typing, das bereits experimentell verfügbar ist, wird mit der kommenden Version fester Bestandteil der […]

  31. Johan says:

    Very nice and interesting article.

    Regarding “Windows Classic” being more responsive than “Aero” … I was wondering what we should do when we’re all forced to work on Windows 10, which no longer has the “Classic” option. Is it possible to avoid the increased latency on Windows 10? If there is no way to have it use a “Stacking Window Manager”, maybe there’s a way to have it not use V-Sync, so we can at least get similar results as with your Lubuntu vs. Ubuntu test?

  32. Pavel says:

    Hi Johan,

    Although it’s important to disable Aero theme to increase the measurement accuracy, it’s not that important for day-to-day use – the editor itself will still be the major factor that determines the resulting responsiveness. However, if you’re especially sensitive to the input latency (like some people do), you might still consider possible improvements.

    Windows 8 / 10 don’t have “Aero Glass” theme, but they do have the compositing (DWM), so the result is the same.

    In Windows 7 it’s possible to disable desktop compositing even with the Aero theme.

    In Windows 8 it’s still possible to disable the compositing, albeit in a less straightforward way.

    In Windows 10, it seems, there’s no way to disable the compositing, so we have to deal with it. A partial workaround is to use a monitor that supports higher refresh rates, e.g. 144 Hz monitor will reduce the compositing-induced lag ~2.5 times. Technically, it should be possible to selectively bypass Vsync on latency-sensitive screen updates, so there’s a hope that Microsoft will make an effort to improve the state of affairs – the company is now in business of producing hardware (Surface Book, Surface Studio) and “low-latency thing” became a critical component for usability of all the enhanced input methods, like touchscreen, pen, etc.

    Adaptive synchronization technologies, such as FreeSync or G-Sync, is an even better solution (though it’s not yet widespread).

  33. Mike says:

    You shouldn’t use blue/green in those first line-graphs. I don’t think I am colorblind, but the labels were pretty difficult for me to distinguish without focusing hard on it.

  34. […] time ago Pavel Fatin, a developer at JetBrains, researched editor latency. Based on his findings, he implemented an experimental feature for IntelliJ IDEA […]

  35. […] Zero-latency typing, introduced previously as an experimental feature to remove possible lags when typing, is now enabled by default. […]

  36. MinchinWeb says:

    I’m curious how Microsoft Word would compare. Although not (typically) a code editor, it is probably more widely used.

    Would you consider benchmarking Word?

    Thanks in advance.

Leave a Reply