synopsis

JetBrains’ IDEs suffer from awful interactive performance out of the box. This is unfortunate, and given the highly interactive nature of programming one would think it would be taken more seriously. This article gives instructions on how to change the “boot JDK” and compile the base classes of the JDK to binary form, so that they are not JIT-compiled every time one starts the IDE.

I suggest you try both operations for the maximum benefit. I am keen to hear about the results, so please tell me via firstname.lastname@gmail.com — check the domain name of this site for the values 🙂 In adjusting all of these settings, YMMV depending on your system. Good luck!

disclaimer / warning

This post contains advice / instructions on how to change your JetBrains installation. Following them may break your installation, and I do not take any responsibility for your actions in doing so 🙂

1 / stop using JDK 11-based JBR, start using a (vanilla) version 16 JDK instead

Install version 16 of the JDK such as Azul Zulu by following the instructions here. Other JVM implementations are available. I am neither paid by, nor associated with, Azul.

Change the boot JDK to the one you’ve just installed by following the instructions here.

All the EAP and latest release versions of JetBrains IDEs seem to work with JDK 16. In all cases, features using the JCEF [1] runtime library, which is bundled with the JBR [2], will not be available. If you run into problems, JDK 15 will work instead.

Change the “Custom VM options” to something like the below. I’m using these options on Debian/unstable on all JetBrains IDEs, with both JDK 15 and JDK 16.

  -XX:+IgnoreUnrecognizedVMOptions
  -XX:+UnlockExperimentalVMOptions
  -XX:+UseZGC
  -XX:-UseG1GC
  -XX:C1CompileThreshold=100
  -XX:C2CompileThreshold=5000
  -Xms4G
  -Xmx4G
  -XX:ReservedCodeCacheSize=512m
  -XX:SoftRefLRUPolicyMSPerMB=50
  -XX:CICompilerCount=2
  --illegal-access=permit

Adding -XX:ReservedCodeCacheSize= and -XX:SoftRefLRUPolicyMSPerMB= made all the difference in keeping heap usage down, although I have not yet experimented with different values for those parameters.

The option –illegal-access=permit is required on JDK 16, but this option was removed in JDK 17, and because of this (but probably not only this) I haven’t yet been able to make any JetBrains IDE work with JDK 17.

When you edit this file, make a note of the location via the “Copy path” feature in the IDE, so that if it breaks, you can find it and remove the mistake quickly. Also, you may wish to start the IDE in the console via the command line (rather than the Toolbox launcher) so that you can easily see the error.

[1] https://github.com/JetBrains/jcef
[2] which is currently based on JDK 11

2 / compile all the JDK classes ahead of time using the jaotc tool

Since day one, startup time has been a painful part of Java. Nowadays we are blessed with modules in the JDK and we can compile them ahead of time and load them into the JDK at runtime. Seems like it might a good idea for big applications, but actually the benefits are greater for smaller programs.

This script invokes the jaotc binary to create a single shared object library file from the modules in the JDK. Note that the ZGC garbage collector is not compatible with the use of jaotc.

The jaotc tool has been removed in version 17 and later of the JDK, but for now, we can make the most of it.

#! /bin/bash

set -o nounset


# java ------------------------------------------------------------------------

java_home=${JAVA_HOME:-/usr/lib/jvm/zulu16}
echo "java_home is [${java_home}]"

java_flags=( -XX:+UseG1GC -XX:-UseZGC )
java_name=$(basename ${java_home})
aot_so=${HOME}/lib/java/native/$(uname -m)-linux/aot.${java_name}.so
echo "output shared object file is [${aot_so}]"


# enumerate modules -----------------------------------------------------------

modules=()

jmod_files=( ${java_home}/jmods/*.jmod )
for _jmod in ${jmod_files[@]}; do
    module=$(basename ${_jmod/.jmod/})
    case ${module} in
        jdk.hotspot.agent) ;;
        jdk.incubator) ;;
        jdk.incubator.*) ;;
        jdk.internal.vm.compiler) ;;
        jdk.jcmd) ;;
        *) modules+=( ${module} ) ;;
    esac
done


# construct command line ------------------------------------------------------

argv=()
argv+=( ${java_home}/bin/jaotc )
for flag in ${java_flags[@]}; do
    argv+=( -J$flag )
done
argv+=( --compile-for-tiered )
argv+=( --output ${aot_so} )
argv+=( --module ${modules[@]} )


# execute --------------------------------------------------------------------

mkdir -p $(dirname ${aot_so})
${argv[@]}

This step above will take about ten minutes or so, depending on your system.

Add these lines to your IntelliJ custom VM options. The path matches the output path in the script above, and the “GC” options match the JVM options in the script as well. (As mentioned above, the ZGC garbage collector is not compatible with the use of jaotc.) Note that this is different from the previous section where ZGC was enabled.

-XX:+UnlockExperimentalVMOptions
-XX:AOTLibrary=/home/username/lib/java/native/x86_64-linux/aot.zulu16.so
-Xshare:on
-XX:+UseG1GC
-XX:-UseZGC
-XX:+PrintAOT

(Note the /home/username and change it to your home directory.) Restart the IDE — and you should see it start much faster. Remove or comment out -XX:+PrintAOT when you are satisfied it’s working as expected.

This is the minimal-but-complete setup required for Debian 10 onwards.

Put the lines below into /etc/modprobe.d/snd_hda_intel.conf and reboot.

# snd_hda_intel ------------------------------

options snd_hda_intel beep_mode=0
options snd_hda_intel enable=1,0,0
options snd_hda_intel enable_msi=1
options snd_hda_intel model=alc221-hp-mic
options snd_hda_intel power_save=0
options snd_hda_intel power_save_controller=N
options snd_hda_intel probe_only=0,0,1

/etc/modprobe.d/snd_hda_intel.conf

You should find that the audio behaves sanely from now on.

My setup / usage is that I have my 3.5mm dual-jack headphones plugged into the headphone and microphone sockets on the front, and my external amplifier and speakers plugged into the line out socket on the rear.

The audio comes out of both front and rear sockets, the front microphone works, and plugging something into the front headphone socket mutes the output on the rear headphone socket.

Normally I have my headset plugged into the headphone and microphone socket on the front for conference calls. If I want to watch a video with proper sound, I partially unplug the front headphone jack (leaving it teetering in the socket) and the rear socket wakes up and sound comes out of the speakers through the external amplifier.

There is a problem here though — after a reboot, with headphones plugged in the front and amplifier plugged into the rear, no sound comes out of neither socket until I pull and replace the jack from the front headphone socket. If you are reading this and know of a way around this, so that sound comes out of both outputs on boot without manual intervention, please let me know. 🙂

As I walked out one evening,
   Walking down Bristol Street,
The crowds upon the pavement
   Were fields of harvest wheat.

And down by the brimming river
   I heard a lover sing
Under an arch of the railway:
   ‘Love has no ending.

‘I’ll love you, dear, I’ll love you
   Till China and Africa meet,
And the river jumps over the mountain
   And the salmon sing in the street,

‘I’ll love you till the ocean
   Is folded and hung up to dry
And the seven stars go squawking
   Like geese about the sky.

‘The years shall run like rabbits,
   For in my arms I hold
The Flower of the Ages,
   And the first love of the world.’

But all the clocks in the city
   Began to whirr and chime:
‘O let not Time deceive you,
   You cannot conquer Time.

‘In the burrows of the Nightmare
   Where Justice naked is,
Time watches from the shadow
   And coughs when you would kiss.

‘In headaches and in worry
   Vaguely life leaks away,
And Time will have his fancy
   To-morrow or to-day.

‘Into many a green valley
   Drifts the appalling snow;
Time breaks the threaded dances
   And the diver’s brilliant bow.

‘O plunge your hands in water,
   Plunge them in up to the wrist;
Stare, stare in the basin
   And wonder what you’ve missed.

‘The glacier knocks in the cupboard,
   The desert sighs in the bed,
And the crack in the tea-cup opens
   A lane to the land of the dead.

‘Where the beggars raffle the banknotes
   And the Giant is enchanting to Jack,
And the Lily-white Boy is a Roarer,
   And Jill goes down on her back.

‘O look, look in the mirror,
   O look in your distress:
Life remains a blessing
   Although you cannot bless.

‘O stand, stand at the window
   As the tears scald and start;
You shall love your crooked neighbour
   With your crooked heart.’

It was late, late in the evening,
   The lovers they were gone;
The clocks had ceased their chiming,
   And the deep river ran on.

W. H. Auden — 1907-1973

What The Heart Of The Young Man Said To The Psalmist.

Tell me not, in mournful numbers,
   Life is but an empty dream!
For the soul is dead that slumbers,
   And things are not what they seem.

Life is real! Life is earnest!
   And the grave is not its goal;
Dust thou art, to dust returnest,
   Was not spoken of the soul.

Not enjoyment, and not sorrow,
   Is our destined end or way;
But to act, that each to-morrow
   Find us farther than to-day.

Art is long, and Time is fleeting,
   And our hearts, though stout and brave,
Still, like muffled drums, are beating
   Funeral marches to the grave.

In the world’s broad field of battle,
   In the bivouac of Life,
Be not like dumb, driven cattle!
   Be a hero in the strife!

Trust no Future, howe’er pleasant!
   Let the dead Past bury its dead!
Act,— act in the living Present!
   Heart within, and God o’erhead!

Lives of great men all remind us
   We can make our lives sublime,
And, departing, leave behind us
   Footprints on the sands of time;

Footprints, that perhaps another,
   Sailing o’er life’s solemn main,
A forlorn and shipwrecked brother,
   Seeing, shall take heart again.

Let us, then, be up and doing,
   With a heart for any fate;
Still achieving, still pursuing,
   Learn to labor and to wait.

Henry Wadsworth Longfellow, 1838

abstract There have been thousands of articles written about Python virtual environments. As of 2020-10-07, a quick search on Google for “python virtualenv article” returns “about 1,450,000 results”. In the majority of cases I have seen, they are partially filled with impractical advice. They gush about pip, pipx, venv, virtualenvwrapper, pipenv, poetry and others, but they don’t explain how to achieve an ergonomic execution of a Python program once the virtualenv is in place, and in all environments. To be clear, too many of these articles imply, infer, or gloss over an ugly reality of their suggested approach — that that command line used to run project components will be different between environments, thus turning something trivial into something diabolical.

This article attempts to correct that by explaining how to achieve a smooth as possible deployment experience across environments (from local development, to remote production). This is at once both pretty basic and very important, but I have found that “doing the basic stuff really well” has proved to be a profitable approach.

preliminaries / terminology

It’s important to be clear about what it is we are discussing.

versions When “Python” is mentioned here, version 3.6 or later of CPython on a Debian-based Linux installation is assumed. Win32 and Python 2, fine platforms that they are, do not come under consideration, except perhaps by way of comparison to illustrate a concept or example.

venv The "venv" module does the same job as the “virtualenv” module from Python 2. It creates and populates the .venv/ directory.

pip The “pip” module is the tool which installs modules into .venv/lib/.../site-packages/ for use by the project. By the convention described here, the packages and their versions are enumerated in the file etc/pip/requirement.txt (singular)

bash We use bash for a wrapper script to make the .venv/ environment easily accessible. [1]

structure Source code arrangement is a topic all-too-often overlooked. I suggest that the filesystem-level layout of your project is as just as important as any other design decision or ergonomic consideration. Think of it like a sign saying “Please keep the workshop and van clean and tidy.” It’s a good idea, even if we don’t manage it all the time, but so much else is easier and quicker if we do.

polyglot Python is not the only game in town. It is common for large projects to use more than one language. These languages and their associated runtime systems will have different conventions on how things work. To reduce friction in daily navigation, all languages, their libraries and tooling, must live in predictable, obvious locations in the project tree. A little consideration goes a long way.

Things that this solution does not do.

Multiple python versions The solution presented here is a minimal-but-complete solution for the common case. It does not attempt to to rival or compete with any of the capabilities of virtualenvwrapper, pipenv, poetry and others. I understand that library maintainers may wish to run their unit tests on multiple versions of the Python runtime, and this is a laudable goal.

Multiple transitive library dependencies If your project requires both library A and B, and A requires library C version 4, and B requires library C version 5 we have a problem which no virtualenv management tool can reasonably resolve unless it starts adjusting module locations at installation time, and rewriting imports of library C within libraries A and B. While this may be conceivable, it is not a robust approach, and on balance should be avoided.

installation

bin/venv-create This invokes the venv module to create the .venv/ directory, and pip to populate the latter with the modules listed in etc/pip/requirement.txt

execution

There are two wrapper scripts for convenience / practicality. (A PhD student working as a TA once asked me what a wrapper script was. I mean, had he no imagination?) The main purpose of wrapper scripts is to ensure the sanity of the execution environment, so to call them “convenience scripts” is to understate their necessity.

bin/venv-python This sets PYTHONPATH to include src/python/main/, and PYTHONDONTWRITEBYTECODE and then calls .venv/bin/python, passing along any arguments.

If invoked via a symlink, it invokes .venv/bin/python3 with the “-m” option, passing along any arguments, the first one being the module name, which it gets from the source name of the symlink. This allows us to create something the symlink

bin/jupyterlab -> venv-python

and running bin/jupyterlab does the right thing, with the expected environment, without typing a long path and all is well.

bin/run-main Purely as a matter of convention, this runs src/main/python/main.py via bin/venv-python — you will likely want to add further “run-*” scripts as copies of this to suit your requirements.

That is pretty much it — some simple execution machinery underneath to create a predictable interface above, allowing the user to run the system in the same fashion in all environments. In the end, isn’t that what it’s all about?

complaints department

Thank you for your time and attention. If you have anything to say (either good or bad) about what I have written above, I would be very grateful to hear from you. Alternatively, if you have nothing to say but you like it anyway, then please tell a friend.

[1] The GNU bash shell is to be tamed, not avoided. If you think that knowing this is beneath you, then you need to think again. A serious chef keeps sharp knives.

Turning and turning in the widening gyre
The falcon cannot hear the falconer;
Things fall apart; the centre cannot hold;
Mere anarchy is loosed upon the world,
The blood-dimmed tide is loosed, and everywhere
The ceremony of innocence is drowned;
The best lack all conviction, while the worst
Are full of passionate intensity.

Surely some revelation is at hand;
Surely the Second Coming is at hand.
The Second Coming! Hardly are those words out
When a vast image out of Spiritus Mundi
Troubles my sight: somewhere in sands of the desert
A shape with lion body and the head of a man,
A gaze blank and pitiless as the sun,
Is moving its slow thighs, while all about it
Reel shadows of the indignant desert birds.
The darkness drops again; but now I know
That twenty centuries of stony sleep
Were vexed to nightmare by a rocking cradle,
And what rough beast, its hour come round at last,
Slouches towards Bethlehem to be born?

W. B. Yeats, 1865-1939

UNIX neophytes are often to be heard raving about how “everything is a file”. In UNIX, things are simple, after all, and this is to be celebrated, we are told.

What they really mean to say is that “every process accesses data via file descriptors using the open(), close(), read(), and write() system calls”, but neither does that roll off the tongue as easily nor sound like a reason for celebration. Now, this uniformity is all well and good from an API perspective, but if everything were indeed a file, then every file, directory, and network socket would be addressable by name system-wide (or perhaps in the same namespace if one is thinking in terms of Linux containers), and not just per-process file descriptor), and the celebrations would be back on.

This article is not about that, though.

In the current POSIX setup, directories are opened via opendir(), read by readdir(), and closed by closedir(). These three are library functions, not system calls (and perhaps this is the problem). Regular files are opened via open(), read from by read(), written to by write(), and closed by close().

But imagine if a given path on a system could be opened either as a file or as a directory depending on the desired usage. What user-level data representations would this enable? What would this prevent? What, if anything, would this added functionality break?

This would allow a normal-looking regular file such as some/where/file.dat to possess “sub-resources” (this is not an official term, just one I’ll use in this article, and they’ll appear as underlined italic for clarity) such as some/where/file.dat/index-001.txt, some/where/file.dat/summary.txt, or some/where/file.dat/results.txt. An entire directory tree could exist in the same places as these sub-resources given as examples.

Similarly, a directory such as some/where/amazing.app/ which looks like an application directory could be opened as a file and therefrom one could some metadata might be readable, such as a set of compatibility requirements, or application signatures.

In short then, under this scheme, all files may be opened as directories, and all directories may be opened as files. The former gives the ability to add multiple files of metadata to an existing directory of data, for example. The latter allows for applying metadata labels to a directory, among other possibilities.

references https://www.microsoft.com/en-us/research/uploads/prod/2011/10/WhatIsAFile.pdf

M.2 SSD is Samsung MZV7S2T0BW This is a 2TB device.

PCI carrier board is HP MS-4365 Make sure you get one with a heatsink and two thermal conductive pads. One of the pads is slightly thicker than the other. The thicker one goes between the heatsink and the M.2 SSD. The thinner one probably won’t be any use.

With the heatsink, the M.2 SSD will run at about 45C. Without the heatsink, it will run at about 65C or higher. It will throttle its performance at these higher temperatures, and presumably wear out sooner rather than later.

HP Z240 SFF has four PCI slots. Install the MS-4365 carrier board into the 1 PCI Express Gen3 slot x 16 mechanical / x 4 electrical (LP, half length) slot with the low-profile tang. Connect the board to the terminal on the motherboard for the activity light. Note that this is not a SATA device, so the activity light on the front of the exterior will not illuminate when this device is err, active.

references

HP Z240 Small Form Factor Workstation Specifications

Samsung 970 EVO Plus 2 TB PCIe NVMe M.2 (2280) Internal Solid State Drive (SSD) (MZ-V7S2T0)

security Security matters are not considered here. If this concerns or bothers you, please stop reading.

display server A display server is something which provides the service of display to something else. That “something else” is a client program. The terminology is sometimes confusing because the user interacts with the client program via the display server, the latter being a program running on the user’s device. This setup is very useful when one needs to have a graphical experience with a program running on a remote system.

remark However, the main effect is to confuse new users with the ambiguous terminology. This is the only situation of which I know where the user controls a client thing through a server thing.

browser The web browser is a case of arrested development. It’s not a display server, but it’s where web applications display themselves. Much has been written about “the browser as operating system”. This sounds clever, but does not really provide much insight, and is only true for a narrow sense of the term “operating system”.

browser apps these occupy an anonymous ground running in the browser on the desktop or on a mobile device.

electron apps these live in a peculiar location — running in a browser which can only execute JavaScript — but one that is used as a desktop application.

universality Given that digital content is made up of many formats, we need a tool to be able to (at least) render all of them — to literally browse collections of local and remote content, and to do so without having to install another application on one’s device.

renderer A universal document renderer is able to gain new abilities to display newly-encountered media types (such as audio), and newly-encountered encoding formats (such as MP3, OGG, JPEG, MOV, …), and does so without interrupting the user, except perhaps for confirming the addition of the new ability.

content types A universal document renderer has a bootstrap architecture, and broadly speaking, is little more than a framework for rendering plugins of various media types. This demotes the DOM / HTML / CSS combination from its current triumvirate status to be that of “just another content type”, and permits new expressions of interactive media.

transfer protocols HTTP could also lose its place as the dominant transfer protocol. URL notation is sufficient to enable the multi-protocol handling central to the bootstrapping nature of the system. HTTP remains as a basic transport for “DOM / HTML / CSS” content and renderer implementations.

implementations Code for rendering lives in repositories on the network. Universal document renderers are directed by content type metadata to fetch implementations of decoders for newly-encountered content types from these repositories. Once fetched, they remain cached locally by the framework until a new version is available, or they are no longer required and may be deleted. Universal document renderers may also check repositories periodically to ensure they have the desired (usually the latest) versions.

conclusion Browsers are lacking in flexibility to be classified as universal document renderers. Browsers have not gained the capability of rendering all possible document formats. In practice, the dominant document format is DOM via HTML and CSS, with PDF closely behind. Video (and to a lesser extent audio), have their own codecs but this is decided mainly by the main video sites and browser producers.

However, as browsers have evolved, they have effectively become a possible successor to display servers such as X11. This may be attributed to the efforts of developers of web frameworks, more than to those of browser authors.

The “next display server” is a universal document renderer, incremental in nature, and invisible in action. A display server is a system-level concept. A universal document renderer is a user- and application-level concept.