lost vocabulary, local maxima… wtf?!

For maximum effect listen to this in the background https://www.youtube.com/watch?v=qrO4YZeyl0I

Friends, I want to tell you about a heist.

Forty years ago, a group of brilliant people at Xerox PARC built a cathedral of ideas. They gave us windows, icons, menus, pointers — and we said “thank you”, and we stopped there. We took their demonstration and made it our destination.

This has to be the biggest wasted opportunity in the history of the information society. Alan Kay didn’t build Smalltalk so you could just double-click on folders. He built it so an eight-year-old could inspect a running system, find out how it worked, and change it with immediate effect.

A computer is a medium for thought, not a static filing cabinet with pictures.

Somewhere between then and now, we committed a crime against ourselves. We took a system designed to make the user powerful and turned it into a system designed to make the user comfortable. We traded agency for familiarity. We exchanged a language with infinite expressiveness for a vocabulary of twelve gestures and a hamburger menu.

Think about what we now accept as normal:

  • You cannot ask your computer “what just changed?”
  • You cannot say “show me everything I was working on last Tuesday.”
  • You cannot point at two things and say “connect these.”
  • You cannot look at your own system and see what it is doing.

and no, I don’t mention or consider any advertising-related content

The desktop metaphor told us

  • that computers work like “desks”
  • Files go in “folders”
  • Applications are separate rooms you visit,

Unfortunately, we believed it so hard that we forgot it was a metaphor — a simplification for beginners that became a prison for everyone, which became the biggest slow-motion disaster in the history of computing.

We live in a world of infinitely “copyable”, instantly “linkable”, universally addressable information, but we interact with it using a spatial model designed for paper that can only be in one place at a time. This is not a technical limitation. This is a failure of imagination that calcified into an industry.

Forty years ago, Steve Jobs invoked Orwell to sell you a beige box. But here’s the truth he didn’t mention: the (Orwellian) telescreen in 1984 wasn’t frightening because it watched you. It was frightening because it reduced the language. Newspeak didn’t add surveillance — it removed words. It made certain thoughts unthinkable by making them unspeakable.

This is exactly what the desktop metaphor has done; it has reduced the vocabulary in the name of simplicity at the cost of agency.

When the only verbs are “open“, “close“, “save“, “copy“, “paste” — when the only nouns are “files” and “folders“, and the only relationship is therefore one of containment, you cannot think thoughts that require richer grammar. You cannot imagine systems where objects talk to each other — where everything is queryable, or where the history of your own work is visible and navigable, or where the computer is not a tool you operate, but a partner in thought.

The Macintosh didn’t free us from Big Brother. It gave us a smaller cage with nicer furniture.

But here is the beautiful secret: the cage was never locked. The good ideas: Morphic. Smalltalk. Hypercard: they never went away. The notion that objects should be tangible — that you should be able to inspect a running system, modify it, or connect it to other things. The idea that a computer is not a appliance but an environment — a place where you live and think and build.

These ideas didn’t fail. They were abandoned.

Fortunately, abandonment is reversible. We do not need permission from Apple, Microsoft, or Google to build something better. We need only the courage to stop genuflecting at the altar of familiarity. So let us ask: what if we remembered what we were originally given ?

If you as an intellectual worker, had a system which had no “applications”, only capabilities that could be composed? If, in this system, every object knew how to describe itself, query itself, and link itself to other objects at your command? What advantage could you gain if your information system’s memory was your memory — searchable, associative, alive?

What if the screen was not a desktop but a viewport upon an infinite space where your work exists spatially, temporally, semantically? Where you could zoom in to the details and zoom out to the patterns? Where nothing was hidden in folders and everything was one question away?

This is not science fiction. This is archaeology. These systems existed. They were working. They were better conceived. They may have been plagued with low-quality graphics and fonts, but their semantic thesis was sound.

We just forgot.

The desktop metaphor is not wrong because it’s old. It’s wrong because it’s a ceiling disguised as a floor. It makes the simple things simple and the powerful things impossible. It was never right to begin with.

Let’s stop seeking to improve the local maxima and start climbing the global maxima, which is in easy view if we only stop to think and consider how best to interact with our media content and each other.

The computer is the most powerful thinking tool humanity has ever invented, and yet we have spent forty years using it to simulate the furniture on a desk from 1973. We can and must do better. The vocabulary of thought (actions on data) should be expanding, not shrinking.

simplicity

“Simplicity is a great virtue but it requires hard work to achieve it and education to appreciate it. And to make matters worse: complexity sells better.” — Edsger W. Dijkstra

Bryan Liles says it clearly as well:

command dispatch via bash ${array[@]} notation, oh my!

“Programs must be written for people to read, and only incidentally for machines to execute.”
— Harold “Hal” Abelson

This highfalutin aphorism applies from the most cerebral application programming language of your choice, right down to the humble (or not-so-humble) bash script. Love or hate them, bash scripts are the primary method of launching processes on Linux in many settings. (You may well run some kind of fancy-pants orchestration tool, but when that calls into your app it is 50/50 if it ends up calling a bash wrapper script or not.) I think of bash as a tool for writing things which set the execution context and run the main program, and no more than that.

All too often, one sees examples on the web of people saying “launch this program under docker via this bash one-liner”, and then they proceed to demonstrate with a command-line which goes on and on for ages and never seems to stop, very much like this sentence.

Sometimes, people are considerate enough to lay out their example over several lines, all but the last of which has a trailing back slash character “\” to indicate to bash that the command continues on the next line. Here’s a contrived example of what I mean.

#! /bin/bash

/usr/bin/java \
    -XX:+UnlockExperimentalVMOptions \
    -XX:+PrintFlagsFinal \
    -version

Unfortunately, there are a few problems with this notation in the real world. (The above example is not the real world.)

  • It does not allow for individual lines to be commented out.
  • It does not allow for any characters after the \, which means that
    • it does not allow for explanatory comments to be added at the end of a line
    • invisible (depending on the text editor) space characters can creep in after the backslash and cause weird and annoying error messages

Because of the above, this syntax looks and feels brittle. In the real world (that thing again) one often wishes to comment and uncomment command line options when experimenting with a new configuration. Never fear, bash “array notation” to the rescue…

#! /bin/bash

argv=( /usr/bin/java )  # plain old java
argv+=( -XX:+UnlockExperimentalVMOptions )
# argv+=( -XX:+PrintFlagsFinal )
argv+=( -version )

"${argv[@]}"

This solves all three of the original problems, but it looks a bit clumsy, what with the repeated mention of the variable name and brackets on every line. All I’m doing is enumerating the arguments to a command for goodness’ sake.

Happily, we can remove the += stuff and most of the brackets, make it a single array, and get straight to the point, viz.

#! /bin/bash

argv=(

    /usr/bin/java
        -XX:+UnlockExperimentalVMOptions  # trailing comment
        # -XX:+PrintFlagsFinal
        -version

)

"${argv[@]}"

Ta-da! Note that this only works with single commands; if you have multiple commands which need to be connected via pipes they will each need to be declared as individual arrays and called as one pipeline thus:

"${first_command[@]}" | "${second_command[@]}"

If you’ve ever had cause to tangle with the joy that is “setenv.sh” on the installation of an Atlassian application, I’m sure you’ll appreciate the clarity and usability of the above syntax. Yes, I know they have to run under other systems like FreeBSD or Solaris, which most likely don’t have bash. I’m talking about in-house stuff, where the toolset is well-known.

Update: I have recently come across this quote from someone, who, as usual, sums it up brilliantly:

“Any fool can write code that a computer can understand.
Good programmers write code that humans can understand.”

— Kent Beck

hp Z240 SFF sound configuration redux

summary

This is all the configuration required, on Linux 6.0.0-6-amd64 on an hp Z240 SFF PC. Put these lines in /etc/modprobe.d/snd_hda_intel.conf and reboot.

options snd_hda_intel model=alc221-hp-mic
options snd_hda_intel power_save=0

options snd_hda_intel power_save_controller=N
options snd_hda_intel enable_msi=1

# /etc/modprobe.d/snd_hda_intel.conf

problems

With enable_msi set to 0 (the default), while happily playing sound for a random indeterminate period of time, the sound system would lose grip and leave me with nothing but buzzing in the headphones, and only a reboot would return things to normal. Sometimes, unloading and reloading the sound modules did the trick, but often things would be stuck waiting for the hardware to return to normal, which it had no intention of doing. Enabling Message Signalled Interrupts avoids this problem.

power-saving mode

Sometimes the sound will play for a fraction of a second, and then immediately change to static noise on the headphones. The fact that this happens at the start of playback, presumably just after the sound hardware has been brought out of power-saving mode, points towards disabling power-saving to be the way forwards. There are two parameters related to power, and they are both recommended, even though they will not be applicable to all hardware.

options snd_hda_intel power_save=0
options snd_hda_intel power_save_controller=N

references

bash / ${APP_HOME}

During the production and deployment of any application larger than “hello world”, careful thought must be given to accommodating a likely set of changes.

One of these is deployment location — the application must not have any hard-coded file system locations present, and must determine its location at runtime.

Every bash script should start with this little section.

readonly _REALPATH=$(realpath "${BASH_SOURCE[ 0 ]}")
readonly _BASENAME=$(basename "${_REALPATH}")
readonly _DIRNAME=$(dirname "${_REALPATH}")
readonly _ROOT=$(dirname "${_DIRNAME}")
readonly _PREFIX=$(dirname "${_ROOT}")

Then you can say, for example:

export THIS_APP_HOME=${_ROOT}

and use that variable within and across the whole application (call it “this-app”) and pass it in as an application property, for example. If the script above lives at /deploy/some/where/this-app/bin/start-component everything will be relative to /deploy/some/where and if you need to refer to another application (call it “other-app”) installed at the same level, you can say

export OTHER_APP_HOME=${_PREFIX}/other-app/ 

Remember that it’s helpful to use a prefix such as “THIS_APP” for environment variables specific to each application.

JetBrains EAP apps now use JDK17-based JBR

This is great news. All JetBrains apps on Linux are now usable. Only on the EAP versions for now.

The right thing to do is to remove all the tweaks put in with JDK11 and before and use the vanilla settings. However, one still might want to bump the JVM max memory -Xmx parameter up from the relatively low default of 750MB to 2G. This can be done via the JetBrains toolbox.

I’m pleased that this has happened, but it’s been a painful time having to use JDK11 as the JBR on Linux thus far. Looking forward…

making JetBrains IDEs run faster…

synopsis

JetBrains’ IDEs suffer from awful interactive performance out of the box. This is unfortunate, and given the highly interactive nature of programming one would think it would be taken more seriously. This article gives instructions on how to change the “boot JDK” and compile the base classes of the JDK to binary form, so that they are not JIT-compiled every time one starts the IDE.

I suggest you try both operations for the maximum benefit. I am keen to hear about the results, so please tell me via firstname.lastname@gmail.com — check the domain name of this site for the values 🙂 In adjusting all of these settings, YMMV depending on your system. Good luck!

disclaimer / warning

This post contains advice / instructions on how to change your JetBrains installation. Following them may break your installation, and I do not take any responsibility for your actions in doing so 🙂

1 / stop using JDK 11-based JBR, start using a (vanilla) version 16 JDK instead

Install version 16 of the JDK such as Azul Zulu by following the instructions here. Other JVM implementations are available. I am neither paid by, nor associated with, Azul.

Change the boot JDK to the one you’ve just installed by following the instructions here.

All the EAP and latest release versions of JetBrains IDEs seem to work with JDK 16. In all cases, features using the JCEF [1] runtime library, which is bundled with the JBR [2], will not be available. If you run into problems, JDK 15 will work instead.

Change the “Custom VM options” to something like the below. I’m using these options on Debian/unstable on all JetBrains IDEs, with both JDK 15 and JDK 16.

  -XX:+IgnoreUnrecognizedVMOptions
  -XX:+UnlockExperimentalVMOptions
  -XX:+UseZGC
  -XX:-UseG1GC
  -XX:C1CompileThreshold=100
  -XX:C2CompileThreshold=5000
  -Xms4G
  -Xmx4G
  -XX:ReservedCodeCacheSize=512m
  -XX:SoftRefLRUPolicyMSPerMB=50
  -XX:CICompilerCount=2
  --illegal-access=permit

Adding -XX:ReservedCodeCacheSize= and -XX:SoftRefLRUPolicyMSPerMB= made all the difference in keeping heap usage down, although I have not yet experimented with different values for those parameters.

The option –illegal-access=permit is required on JDK 16, but this option was removed in JDK 17, and because of this (but probably not only this) I haven’t yet been able to make any JetBrains IDE work with JDK 17.

When you edit this file, make a note of the location via the “Copy path” feature in the IDE, so that if it breaks, you can find it and remove the mistake quickly. Also, you may wish to start the IDE in the console via the command line (rather than the Toolbox launcher) so that you can easily see the error.

[1] https://github.com/JetBrains/jcef
[2] which is currently based on JDK 11

2 / compile all the JDK classes ahead of time using the jaotc tool

Since day one, startup time has been a painful part of Java. Nowadays we are blessed with modules in the JDK and we can compile them ahead of time and load them into the JDK at runtime. Seems like it might a good idea for big applications, but actually the benefits are greater for smaller programs.

This script invokes the jaotc binary to create a single shared object library file from the modules in the JDK. Note that the ZGC garbage collector is not compatible with the use of jaotc.

The jaotc tool has been removed in version 17 and later of the JDK, but for now, we can make the most of it.

#! /bin/bash

set -o nounset


# java ------------------------------------------------------------------------

java_home=${JAVA_HOME:-/usr/lib/jvm/zulu16}
echo "java_home is [${java_home}]"

java_flags=( -XX:+UseG1GC -XX:-UseZGC )
java_name=$(basename ${java_home})
aot_so=${HOME}/lib/java/native/$(uname -m)-linux/aot.${java_name}.so
echo "output shared object file is [${aot_so}]"


# enumerate modules -----------------------------------------------------------

modules=()

jmod_files=( ${java_home}/jmods/*.jmod )
for _jmod in ${jmod_files[@]}; do
    module=$(basename ${_jmod/.jmod/})
    case ${module} in
        jdk.hotspot.agent) ;;
        jdk.incubator) ;;
        jdk.incubator.*) ;;
        jdk.internal.vm.compiler) ;;
        jdk.jcmd) ;;
        *) modules+=( ${module} ) ;;
    esac
done


# construct command line ------------------------------------------------------

argv=()
argv+=( ${java_home}/bin/jaotc )
for flag in ${java_flags[@]}; do
    argv+=( -J$flag )
done
argv+=( --compile-for-tiered )
argv+=( --output ${aot_so} )
argv+=( --module ${modules[@]} )


# execute --------------------------------------------------------------------

mkdir -p $(dirname ${aot_so})
${argv[@]}

This step above will take about ten minutes or so, depending on your system.

Add these lines to your IntelliJ custom VM options. The path matches the output path in the script above, and the “GC” options match the JVM options in the script as well. (As mentioned above, the ZGC garbage collector is not compatible with the use of jaotc.) Note that this is different from the previous section where ZGC was enabled.

-XX:+UnlockExperimentalVMOptions
-XX:AOTLibrary=/home/username/lib/java/native/x86_64-linux/aot.zulu16.so
-Xshare:on
-XX:+UseG1GC
-XX:-UseZGC
-XX:+PrintAOT

(Note the /home/username and change it to your home directory.) Restart the IDE — and you should see it start much faster. Remove or comment out -XX:+PrintAOT when you are satisfied it’s working as expected.

Debian 10 / hp Z240 SFF / audio setup

This is the minimal-but-complete setup required for Debian 10 onwards.

Put the lines below into /etc/modprobe.d/snd_hda_intel.conf and reboot.

# snd_hda_intel ------------------------------

options snd_hda_intel beep_mode=0
options snd_hda_intel enable=1,0,0
options snd_hda_intel enable_msi=1
options snd_hda_intel model=alc221-hp-mic
options snd_hda_intel power_save=0
options snd_hda_intel power_save_controller=N
options snd_hda_intel probe_only=0,0,1

/etc/modprobe.d/snd_hda_intel.conf

You should find that the audio behaves sanely from now on.

My setup / usage is that I have my 3.5mm dual-jack headphones plugged into the headphone and microphone sockets on the front, and my external amplifier and speakers plugged into the line out socket on the rear.

The audio comes out of both front and rear sockets, the front microphone works, and plugging something into the front headphone socket mutes the output on the rear headphone socket.

Normally I have my headset plugged into the headphone and microphone socket on the front for conference calls. If I want to watch a video with proper sound, I partially unplug the front headphone jack (leaving it teetering in the socket) and the rear socket wakes up and sound comes out of the speakers through the external amplifier.

There is a problem here though — after a reboot, with headphones plugged in the front and amplifier plugged into the rear, no sound comes out of neither socket until I pull and replace the jack from the front headphone socket. If you are reading this and know of a way around this, so that sound comes out of both outputs on boot without manual intervention, please let me know. 🙂

As I Walked Out One Evening

As I walked out one evening,
   Walking down Bristol Street,
The crowds upon the pavement
   Were fields of harvest wheat.

And down by the brimming river
   I heard a lover sing
Under an arch of the railway:
   ‘Love has no ending.

‘I’ll love you, dear, I’ll love you
   Till China and Africa meet,
And the river jumps over the mountain
   And the salmon sing in the street,

‘I’ll love you till the ocean
   Is folded and hung up to dry
And the seven stars go squawking
   Like geese about the sky.

‘The years shall run like rabbits,
   For in my arms I hold
The Flower of the Ages,
   And the first love of the world.’

But all the clocks in the city
   Began to whirr and chime:
‘O let not Time deceive you,
   You cannot conquer Time.

‘In the burrows of the Nightmare
   Where Justice naked is,
Time watches from the shadow
   And coughs when you would kiss.

‘In headaches and in worry
   Vaguely life leaks away,
And Time will have his fancy
   To-morrow or to-day.

‘Into many a green valley
   Drifts the appalling snow;
Time breaks the threaded dances
   And the diver’s brilliant bow.

‘O plunge your hands in water,
   Plunge them in up to the wrist;
Stare, stare in the basin
   And wonder what you’ve missed.

‘The glacier knocks in the cupboard,
   The desert sighs in the bed,
And the crack in the tea-cup opens
   A lane to the land of the dead.

‘Where the beggars raffle the banknotes
   And the Giant is enchanting to Jack,
And the Lily-white Boy is a Roarer,
   And Jill goes down on her back.

‘O look, look in the mirror,
   O look in your distress:
Life remains a blessing
   Although you cannot bless.

‘O stand, stand at the window
   As the tears scald and start;
You shall love your crooked neighbour
   With your crooked heart.’

It was late, late in the evening,
   The lovers they were gone;
The clocks had ceased their chiming,
   And the deep river ran on.

W. H. Auden — 1907-1973

A Psalm of Life

What The Heart Of The Young Man Said To The Psalmist.

Tell me not, in mournful numbers,
   Life is but an empty dream!
For the soul is dead that slumbers,
   And things are not what they seem.

Life is real! Life is earnest!
   And the grave is not its goal;
Dust thou art, to dust returnest,
   Was not spoken of the soul.

Not enjoyment, and not sorrow,
   Is our destined end or way;
But to act, that each to-morrow
   Find us farther than to-day.

Art is long, and Time is fleeting,
   And our hearts, though stout and brave,
Still, like muffled drums, are beating
   Funeral marches to the grave.

In the world’s broad field of battle,
   In the bivouac of Life,
Be not like dumb, driven cattle!
   Be a hero in the strife!

Trust no Future, howe’er pleasant!
   Let the dead Past bury its dead!
Act,— act in the living Present!
   Heart within, and God o’erhead!

Lives of great men all remind us
   We can make our lives sublime,
And, departing, leave behind us
   Footprints on the sands of time;

Footprints, that perhaps another,
   Sailing o’er life’s solemn main,
A forlorn and shipwrecked brother,
   Seeing, shall take heart again.

Let us, then, be up and doing,
   With a heart for any fate;
Still achieving, still pursuing,
   Learn to labor and to wait.

Henry Wadsworth Longfellow, 1838