“The future masters of technology will have to be light-hearted and intelligent. The machine easily masters the grim and the dumb.'' — Marshall McLuhan
“Simplicity is a great virtue but it requires hard work to achieve it and education to appreciate it. And to make matters worse: complexity sells better.” — Edsger W. Dijkstra
Bryan Liles says it clearly as well:
Simple solutions are easier to implement and scale than complex solutions. However promotions often are rewarded to those who create complex solutions. We need to start acknowledging the power of simple. Don’t accept complex. It’s a trap.
“Programs must be written for people to read, and only incidentally for machines to execute.” — Harold “Hal” Abelson
This highfalutin aphorism applies from the most cerebral application programming language of your choice, right down to the humble (or not-so-humble) bash script. Love or hate them, bash scripts are the primary method of launching processes on Linux in many settings. (You may well run some kind of fancy-pants orchestration tool, but when that calls into your app it is 50/50 if it ends up calling a bash wrapper script or not.) I think of bash as a tool for writing things which set the execution context and run the main program, and no more than that.
All too often, one sees examples on the web of people saying “launch this program under docker via this bash one-liner”, and then they proceed to demonstrate with a command-line which goes on and on for ages and never seems to stop, very much like this sentence.
Sometimes, people are considerate enough to lay out their example over several lines, all but the last of which has a trailing back slash character “\” to indicate to bash that the command continues on the next line. Here’s a contrived example of what I mean.
Unfortunately, there are a few problems with this notation in the real world. (The above example is not the real world.)
It does not allow for individual lines to be commented out.
It does not allow for any characters after the \, which means that
it does not allow for explanatory comments to be added at the end of a line
invisible (depending on the text editor) space characters can creep in after the backslash and cause weird and annoying error messages
Because of the above, this syntax looks and feels brittle. In the real world (that thing again) one often wishes to comment and uncomment command line options when experimenting with a new configuration. Never fear, bash “array notation” to the rescue…
This solves all three of the original problems, but it looks a bit clumsy, what with the repeated mention of the variable name and brackets on every line. All I’m doing is enumerating the arguments to a command for goodness’ sake.
Happily, we can remove the += stuff and most of the brackets, make it a single array, and get straight to the point, viz.
Ta-da! Note that this only works with single commands; if you have multiple commands which need to be connected via pipes they will each need to be declared as individual arrays and called as one pipeline thus:
"${first_command[@]}" | "${second_command[@]}"
If you’ve ever had cause to tangle with the joy that is “setenv.sh” on the installation of an Atlassian application, I’m sure you’ll appreciate the clarity and usability of the above syntax. Yes, I know they have to run under other systems like FreeBSD or Solaris, which most likely don’t have bash. I’m talking about in-house stuff, where the toolset is well-known.
Update: I have recently come across this quote from someone, who, as usual, sums it up brilliantly:
“Any fool can write code that a computer can understand. Good programmers write code that humans can understand.”
With enable_msi set to 0 (the default), while happily playing sound for a random indeterminate period of time, the sound system would lose grip and leave me with nothing but buzzing in the headphones, and only a reboot would return things to normal. Sometimes, unloading and reloading the sound modules did the trick, but often things would be stuck waiting for the hardware to return to normal, which it had no intention of doing. Enabling Message Signalled Interrupts avoids this problem.
power-saving mode
Sometimes the sound will play for a fraction of a second, and then immediately change to static noise on the headphones. The fact that this happens at the start of playback, presumably just after the sound hardware has been brought out of power-saving mode, points towards disabling power-saving to be the way forwards. There are two parameters related to power, and they are both recommended, even though they will not be applicable to all hardware.
During the production and deployment of any application larger than “hello world”, careful thought must be given to accommodating a likely set of changes.
One of these is deployment location — the application must not have any hard-coded file system locations present, and must determine its location at runtime.
Every bash script should start with this little section.
and use that variable within and across the whole application (call it “this-app”) and pass it in as an application property, for example. If the script above lives at /deploy/some/where/this-app/bin/start-component everything will be relative to /deploy/some/where and if you need to refer to another application (call it “other-app”) installed at the same level, you can say
export OTHER_APP_HOME=${_PREFIX}/other-app/
Remember that it’s helpful to use a prefix such as “THIS_APP” for environment variables specific to each application.
This is great news. All JetBrains apps on Linux are now usable. Only on the EAP versions for now.
The right thing to do is to remove all the tweaks put in with JDK11 and before and use the vanilla settings. However, one still might want to bump the JVM max memory -Xmx parameter up from the relatively low default of 750MB to 2G. This can be done via the JetBrains toolbox.
I’m pleased that this has happened, but it’s been a painful time having to use JDK11 as the JBR on Linux thus far. Looking forward…
JetBrains’ IDEs suffer from awful interactive performance out of the box. This is unfortunate, and given the highly interactive nature of programming one would think it would be taken more seriously. This article gives instructions on how to change the “boot JDK” and compile the base classes of the JDK to binary form, so that they are not JIT-compiled every time one starts the IDE.
I suggest you try both operations for the maximum benefit. I am keen to hear about the results, so please tell me via firstname.lastname@gmail.com — check the domain name of this site for the values 🙂 In adjusting all of these settings, YMMV depending on your system. Good luck!
disclaimer / warning
This post contains advice / instructions on how to change your JetBrains installation. Following them may break your installation, and I do not take any responsibility for your actions in doing so 🙂
1 / stop using JDK 11-based JBR, start using a (vanilla) version 16 JDK instead
Install version 16 of the JDK such as Azul Zulu by following the instructions here. Other JVM implementations are available. I am neither paid by, nor associated with, Azul.
Change the boot JDK to the one you’ve just installed by following the instructions here.
All the EAP and latest release versions of JetBrains IDEs seem to work with JDK 16. In all cases, features using the JCEF [1] runtime library, which is bundled with the JBR [2], will not be available. If you run into problems, JDK 15 will work instead.
Change the “Custom VM options” to something like the below. I’m using these options on Debian/unstable on all JetBrains IDEs, with both JDK 15 and JDK 16.
Adding -XX:ReservedCodeCacheSize= and -XX:SoftRefLRUPolicyMSPerMB= made all the difference in keeping heap usage down, although I have not yet experimented with different values for those parameters.
The option –illegal-access=permit is required on JDK 16, but this option was removed in JDK 17, and because of this (but probably not only this) I haven’t yet been able to make any JetBrains IDE work with JDK 17.
When you edit this file, make a note of the location via the “Copy path” feature in the IDE, so that if it breaks, you can find it and remove the mistake quickly. Also, you may wish to start the IDE in the console via the command line (rather than the Toolbox launcher) so that you can easily see the error.
2 / compile all the JDK classes ahead of time using the jaotc tool
Since day one, startup time has been a painful part of Java. Nowadays we are blessed with modules in the JDK and we can compile them ahead of time and load them into the JDK at runtime. Seems like it might a good idea for big applications, but actually the benefits are greater for smaller programs.
This script invokes the jaotc binary to create a single shared object library file from the modules in the JDK. Note that the ZGC garbage collector is not compatible with the use of jaotc.
The jaotc tool has been removed in version 17 and later of the JDK, but for now, we can make the most of it.
#! /bin/bash
set -o nounset
# java ------------------------------------------------------------------------
java_home=${JAVA_HOME:-/usr/lib/jvm/zulu16}
echo "java_home is [${java_home}]"
java_flags=( -XX:+UseG1GC -XX:-UseZGC )
java_name=$(basename ${java_home})
aot_so=${HOME}/lib/java/native/$(uname -m)-linux/aot.${java_name}.so
echo "output shared object file is [${aot_so}]"
# enumerate modules -----------------------------------------------------------
modules=()
jmod_files=( ${java_home}/jmods/*.jmod )
for _jmod in ${jmod_files[@]}; do
module=$(basename ${_jmod/.jmod/})
case ${module} in
jdk.hotspot.agent) ;;
jdk.incubator) ;;
jdk.incubator.*) ;;
jdk.internal.vm.compiler) ;;
jdk.jcmd) ;;
*) modules+=( ${module} ) ;;
esac
done
# construct command line ------------------------------------------------------
argv=()
argv+=( ${java_home}/bin/jaotc )
for flag in ${java_flags[@]}; do
argv+=( -J$flag )
done
argv+=( --compile-for-tiered )
argv+=( --output ${aot_so} )
argv+=( --module ${modules[@]} )
# execute --------------------------------------------------------------------
mkdir -p $(dirname ${aot_so})
${argv[@]}
This step above will take about ten minutes or so, depending on your system.
Add these lines to your IntelliJ custom VM options. The path matches the output path in the script above, and the “GC” options match the JVM options in the script as well. (As mentioned above, the ZGC garbage collector is not compatible with the use of jaotc.) Note that this is different from the previous section where ZGC was enabled.
(Note the /home/username and change it to your home directory.) Restart the IDE — and you should see it start much faster. Remove or comment out -XX:+PrintAOT when you are satisfied it’s working as expected.
You should find that the audio behaves sanely from now on.
My setup / usage is that I have my 3.5mm dual-jack headphones plugged into the headphone and microphone sockets on the front, and my external amplifier and speakers plugged into the line out socket on the rear.
The audio comes out of both front and rear sockets, the front microphone works, and plugging something into the front headphone socket mutes the output on the rear headphone socket.
Normally I have my headset plugged into the headphone and microphone socket on the front for conference calls. If I want to watch a video with proper sound, I partially unplug the front headphone jack (leaving it teetering in the socket) and the rear socket wakes up and sound comes out of the speakers through the external amplifier.
There is a problem here though — after a reboot, with headphones plugged in the front and amplifier plugged into the rear, no sound comes out of neither socket until I pull and replace the jack from the front headphone socket. If you are reading this and know of a way around this, so that sound comes out of both outputs on boot without manual intervention, please let me know. 🙂
abstract There have been thousands of articles written about Python virtual environments. As of 2020-10-07, a quick search on Google for “python virtualenv article” returns “about 1,450,000 results”. In the majority of cases I have seen, they are partially filled with impractical advice. They gush about pip, pipx, venv, virtualenvwrapper, pipenv, poetry and others, but they don’t explain how to achieve an ergonomic execution of a Python program once the virtual environment is in place, and how to make this consistent in all environments, from development to production. To be clear, too many of these articles gloss over an ugly reality of their suggested approach — that the command line used to run project components will be different between environments, thus turning something which should be trivial into something diabolical with several degrees of freedom and opportunites for confusion.
This article (and the template project linked above) attempts to correct that by explaining how to achieve a smooth as possible a deployment experience across environments (from local development, to remote production). This is at once both pretty basic and very important, but I have found that “doing the basic stuff really well” has proved to be a profitable approach.
preliminaries / terminology
It’s important to be clear about what it is we are discussing.
versions When “Python” is mentioned here, version 3.6 or later of CPython on a Debian-based Linux installation is assumed. Win32 and Python 2, fine platforms that they are, do not come under consideration, except perhaps by way of comparison to illustrate a concept or example.
venv The "venv" module does the same job as the “virtualenv” module from Python 2. It creates and populates the .venv/ directory.
pip The “pip” module is the tool which installs modules into a Python installation so that they may be used via “import” In this case they are being installed into .venv/lib/.../site-packages/ for use by the project. By the convention described here, the packages and their versions are enumerated in the file etc/pip/requirement.txt (singular)
bash We use bash for a wrapper script to make the .venv/ environment easily accessible. [1]
structure Source code arrangement is a topic all-too-often overlooked. I suggest that the filesystem-level layout of your project is as just as important as any other design decision or ergonomic consideration. Think of it like a sign saying “Please keep the workshop and van clean and tidy.” It’s a good idea, even if we don’t manage it all the time, but so much else is easier and quicker if we do.
polyglot Python is not the only game in town. It is common for large projects to use more than one language. These languages and their associated runtime systems will have different conventions on how things work. To reduce friction in daily navigation, all languages, their libraries and tooling, must live in predictable, obvious locations in the project tree. A little consideration goes a long way.
Things that this solution does not do.
Multiple python versions The solution presented here is a minimal-but-complete solution for the common case. It does not attempt to to rival or compete with any of the capabilities of virtualenvwrapper, pipenv, poetry and others. I understand that library maintainers may wish to run their unit tests on multiple versions of the Python runtime, and this is a laudable goal.
Multiple transitive library dependencies If your project requires both library A and B, and A requires library C version 4, and B requires library C version 5 we have a problem which no virtualenv management tool can reasonably resolve unless it starts adjusting module locations at installation time, and rewriting imports of library C within libraries A and B. While this may be conceivable, it is not a robust approach, and on balance should be avoided.
installation
bin/venv-create This invokes the venv module to create the .venv/ directory, and pip to populate the latter with the modules listed in etc/pip/requirement.txt
execution
There are two wrapper scripts for convenience / practicality. (A PhD student working as a TA once asked me what a wrapper script was. I mean, had he no imagination?) The main purpose of wrapper scripts is to ensure the sanity of the execution environment, so to call them “convenience scripts” is to understate their necessity.
bin/venv-python This sets the following environment variables
PYTHONPATH to include src/python/main/
PYTHONDONTWRITEBYTECODE to zero
PYTHONHASHSEED to zero
PYTHONUNBUFFERED to zero
and is also the right place to set LC_* and anything pertinent to modules.
and then calls .venv/bin/python, passing along any arguments. Using bin/venv-python every time ensures the same execution environment for all python code.
If invoked via a symlink, it invokes .venv/bin/python3 with the “-m” option, passing along any arguments, the first one being the module name, which it gets from the source name of the symlink. This allows us to create something the symlink
bin/jupyterlab -> venv-python
and running bin/jupyterlab does the right thing, with the expected environment, without typing a long path and all is well.
bin/run-main Purely as a matter of convention, this runs src/main/python/main.py via bin/venv-python — you will likely want to add further “run-*” scripts as copies of this to suit your requirements.
That is pretty much it — some simple execution machinery underneath to create a predictable interface above, allowing the user to run the system in the same fashion in all environments. In the end, isn’t that what it’s all about?
complaints department
Thank you for your time and attention. If you have anything to say (either good or bad) about what I have written above, I would be very pleased to hear from you. Alternatively, if you have nothing to say but you like it anyway, then please tell a friend.
The solution presented here avoids much of this.
[1] The GNU bash shell is to be tamed, not avoided. If you think that knowing this is beneath you, then you need to think again. A serious chef keeps sharp knives.
