When Java AWT/Swing got JavaFX-style automatic high DPI scaling in Java SE 9, there were also several new system properties to control that scaling. Unfortunately, as with the various glass.win.* properties of JavaFX (see e.g. here) they are not officially documented. You need to check the developer mailing list and the JDK test sources to find out what they are and how they work.
That’s what I did and here is what I found. First, there is a switch disable (or re-enable) high DPI scaling entirely, for example in command-line format with the -D switch:
-Dsun.java2d.uiScale.enabled=false
Two more settings control the exact scaling factor for each dimension. Three formats are supported: the factor itself, the factor as a percentage value, and the actual DPI resolution where 96 DPI equals a factor of 1 or 100%. Here are equivalent examples for 120 DPI:
-Dsun.java2d.win.uiScaleX=1.25 -Dsun.java2d.win.uiScaleY=1.25
-Dsun.java2d.win.uiScaleX=125% -Dsun.java2d.win.uiScaleY=125%
-Dsun.java2d.win.uiScaleX=120dpi -Dsun.java2d.win.uiScaleY=120dpi
You can indeed specify different scaling factors for each dimension, resulting in a funhouse mirror effect. I’m not sure if there is any practical use for that, though!
I saw another property called sun.java2d.win.uiScale that may once have set both dimensions, but in my testing this property does nothing now. I ran all tests on Java SE 10.0.2 and the current Windows 10 update, so the situation on Linux may be different.