Broken Font support (Fontconfig) in AWS-Lambda (java11)

[ggrandes]

Describe the bug

Broken Font support in Corretto-11 on AWS-Lambda. This bug is similar to [#95] and [corretto/corretto-8/issues/64]. With no known workaround.

To Reproduce

In AWS-Lambda console

1. Create function, author from scratch: Name=FontSupport, Runtime=Java 11
2. In Function code (actions): Upload .zip file: fontsupport-1.0.0-shaded.zip
3. Basic Setting: Handler=mytest.FontSupport::handleRequest
4. Configure new test event, only "test" as content and run Test button.

This is the source code for reference:

package mytest;

import java.awt.Font;
import java.awt.FontMetrics;
import java.awt.Graphics;
import java.awt.image.BufferedImage;

import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.RequestHandler;

/*
		<dependency>
			<groupId>com.amazonaws</groupId>
			<artifactId>aws-lambda-java-core</artifactId>
			<version>1.2.1</version>
		</dependency>
		<dependency>
			<groupId>com.amazonaws</groupId>
			<artifactId>aws-lambda-java-events</artifactId>
			<version>3.1.0</version>
		</dependency>
 */
public class FontSupport implements RequestHandler<String, String> {
	@Override
	public String handleRequest(final String input, final Context context) {
		try {
			return "OK:" + test();
		} catch (Throwable e) {
			e.printStackTrace();
			return String.valueOf(e) + "\n" + ("SystemProperties: " + System.getProperties());
		}
	}

	private int test() {
		final BufferedImage canvas = new BufferedImage(5, 5, BufferedImage.TYPE_INT_RGB);
		final Graphics graphics = canvas.getGraphics();
		final Font font = new Font("Verdana", Font.PLAIN, 110);
		final FontMetrics metrics = graphics.getFontMetrics(font); // Crash!!
		final int maxAdvance = metrics.getMaxAdvance();
		System.out.println("MaxAdvance: " + metrics.getMaxAdvance());
		return maxAdvance;
	}
}

Expected behavior

• Result in lambda console: "OK:number"
• Log output: MaxAdvance: number, without any java.lang.Exceptions

StackTrace

START RequestId: 511c3a60-4046-4fac-9046-29ce8cdda2b7 Version: $LATEST
java.lang.InternalError: java.lang.reflect.InvocationTargetException
	at java.desktop/sun.font.FontManagerFactory$1.run(Unknown Source)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.desktop/sun.font.FontManagerFactory.getInstance(Unknown Source)
	at java.desktop/sun.font.SunFontManager.getInstance(Unknown Source)
	at java.desktop/sun.font.FontDesignMetrics.getMetrics(Unknown Source)
	at java.desktop/sun.java2d.SunGraphics2D.getFontMetrics(Unknown Source)
	at mytest.FontSupport.test(FontSupport.java:38)
	at mytest.FontSupport.handleRequest(FontSupport.java:27)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
	at lambdainternal.EventHandlerLoader$PojoMethodRequestHandler.handleRequest(EventHandlerLoader.java:280)
	at lambdainternal.EventHandlerLoader$PojoHandlerAsStreamHandler.handleRequest(EventHandlerLoader.java:197)
	at lambdainternal.EventHandlerLoader$2.call(EventHandlerLoader.java:897)
	at lambdainternal.AWSLambda.startRuntime(AWSLambda.java:228)
	at lambdainternal.AWSLambda.startRuntime(AWSLambda.java:162)
	at lambdainternal.AWSLambda.main(AWSLambda.java:157)
Caused by: java.lang.reflect.InvocationTargetException
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
	at java.base/java.lang.reflect.Constructor.newInstance(Unknown Source)
	... 18 more
Caused by: java.lang.NullPointerException
	at java.desktop/sun.awt.FontConfiguration.getVersion(Unknown Source)
	at java.desktop/sun.awt.FontConfiguration.readFontConfigFile(Unknown Source)
	at java.desktop/sun.awt.FontConfiguration.init(Unknown Source)
	at java.desktop/sun.awt.X11FontManager.createFontConfiguration(Unknown Source)
	at java.desktop/sun.font.SunFontManager$2.run(Unknown Source)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.desktop/sun.font.SunFontManager.<init>(Unknown Source)
	at java.desktop/sun.awt.FcFontManager.<init>(Unknown Source)
	at java.desktop/sun.awt.X11FontManager.<init>(Unknown Source)
	... 22 more
END RequestId: 511c3a60-4046-4fac-9046-29ce8cdda2b7

Platform information

{
    "awt.toolkit": "sun.awt.X11.XToolkit",
    "java.specification.version": "11",
    "java.class.path": "/var/runtime/lib/aws-lambda-java-core-1.2.0.jar:/var/runtime/lib/aws-lambda-java-runtime-0.2.0.jar:/var/runtime/lib/aws-lambda-java-serialization-0.2.0.jar",
    "java.vm.vendor": "Amazon.com Inc.",
    "java.vendor.url": "https://aws.amazon.com/corretto/",
    "os.name": "Linux",
    "sun.boot.library.path": "/var/lang/lib",
    "sun.java.command": "lambdainternal.AWSLambda",
    "java.version.date": "2020-04-14",
    "java.home": "/var/lang",
    "java.vm.compressedOopsMode": "32-bit",
    "java.specification.name": "Java Platform API Specification",
    "java.vm.specification.vendor": "Oracle Corporation",
    "java.awt.graphicsenv": "sun.awt.X11GraphicsEnvironment",
    "sun.management.compiler": "HotSpot 64-Bit Tiered Compilers",
    "java.runtime.version": "11.0.7+10-LTS",
    "os.version": "4.14.165-102.205.amzn2.x86_64",
    "java.runtime.name": "OpenJDK Runtime Environment",
    "java.vm.name": "OpenJDK 64-Bit Server VM",
    "java.vendor.version": "Corretto-11.0.7.10.1",
    "java.vendor.url.bug": "https://github.com/corretto/corretto-11/issues/",
    "java.version": "11.0.7",
    "user.dir": "/var/task",
    "os.arch": "amd64",
    "java.vm.specification.name": "Java Virtual Machine Specification",
    "sun.os.patch.level": "unknown",
    "java.library.path": "/var/lang/lib:/lib64:/usr/lib64:/var/runtime:/var/runtime/lib:/var/task:/var/task/lib:/opt/lib:/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib",
    "java.vendor": "Amazon.com Inc.",
    "java.vm.info": "mixed mode, sharing",
    "java.vm.version": "11.0.7+10-LTS",
    "sun.io.unicode.encoding": "UnicodeLittle",
}

Additional context

• This code was tested on AWS-Lambda (2020-07-05), using runtime: Java 8 (run without problem) and Java 11 (fail).
• This workaround don't works in this case.

[cliveverghese]

Hi,

Thank you for contacting us regarding this issue, we are in touch with the AWS Lambda Team.

Regards, Clive Verghese

[mhoennig]

Any chance that there will be a general solution for this issue or do we need ugly workarounds like a Lambda layer?

It would be great to know if it's better to wait a few weeks with an upgrade to Coretto JDK11 or spend time on a workaround.

Best regards ... Michael

[mhoennig]

We found an easy workaround for simple Excel-Sheets, avoiding methods which use Fonts, e.g. using setColumnWidth instead of autoSizeColumn. But this certainly does not help in all cases.

[wilerson]

Any update about this (and related) issue?

[deandelponte]

Even a work around would be much appreciated

[ggrandes]

@deandelponte

Even a work around would be much appreciated

My (very specific) workaround: here. It has basically been precalculating on desktop and then exporting it for use in lambda; it has many limitations.

[jhuamanchumo]

Steps from here worked like a charm.

[ghost]

Issue reported a year ago, did the dev team stop looking into it? I need to get the font metrics for a font and the workarounds identified above are not going to work for us.

[ggrandes]

At this day, font-metrics still don't work in corretto-11 on AWS.

[sema4-daniilvolkov]

The same issue

   // Raleway
   SimpleFontFamily raleway = new SimpleFontFamily(context);
   raleway.setName("Raleway");
   raleway.setPdfEmbedded(true);
   raleway.setPdfEncoding("Identity-H");

   SimpleFontFace ralewayRegular = new SimpleFontFace(context);
   ralewayRegular.setTtf("fonts/Raleway/Raleway-Regular.ttf");
   raleway.setNormalFace(ralewayRegular);

the ttf is present in the resources folder

and got the next exception message:

Caused by: net.sf.jasperreports.engine.fonts.InvalidFontException: Error loading font "fonts/Raleway/Raleway-Regular.ttf".
	at net.sf.jasperreports.engine.fonts.SimpleFontFace.loadFont(SimpleFontFace.java:206)
	at net.sf.jasperreports.engine.fonts.SimpleFontFace.setTtf(SimpleFontFace.java:167)
	at net.sf.jasperreports.engine.fonts.SimpleFontFace.setTtf(SimpleFontFace.java:157)

this issue is related to Java8 Corretto and Java11 Corretto

with Java8 (OpenJDK) - works without any problems Due to https://aws.amazon.com/blogs/compute/announcing-migration-of-the-java-8-runtime-in-aws-lambda-to-amazon-corretto/ we have to make it work on Corretto.

[jhuamanchumo]

Steps from here worked like a charm.

This approach does not work with jasper reports, with this function I had to change to java 8 to make it work without any problems (stracktrace below).

Probable fatal error: No physical fonts found.: java.lang.Error
java.lang.Error: Probable fatal error: No physical fonts found.
	at java.desktop/sun.font.SunFontManager.lambda$getDefaultPhysicalFont$0(Unknown Source)
	at java.base/java.util.Optional.orElseThrow(Unknown Source)
	at java.desktop/sun.font.SunFontManager.getDefaultPhysicalFont(Unknown Source)
	at java.desktop/sun.font.CompositeFont.doDeferredInitialisation(Unknown Source)
	at java.desktop/sun.font.CompositeFont.getSlotFont(Unknown Source)
	at java.desktop/sun.font.CompositeStrike.getStrikeForSlot(Unknown Source)
	at java.desktop/sun.font.CompositeStrike.getFontMetrics(Unknown Source)
	at java.desktop/sun.font.Font2D.getFontMetrics(Unknown Source)
	at java.desktop/java.awt.Font.defaultLineMetrics(Unknown Source)
	at java.desktop/java.awt.Font.getLineMetrics(Unknown Source)
	at net.sf.jasperreports.engine.fill.SimpleTextLineWrapper.determineLeading(SimpleTextLineWrapper.java:391)
	at net.sf.jasperreports.engine.fill.SimpleTextLineWrapper.getGeneralFontInfo(SimpleTextLineWrapper.java:343)
	at net.sf.jasperreports.engine.fill.SimpleTextLineWrapper.createFontInfo(SimpleTextLineWrapper.java:279)
	at net.sf.jasperreports.engine.fill.SimpleTextLineWrapper.start(SimpleTextLineWrapper.java:241)
	at net.sf.jasperreports.engine.fill.TextMeasurer.measure(TextMeasurer.java:533)
	at net.sf.jasperreports.engine.fill.JRFillTextElement.chopTextElement(JRFillTextElement.java:671)
	at net.sf.jasperreports.engine.fill.JRFillTextField.prepare(JRFillTextField.java:777)
	at net.sf.jasperreports.engine.fill.JRFillElementContainer.prepareElements(JRFillElementContainer.java:536)
	at net.sf.jasperreports.engine.fill.JRFillBand.fill(JRFillBand.java:411)
	at net.sf.jasperreports.engine.fill.JRFillBand.fill(JRFillBand.java:386)
	at net.sf.jasperreports.engine.fill.JRVerticalFiller.fillTitle(JRVerticalFiller.java:313)
	at net.sf.jasperreports.engine.fill.JRVerticalFiller.fillReportStart(JRVerticalFiller.java:247)
	at net.sf.jasperreports.engine.fill.JRVerticalFiller.fillReport(JRVerticalFiller.java:115)
	at net.sf.jasperreports.engine.fill.JRBaseFiller.fill(JRBaseFiller.java:580)
	at net.sf.jasperreports.engine.fill.BaseReportFiller.fill(BaseReportFiller.java:414)
	at net.sf.jasperreports.engine.fill.JRFiller.fill(JRFiller.java:121)
	at net.sf.jasperreports.engine.JasperRunManager.runToPdfFile(JasperRunManager.java:389)
	at net.sf.jasperreports.engine.JasperRunManager.runReportToPdfFile(JasperRunManager.java:829)
	at com.xxx.yyy.zzz.util.JasperUtil.generate(JasperUtil.java:79)
	at com.xxx.yyy.handler.App.handleRequest(App.java:94)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.base/java.lang.reflect.Method.invoke(Unknown Source)

[stuartleylandcole]

I have also encountered this error. My use case was using Aspose to convert Office documents to images but that's neither here nor there. I tried manually creating a Lambda layer with the fontconfig package but did not succeed until I found yumda from lamb-ci which made it much easier. In case it helps anyone else this is the Makefile I wrote to build a Lambda layer:

# Uses https://github.com/lambci/yumda to install Linux packages using YUM and create a Lambda layer

PROJECT_ROOT = $(dir $(abspath $(lastword $(MAKEFILE_LIST))))

DOCKER_IMAGE ?= lambci/yumda:2

TARGET ?=/lambda/opt
BUILD_DIR = build
MOUNTS = -v $(PROJECT_ROOT)$(BUILD_DIR):$(TARGET)

DOCKER = docker run --rm

YUM_INSTALL_COMMAND = yum install -y
PACKAGE = fontconfig

RESULT_DIR = result

clean:
	rm -rf $(BUILD_DIR) $(RESULT_DIR)

build:
	$(DOCKER) $(MOUNTS) $(DOCKER_IMAGE) $(YUM_INSTALL_COMMAND) $(PACKAGE)

create_layer:
	mkdir -p $(PROJECT_ROOT)$(RESULT_DIR)
	cd $(BUILD_DIR) && zip -yr $(PROJECT_ROOT)$(RESULT_DIR)/package.zip .

all: clean build create_layer

Run make all to build result/package.zip.

Hope this is useful.

Edit: There was a mistake in my create_layer rule (the cd and zip operations were executed separately and so the zip file contained everything in the folder).

[mkavalecz]

This issue applies both to Corretto 8 and 11, and it became quite urgent, because AWS is migrating lambda to Corretto (see https://aws.amazon.com/blogs/compute/announcing-migration-of-the-java-8-runtime-in-aws-lambda-to-amazon-corretto/).

All it needs to break is just one line of code: java.awt.GraphicsEnvironment.getLocalGraphicsEnvironment().getAvailableFontFamilyNames()

It throws: java.lang.UnsatisfiedLinkError: /var/lang/lib/amd64/libfontmanager.so: libfreetype.so.6: cannot open shared object file: No such file or directory

This will break our application in less than 2 weeks, and we'd very much prefer not having to use a custom runtime because of a bug like this. Any news about an actual fix? @cliveverghese

[davecurrie]

Hi. We are working with the Lambda team to ensure the automated migration won't trigger this bug. I can't talk about what exactly it will look like yet but we do have folks working on it.

[smirnoal]

This will break our application in less than 2 weeks, and we'd very much prefer not having to use a custom runtime because of a bug like this. Any news about an actual fix? @cliveverghese

Hi Máté, could you please try adding a layer arn:aws:lambda:::awslayer:Java8Corretto to your function, as it is described in the "How can I test for compatibility with Amazon Corretto 8?" chapter of the aforementioned communication and post error details here if the issue still persists?

[mkavalecz]

Hi Máté, could you please try adding a layer arn:aws:lambda:::awslayer:Java8Corretto to your function, as it is described in the "How can I test for compatibility with Amazon Corretto 8?" chapter of the aforementioned communication and post error details here if the issue still persists?

When I use "Java 8 on Amazon Linux 1" and add the layer you mentioned, I don't get an exception, it works properly. Does this mean, that there's a difference between the current "Java 8 on Amazon Linux 2" runtime, and this layer, so we should not worry about this, because when it goes live, it will have the updated version? That would be a relief.

Thanks for the help.

[smirnoal]

thanks for confirming, Máté. Given that your functions are working fine with the layer, you will not be affected with the migration OpenJDK => Corretto.

@sema4-daniilvolkov, could you please try the layer solution to see if you it works for your case?

The original issue still remans for "Java11 (Corretto)" and "Java 8 on Amazon Linux 2" runtimes, we are working on it!

[smirnoal]

while we are looking for a permanent solution, as a bandaid for the original problem with "Java11 (Corretto)" and "Java 8 on Amazon Linux 2" runtimes you can use this fontconfig layer (attached to this message)

fontconfig.zip

[mkavalecz]

@smirnoal Just as a confirmation, I've also tested with the al2 + fontconfig.zip layer, and it works fine, but the list of supported fonts is much smaller.

[DejaVu Sans, DejaVu Sans Condensed, DejaVu Sans Light, Dialog, DialogInput, Monospaced, SansSerif, Serif]

vs

[Bitstream Charter, Century Schoolbook L, Courier 10 Pitch, Cursor, DejaVu Sans, DejaVu Sans Condensed, DejaVu Sans Light, DejaVu Serif, DejaVu Serif Condensed, Dialog, DialogInput, Dingbats, Hershey, Monospaced, Nimbus Mono L, Nimbus Roman No9 L, Nimbus Sans L, SansSerif, Serif, Standard Symbols L, URW Bookman L, URW Chancery L, URW Gothic L, URW Palladio L, Utopia]

[sema4-daniilvolkov]

i tried to add the layer following your reccomendations:

functions:
  report-generation:
    handler: handler.ReportGenerationHandler::handleRequest
    layers:
      - arn:aws:lambda:::awslayer:Java8Corretto

and got the next error:

An error occurred: ReportDashgenerationLambdaFunction - Resource handler returned message: 

"arn:aws:lambda:::awslayer:Java8Corretto is not a valid execution environment 

the common error i got when tried to add the layer using AWS Lambda UI Console

[smirnoal]

@sema4-daniilvolkov could you please confirm you are using java8 runtime, not java8.al2? if you wish to use java8.al2, then fontconfig.zip layer could be used

[sema4-daniilvolkov]

After adding a fontconfig.zip as a custom layer it start working. Thanks!

[raffi-minassian]

We are seeing this problem as well as we prepare for the switch to Corretto and I wanted to confirm also that using fontconfig.zip layer with Java8 on Amazon Linux 2 runtime solves the /var/lang/lib/amd64/libfontmanager.so: libfreetype.so.6: cannot open shared object file: No such file or directory error. Thanks @smirnoal

[leviwilson]

while we are looking for a permanent solution, as a bandaid for the original problem with "Java11 (Corretto)" and "Java 8 on Amazon Linux 2" runtimes you can use this fontconfig layer (attached to this message)

fontconfig.zip

Do you have a Dockerfile or something or a script used to produce this? I added this layer and while it gets my past where it blows up, when converting documents that rely on fonts (like a pptx :arrow_right: pdf) it does not have the same fonts as Amazon Linux 1 had, so while it produces a PDF, the fonts aren't correct.

[leviwilson]

@smirnoal that fontconfig.zip layer gets past the original issue (where the layer blows up when trying to use fonts) but it still does not seem to be a 1:1.

We have a lambda (that uses Aspose as well to make PDF documents) which works fine in java8. In java8.al2 it blows up because of the missing fontconfig layer.

When using the provided fontconfig.zip as a layer, it doesn't blow up...but it also fails on some fonts (they just don't show up) when generating the PDF.

If rather than using this layer I yum install -y fontconfig within amazon/aws-lambda-java:8.al2, when I generate my PDF it's fine like it was on java8.

I would gladly build my own fontconfig layer if I knew what was installed and where when I yum install -y fontconfig, but all of my efforts are still producing the missing fonts.

[ghost]

Is there an update on this or some sort of ETA. In November, non-Corretto Java 8 will be disabled. Many of our Lambdas generate PDFs from various sources and this issue is costing us a LOT of time. The font layer workaround does not provide the fonts we need. We really don't want to move backward to Java 8 when Java 17 will be released this month.

[leviwilson]

@ens121 it's very easy to convert your lambda to a containerized version. That's what we ended up doing; making a Dockerfile from the same base image and installing fontconfig.

[ghost]

@leviwilson So the answer is that we need to introduce a new technology that we currently don't use (Docker), which includes training/education on how to use Docker and using Docker in AWS. If Corretto isn't a drop-in replacement for OpenJDK, then we shouldn't be forced to use it.

[leviwilson]

@ens121 I don't know, I don't work for Amazon. That's a workaround that will work for you that gives you complete control of the runtime for things like fonts. Up to you if you want to use it.

[raffi-minassian]

@ens121 we also run a Java Lambda to generate PDFs but thankfully the library we use is allowing for all needed fonts to be packaged and added to the classpath in a .jar file and that seems to be working. Sounds like that is saving us from the problem you are seeing.

I'm not sure what library or framework you are using to generate your PDFs but it might be worth a look if a solution like that can work for you ... in other words, a way to provide the needed fonts outside of having them installed on the system.

Just a thought. Hope it can help.