Improve WFS-T interface to support mass insert or updates to split into chunks

[tfr42]

Currently a WFS-T Insert operation requires that a <FeatureCollection/> is self contained and all inner references between features are resolvable. In case of processing big (> 1GB) WFS-T Insert requests the JVM may run into OoM error (depending on the max heap size settings of the JVM). Support for mass data inserts is required.

[tfr42]

Workarounds and known limits for import of big GML files:

• When using the WFS-T INSERT action the following points shall be considered:
  • The FeatureCollection must be self-contained and all inner references between the features must be resolvable, (see #749 for more information how to use the enhanced reference resolver)
  • the rule of thumb for the configuration of the JVM is that you need to set the heap size at least five times bigger than the size of the GML file (size on disk)
  • you need to configure a FeatureStore based on the GML application schema and a WFS with Transactions enabled as described in Mapping GML application schemas
• When using the FeatureStore Loader the same hints as for the WFS-T INSERT applies
  • with the important difference that the introduced function with 3.4-RC5 to disable the resolution of code list values (PR #787, #837) is not effective for the FeatureStore Loader!
• When you want to build your own client using deegrees FeatureStore API you have to provide
  • a deegree workspace with a valid SQLFeatureStore configuration
  • you can use the deegree CLI tool to generate the SQL DDL scripts for the underlying database (PostgreSQL/PostGIS and Oracle DBMS are supported) and the SQLFeatureStore configuration files (BLOB and relational mapping is supported)
• Writing data directly into the database you have to create the database schema
  • you can use the deegree CLI tool to generate the SQL DDL scripts for the underlying database (PostgreSQL/PostGIS and Oracle DBMS are supported)
  • for this approach the relational mapping is recommended, but using BLOB modus works as long as the tool you are using is able to insert GML objects into the database.

[tfr42]

Instead of enhancing the WFS-T interface we now favour to extend the deegree-cli-utility with improved capabilities of the FeatureStore Loader.

[tfr42]

Current findings and recommendations when importing large GML files (>500 MB):

• use deegree 3.4-RC5 or higher
• enable WFS-T with <DisabledResources> for code list values (see PR #787)
• use Oracle JDK 8 and configure your JVM with -server -Xms<N>g -Xmx<N>g -Xmn<M>g -XX:MaxMetaspaceSize=256m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
  • where N is about factor 8 of the size of the GML file (size on disk)
  • where M is about factor 0.4 of N
  • for example you want to import GML file with a size of 1.5 GB you want to set the JVM options for the heap size to -Xms12g -Xmx12g -Xmn4g
• and ensure that the server provides at least 4 CPU/cores (required for parallel GC) and 16 GB RAM (or for larger files than 1.5 GB provide N+2GB RAM).