perun icon indicating copy to clipboard operation
perun copied to clipboard

ExtSourceSql with pooling connections

Open martin-kuba opened this issue 5 years ago • 2 comments

Changed implementation of ExtSourceSql and ExtSourcesManagerImpl to use database connections obtained from pooling DataSources instead of keeping a special connection open for each one.

  • changed parsing of XML file perun-extSources.xml in ExtSourcesManagerImpl
    • replaced parsing library DOM with DOM4J, it is much easier to use
    • replaced project dependency on JDOM, which was not used, with dependency on DOM4J
    • parsing of perun-extSources.xml detects new tags <dbpool> and creates a read-only Hikari pool for each one (special mark <main/> is available for referencing the main pool instead of creating a new one)
    • extSources of type ExtSourceSql now support a new attribute dbpool referencing a db pool
    • for backward compatibility, extSources of type ExtSourceSql with legacy attributes url, driver, user and password are detected and dbpools are created for them
  • added method destroy() to PerunBlImpl, ExtSourcesManagerBl, ExtSourcesManagerBlImpl, ExtSourcesManagerImplApi and ExtSourcesManagerImpl which closes created DataSources on shutdown
  • changed implementation of methods querySource() and groupQuery() to be faster by pre-processing result set metadata outside of the reading loop and reading only available columns, thus avoiding throwing-and-catching exceptions just to detect missing columns on each row
  • rewrote ExtSourcesManagerBlImpl to cache instances of ExtSources in memory to avoid a lot of database queries
  • replaced injection of PerunBl instance into static variable in ExtSource* classes from perun-core.xml with all ExtSource descendants implementing a new class EtxSourceImpl that has a setter for PerunBl, and this is called right after creation of a new instance in EXTSOURCE_MAPPER
  • replaced usages of EXT_SOURCES_EXTRACTOR with EXTSOURCE_MAPPER in ExtSourcesManagerImpl and changed SQL queries to spare useless extracting and immediate throwing away of attribute values when reading just ExtSources
  • in AuthzResolverIntegrationTest added refreshing ExtSources before every test

martin-kuba avatar Apr 27 '20 14:04 martin-kuba

deleted branch by mistake

martin-kuba avatar May 06 '20 10:05 martin-kuba

Please rebase this PR. Thank you.

stavamichal avatar Jul 09 '20 11:07 stavamichal

done as #3895

martin-kuba avatar Jun 28 '23 13:06 martin-kuba