magento-lts icon indicating copy to clipboard operation
magento-lts copied to clipboard

Change initStatements to charset option when create db connection.

Open forfin opened this issue 1 year ago • 7 comments

  • This option pass to Zend_Db_Adapter_Pdo_Abstract (parent of Mage_Core_Model_Resource_Type_Db_Pdo_Mysql) when init new PDO
  • SET NAMES utf8 might be there as a workaround for php < 5.3.6 but should not be a case anymore.
  • Ref
    • https://stackoverflow.com/questions/4361459/php-pdo-charset-set-names
    • https://stackoverflow.com/questions/1650591/whether-to-use-set-names

*When Openmage start. It seems to merge every etc/*.xml config including this file. So before this commit, end-user who want to remove initStatements must explicitly set empty to <initStatements /> in later file. Otherwise it still be there.

Contribution checklist (*)

  • [x] Pull request has a meaningful description of its purpose
  • [x] All commits are accompanied by meaningful commit messages
  • [x] All automated tests passed successfully (all builds are green)
  • [x] Add yourself to contributors list

forfin avatar Aug 28 '24 21:08 forfin

Have you put this in production? What are the effects/benefits with or without this PR?

kiatng avatar Aug 29 '24 03:08 kiatng

Yes, I have used it in production without breaking anythings.

I found this because I was thinking that SET NAMES utf8 statement cause connection pinning in my AWS RDS Proxy env. but seems it not the case.

So, this commit did not breaking anythings and did not solve my original problem either. But 1 lesser query on every connections might be a beneficial for overall.

forfin avatar Aug 29 '24 09:08 forfin

Ref https://dev.mysql.com/doc/refman/8.4/en/charset-connection.html

SET NAMES 'charset_name' [COLLATE 'collation_name']

SET NAMES indicates what character set the client uses to send SQL statements to the server. Thus, SET NAMES 'cp1251' tells the server, “future incoming messages from this client are in character set cp1251.” It also specifies the character set that the server should use for sending results back to the client. (For example, it indicates what character set to use for column values if you use a SELECT statement that produces a result set.)

I also read the referenced stackoverflow. I am not sure how it related to our use case. UTF8 has been default in OpenMage since the beginning and we don't have any issues mentioned in the stackoverflow such as threat and wrong encoding in client.

I propose to close this PR unless there is a more compelling reason.

kiatng avatar Aug 30 '24 08:08 kiatng

Sorry that I give bad reference.

It start when Openmage create new PDO in Zend_Db_Adapter_Pdo_Abstract

  • When init new PDO($dsn, ...); $dsn string are produce by config xml files.
  • Before this commit $dsn was
$dsn = "$mysql:model=mysql4;initStatements=SET NAMES utf8;type=pdo_mysql;host=mysql;dbname=openmage;active=1";
  • After this commit $dsn will be
$dsn = "$mysql:model=mysql4;charset=utf8;type=pdo_mysql;host=mysql;dbname=openmage;active=1";
  • PDO dsn valid options are host, port, dbname, unix_socket, charset for MySQL
  • PDO dsn ignore non-existing option so initStatements is ignore.
  • charset=xxx option is equivalent to SET NAMES xxx .
    • https://www.php.net/manual/en/ref.pdo-mysql.connection.php
    • SET NAMES utf8 might be there as a workaround for php < 5.3.6 but should not be a case anymore.

So when no initStatements was given, 1 extra query is not call. https://github.com/OpenMage/magento-lts/blob/611bbe7ba8b1087452d18e423a93d0eb0ae52f02/app/code/core/Mage/Core/Model/Resource/Type/Db/Pdo/Mysql.php#L33-L35

forfin avatar Aug 30 '24 16:08 forfin

Seems OK to me with my tests. Do you see any problems with this change?

Axepih avatar Dec 21 '24 21:12 Axepih

I'm leaning towards merging this for a few reasons:

  • Cleaner implementation via DSN construction than a distinct command each connection
  • While minimal latency savings, it does save an additional command to the database
  • I believe we're under the assumption OpenMage is MySQL/MariaDB only at this point (correct me if I'm wrong), but <charset> is more DB-agnostic than a specific command in <initStatements>

I haven't tested this yet, but will approve once confirmed unless there are other objections.

mattdavenport avatar Mar 02 '25 02:03 mattdavenport

I have 2 issues with this PR:

  1. app/etc/local.xml often has <initStatements>SET NAMES utf8</initStatements>, which will override config.xml and add the initStatements anyway, making this PR ineffective.

  2. app/etc/local.xml.template contains

                    <initStatements>{{db_init_statemants}}</initStatements>

How do we deal with these 2 issues?

kiatng avatar Oct 02 '25 06:10 kiatng

For me.

  • app/etc/local.xml is userspace config, So Openmage can't enforce this unless user change it themself.

  • for app/etc/local.xml.template

    • It seems it convert to app/etc/local.xml on installation https://github.com/OpenMage/magento-lts/blob/18140508a9c58a0e844c465329f9b28807bd16d8/app/code/core/Mage/Install/Model/Installer/Config.php#L90-L95
    • If db_init_statemants is empty, It should produce app/etc/local.xml with empty initStatements
         ...
         <initStatements></initStatements>
         ...
      

And when Openmage evaluate configuration, It seems it look for every etc/*.xml sort by filename alphabetically.

  • app/etc/config.xml (Openmage enforce config)
  • app/etc/local.xml (User customization. Apply later, win app/etc/config.xml)

Here example setting on user side (local.xml)

Before this commit

<!-- local.xml, To enable charset -->
<config>
    <global>
        <resources>
            <default_setup>
               <connection>
                    <host>host</host>
                    <username>user</username>
                    <password>pass</password>
                    <dbname>openmage</dbname>
                    <!-- extra 2 line needs before this commit, To override config.xml  -->
                    <initStatements/>
                    <charset>utf8</charset>
               </connection>
            </default_setup>
            ...
        </resources>
        ...
    </global>
</config>
<!-- local.xml, To use initStatements -->
<config>
    <global>
        <resources>
            <default_setup>
               <connection>
                    <host>host</host>
                    <username>user</username>
                    <password>pass</password>
                    <dbname>openmage</dbname>
                    <!-- The default config in config.xml is already declare initStatements -->
               </connection>
            </default_setup>
            ...
        </resources>
        ...
    </global>
</config>

After this commit

<!-- local.xml, To enable charset -->
<config>
    <global>
        <resources>
            <default_setup>
               <connection>
                    <host>host</host>
                    <username>user</username>
                    <password>pass</password>
                    <dbname>openmage</dbname>
                    <!-- No extra config need to override config.xml -->
               </connection>
            </default_setup>
            ...
        </resources>
        ...
    </global>
</config>
<!-- local.xml, To use old initStatements -->
<config>
    <global>
        <resources>
            <default_setup>
               <connection>
                    <host>host</host>
                    <username>user</username>
                    <password>pass</password>
                    <dbname>openmage</dbname>
                    <!-- If user prefer initStatement, Need to do this -->
                   <initStatements>SET NAMES utf8 </initStatements>
                   <charset/>
               </connection>
            </default_setup>
            ...
        </resources>
        ...
    </global>
</config>
  • edit: Add example config on user side.

forfin avatar Oct 28 '25 04:10 forfin