[2.1 & 3.0]: Upgrader serialize-to-json logic issues #9259

Activity

1. sbulen

   added

   Not Verified

   on Jun 7, 2026
2. sbulen commented on Jun 8, 2026

   

   sbulen

   on Jun 8, 2026

   Last edited by sbulen

   ContributorAuthor

   More actions

   Peeling the onion...

   At start, this upgrade took 18 hours & would not complete. It timed out repeatedly, & crashed during the serialize_to_json step. The serialize_to_json step was partially completed, and the parts that were completed were mostly incorrect.

   Steps taken:

   • Convert to InnoDB before upgrading. This reduced the time of execution from 18 hours to < 5 hours. The reason is that MyISAM tables must be rebuilt during most ALTER TABLE operations. And the upgrade has A LOT of ALTER TABLE operations on the smf_messages table. Don't do that.
   • Change the upgrader's set_time_limit calls to set_time_limit(0) - unlimited. The upgrader no longer crashes silently on timeouts, & no longer needs restarts. It just runs all the way through, like it should.
   • Add $db_character_set = 'latin1'; to Settings.php prior to upgrade. This causes SMF & the upgrader to issue an explicit SET NAMES, and queries return the proper latin1 values as stored in the DB. After doing this, the serialize_to_json step is 99% successful.

   The above 3 steps make the upgrader far faster & cleaner. It just runs, in a fraction of the time, and does a far better job on the json conversion. However...

   Open issue:

   • serialize_to_json still crashes under some circumstances. The circumstance is when the string includes extended latin1 characters, such as single-byte umlauts.

   Why? Because json_encode ONLY operates on utf8, and those extended latin1 characters are not valid utf8.

   In this instance, there are usernames with extended characters in the memberlist_cache... Although the memberlist_cache is now being properly unserialized, the subsequent json_encode is failing & returning false due to invalid utf8 chars. This triggers the 'value - 1' in updateSettings(), and the crash.

   There is a json_encode option to ignore invalid utf8. This avoids the crash, however, json_encode then substitutes entity encoded values in their place, e.g., '\u0665\u01a5'. But even this is incomplete - there are many chars it simply strips (yikes!!!)... I haven't found a set of json_encode flags that just leave the string alone.

   Still exploring solutions.
3. sbulen commented on Jun 8, 2026

   

   sbulen

   on Jun 8, 2026

   ContributorAuthor

   More actions

   In the error above, where MariaDB was asked to decrement the value of memberlist_cache & instead blew up...

   The next obvious question is... what about MySQL?

   This is what MySQL did when asked to decrement the memberlist_cache string: 😆

   
4. sbulen commented on Jun 8, 2026

   

   sbulen

   on Jun 8, 2026

   Last edited by sbulen

   ContributorAuthor

   More actions

   This isn't helping... It should be $new_sting, not $string... So it's not even using the fixed serialized strings.
   

   SMF/other/upgrade.php

   Line 3528 in 2a89e0b

   $data = @safe_unserialize($string);
5. jdarwood007 commented on Jun 8, 2026

   

   jdarwood007

   on Jun 8, 2026

   Member

   More actions

   Send the fixes to move the InnoDB around for 3.0. I don't know if we can for 2.1. Seems like it shouldn't cause any possible issues, but I would like to see what @Sesquipedalian feels on that.

   The timeout issue, send for both.

   For the character set fix.. This seems dangerous if the character set is actually UTF-8 already to specify latin1.

   The new_string fix should be sent up as well.

   👍React with 👍1Reacted by sbulen
6. jdarwood007

   added

   Upgrader

   and removed

   Not Verified

   on Jun 8, 2026
7. jdarwood007

   added the

   Bug

   issue type on Jun 8, 2026
8. jdarwood007

   added this to the 2.1.8 milestone on Jun 8, 2026
9. sbulen commented on Jun 9, 2026

   

   sbulen

   on Jun 9, 2026

   Last edited by sbulen

   ContributorAuthor

   More actions

   Yeah, I wasn't going to send InnoDB conversion up for 2.1. Fortunately, 3.0 already does it. Just sharing for info. I certainly wasn't aware before this that, for these operations, MyISAM is dramatically slower, like 3-4x slower.

   The others are pretty straightforward.

   The charset issues are actually getting the db to treat latin1 like latin1... I specified latin1 above because the test DB I'm working with is latin1.
10. sbulen commented on Jul 20, 2026

    

    sbulen

    on Jul 20, 2026

    ContributorAuthor

    More actions

    I wrote a little QnD utility to populate a 2.0 DB with a few million rows populated with heaps of latin1 (one-byte, non-utf8) data:
    https://github.com/sbulen/sjrbTools/blob/master/smf_populate_db_20.php

    Ugly, but effective...

    This will help folks trying to reproduce issues such as those discussed in #9259, #9270, #9271, #9272 & #9278 including:

    • various issues caused by sheer volume & execution time
    • charset issues
    • json encoding issues
11. jdarwood007

    modified the milestones: 2.1.8, 2.1.9 on Jul 25, 2026