+ Reply to Thread
Results 1 to 2 of 2

  Click here to go to the first Rift Team post in this thread.   Thread: Table serializing/deserializing, truncated SavedVariables & LuaJIT limitations

  1. #1
    Plane Touched
    Join Date
    Feb 2012
    Posts
    228

    Default Table serializing/deserializing, truncated SavedVariables & LuaJIT limitations

    This thread discusses about these three topics (see thread title) and how they could be the cause of long SavedVariables files being truncated on addon load.

    My experience begins with the first releases of my addon (BananAH), where I just dumped my auction table into shard saved variables. The problem was whenever the saved variables file exceeded a certain size (10MB), the next time it was loaded all saved variables were lost, so I came to the conclusion that the addon system was preventing large files from being loaded. The auction table was able to store about 2/3 days of data before reaching this limit.

    Between 0.2.1 and 0.2.2 I started packing the auction data to reduce its size (achieving a 85% compression rate) and pruning data older than 7 days, so it never reached more than 5MB anymore and the problem was solved.

    With 1.8 I tested the new zlib API, achieving a 93% compression rate when just feeding it with the table serialized through Utility.Serialize.Inline, and a 96% compression rate when feeding it the packed table. As the difference was negligible, and my packing/unpacking functions were expensice, I choose to use the first approach.

    Now, three days of data later, when I try to load my auction data, I get this:

    Code:
    dump({pcall(loadstring, "return " .. zlib.inflate()(BananAHAuctionTable))})
    Returns: {true, nil, "[string \"return {[\"I00292565575E24C6,2BDAE862E3253166,...\"]:1: main function has more than 65536 constants"}
    Searching for this error points me to this thread: http://lua-users.org/lists/lua-l/2010-03/msg00238.html, which is consistent with my experience, as each auction is stored in a nested table, and the auction volume in my shard is about 20.000 auctions per day.

    As the SavedVariables processes use a method similar to what came to be Utility.Serialize.Full (by this Zorba post) to serialize/deserialize tables, is it possible that the 10MB limitation on SavedVariables never existed, but they were truncated as a consequence of this LuaJIT limitation making the deserialization fail?

    TL;DR

    1. Due to a LuaJIT limitation, when you try to deserialize a tree structure serialized through Utility.Serialize.* which had more than 65536 nested tables, it fails silently, returning a nil value, and, in fact,truncating the variable.

    2. If SavedVariables is using a serializer similar to our Utility.Serialize.* calls, that would explain SavedVariables being truncated sometimes for no apparent reason.

  2.   This is the last Rift Team post in this thread.   #2
    Rift Team
    Join Date
    Oct 2010
    Posts
    927

    Default

    Quote Originally Posted by Baanano View Post
    TL;DR

    1. Due to a LuaJIT limitation, when you try to deserialize a tree structure serialized through Utility.Serialize.* which had more than 65536 nested tables, it fails silently, returning a nil value, and, in fact,truncating the variable.

    2. If SavedVariables is using a serializer similar to our Utility.Serialize.* calls, that would explain SavedVariables being truncated sometimes for no apparent reason.
    Yep, I bet that's exactly what's happening. We'll get on that!

    SavedVariables is actually using the same exact serializer as Utility.Serialize.Full(). The solution is going to be rigging up Utility.Serialize.Full() so it splits apart lines with more than 64k inline tables. This will simultaneously fix the SavedVariables limit (which I didn't realize existed until now) and the limit with the Serialize API.

    Thanks for the info

    Be warned, this may take a week or so to flush through, maybe a little longer if it turns out to be a difficult fix.

+ Reply to Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts