+ Reply to Thread
Page 1 of 2 1 2 LastLast
Results 1 to 15 of 23

Thread: Storage as cache

  1. #1
    RIFT Community Ambassador the_real_seebs's Avatar
    Join Date
    Jan 2011
    Posts
    16,859

    Default Storage as cache

    Proposal: LibStorageCache

    What it does: Provide a large chunk of local savedvariables-backed storage which can be retrieved through Message.Send queries, and yet which also uses the Storage API to streamline.

    Basic design:

    lsc.Store(identifier, access, data)
    => stores this identifier in the local cache. May or may not store in player storage. Clears data if 'nil'. A clear will also attempt to clear the given tag from storage. "access" is the same as the read attribute for the player storage.
    lsc.Query(target, identifier, callback)
    => attempts to retrieve identifier from target's player storage, or failing that, sends a message and hopes for a reply.
    lsc.Info()
    => informs you as to the identifiers, size of storage, and number of times they've been directly queried, in your local cache.

    The idea is that this is super easy to use, but moderately hard to write. If this existed, you could use it without worrying about the total size of all stored items, and leave caching logic to the addon.
    You can play WoW in any MMO. You don't have to play WoW in RIFT. Oh, and no, RIFT is not a WoW clone. Not having fun any more? Learn to play, noob! I don't speak for Riftui, but I moderate stuff there. Just came back? Welcome back! Here's what's changed. (Updated for 2.5!)

  2. #2
    Telaran
    Join Date
    Mar 2012
    Posts
    83

    Default

    That idea sounds familiar. ;)

    I was thinking about a more database style to be a bit more abstract and therefore have the option to expand the system later on. In my head it looks like this:

    Code:
    local db = libStorage.open()   -- uses Inspect.Addon.Current() to identify the addon
    local myvar = db:get("myvalue", "defaultValue")
    db:set("myvalue", myvar)
    db:flush()
    There are several ideas hidden in the code (all yet to be filled up with details ).
    • The Inspect.Addon.Current() function to identify addons would allow the library to mark which addon actually queried data for sure, if the developer provides his own value that might or might not be the case.
    • The storage library preloads all data and compare it to the loaded addons by hooking the Event.Addon events. This would allow to organize the storage data without having to wait for addons to actually query it. This as well reduced bandwith usage to the absolute minimum and removes the need for a callback.
    • During the open() step the entire content of the calling addon is prepared for immediate access.
    • Data is only written if flush() is called. Same reason as above: reduce bandwith usage to the minimum and remove the need for a callback. Depending in the implementation, shutting down the library could enforce a flush for each/some addon.

  3. #3
    RIFT Community Ambassador the_real_seebs's Avatar
    Join Date
    Jan 2011
    Posts
    16,859

    Default

    The thing about a :get is that you can't actually expect it to return a value -- you may be several round trips to the server away from finding out the value.

    Hmm. Does querying a value in someone else's data distinguish between "no such value" and "permission denied"? My assumption would be that if permission is denied, there's no reason to fall back on message sending.
    You can play WoW in any MMO. You don't have to play WoW in RIFT. Oh, and no, RIFT is not a WoW clone. Not having fun any more? Learn to play, noob! I don't speak for Riftui, but I moderate stuff there. Just came back? Welcome back! Here's what's changed. (Updated for 2.5!)

  4. #4
    Telaran
    Join Date
    Mar 2012
    Posts
    83

    Default

    Further ideas to reduce the storage space used:

    Data could be queried structured like:
    Code:
    local myTable = db:getTable("myTable", {name, level, dps})
    myTable:get("name")
    This would allow to compress the content to a mere ; separated list of values, removing the need for all those identifiers to be stored. The table name could further be compressed by calculating a hash value instead (something about 16 bit should be sufficient for a table name, probably more for addon identifiers). Put all that through zlib and the content should be very small.

  5. #5
    Telaran
    Join Date
    Mar 2012
    Posts
    83

    Default

    Quote Originally Posted by the_real_seebs View Post
    The thing about a :get is that you can't actually expect it to return a value -- you may be several round trips to the server away from finding out the value.
    Not if you query the entire storage to local memory when the library loads and serve those gets from that local memory.

  6. #6
    RIFT Community Ambassador the_real_seebs's Avatar
    Join Date
    Jan 2011
    Posts
    16,859

    Default

    I am currently slightly inclined to view this as two separate questions. One is how you request a named datum from another player's storage (or possibly directly from them), the other is what the datum holds.

    Although obviously combining them is sorta useful.
    You can play WoW in any MMO. You don't have to play WoW in RIFT. Oh, and no, RIFT is not a WoW clone. Not having fun any more? Learn to play, noob! I don't speak for Riftui, but I moderate stuff there. Just came back? Welcome back! Here's what's changed. (Updated for 2.5!)

  7. #7
    Plane Walker Imhothar's Avatar
    Join Date
    Feb 2012
    Posts
    439

    Default

    Well I agree a common way of organizing such stuff is good, but I dont like the "May or may not store in player storage" part. The serverside storage is way too precious to let a library, which completely lacks any domain information, decide whether the data goes to the server or not.

    I suggest to see it fom two sides:
    1. Organization of the serverside storage
    2. Synchronization of data between players.

  8. #8
    Telaran
    Join Date
    Mar 2012
    Posts
    83

    Default

    Let's assume the storage contains the following content (not using hash values for the sake of readability):

    "libstorage={SimpleMeter=dps{Mike;50;1000;John;50; 600};Altometer=chars{Gahndalf;5p;Frodoo;10p;Gilmi; 2p}}"

    Then you could load all data through the libstorage content, split it up by the various addon names and turn them into hash tables as soon as the author of those addons tells you the structure of it. E.g for SimpleMeter:

    Code:
    local db = libStorage.open()
    local data = db:getTable("dps", {"name", "level", "dps"})
    for i, row in ipairs(data) do
      if (row:get("name") == "Mike") then
        print("Mike's dps is: ", row:get("dps"))
      end
    end
    This is just an example and the final implementation definitely would need some more thinking, like row lookup by value search (where name = "Mike").

    Edit: after reading your question again I figured that it is too late here for thinking. ;)
    Last edited by TwoThe; 03-28-2012 at 02:19 PM.

  9. #9
    RIFT Community Ambassador the_real_seebs's Avatar
    Join Date
    Jan 2011
    Posts
    16,859

    Default

    I was thinking that the local storage for the addon would probably be more a table of identifier:value pairs, with no attempt at compression at all; then if it wants to upload stuff, it should be uploaded with specific identifiers, so the Command.Storage.Get stuff can work with it.

    I think we're talking about two unrelated problems. I'm talking about the mechanism by which we get data from another player without needing to know whether the data is in the storage table or being queried from an addon they're running. You seem to be talking about how to represent stored data.
    You can play WoW in any MMO. You don't have to play WoW in RIFT. Oh, and no, RIFT is not a WoW clone. Not having fun any more? Learn to play, noob! I don't speak for Riftui, but I moderate stuff there. Just came back? Welcome back! Here's what's changed. (Updated for 2.5!)

  10. #10
    Plane Walker Imhothar's Avatar
    Join Date
    Feb 2012
    Posts
    439

    Default

    Adding some thoughts into this:
    1. You can only read the serverside storage from players who are online.
    2. When a player is online, and you want information from them, the easiest way is still to ask it directly using messages and ignoring the storage.
    3. Some people seem like they want to misuse the storage as a replacement for saved variables. Always ask yourself: What does the storage offer what saved variables cannot solve?
    4. Most scenarios I can come up with for player storage can be solved using messages and saved variables. Guild storage, however is a completely different case because data needs to be persisted even while no member is online.

  11. #11
    RIFT Community Ambassador the_real_seebs's Avatar
    Join Date
    Jan 2011
    Posts
    16,859

    Default

    Storage does two things:
    1. Really Important Stuff can be in storage so you have it even if you are on a new computer with no prior stuff. Also, it's available even if you haven't got any addons loaded!
    2. Massive efficiency boost for frequently-requested data.

    I'm guessing the size will end up being a little larger. Server cap is about 2k, so we're looking at about 4MB of storage, which sounds like it's pretty small by modern hardware standards (and keep in mind, a "server" is really a collection of machines, only one of which needs to have that data loaded). But the basic idea is, storage is there to be an efficiency cache for stuff that people request often, because querying storage costs about half the bandwidth of a direct message.

    So I'm trying to provide an interface that lets us do the {player,identifier}->datum mapping generically without needing to know whether it's in storage, so we don't care how large it is.
    You can play WoW in any MMO. You don't have to play WoW in RIFT. Oh, and no, RIFT is not a WoW clone. Not having fun any more? Learn to play, noob! I don't speak for Riftui, but I moderate stuff there. Just came back? Welcome back! Here's what's changed. (Updated for 2.5!)

  12. #12
    Plane Walker Imhothar's Avatar
    Join Date
    Feb 2012
    Posts
    439

    Default

    Quote Originally Posted by the_real_seebs View Post
    Storage does two things:
    1. Really Important Stuff can be in storage so you have it even if you are on a new computer with no prior stuff. Also, it's available even if you haven't got any addons loaded!
    1. What data can be so crucial, that it must be present on every machine?
    2. If you don't have any Addon than you don't have any data in your storage, because only the player himself can modify the player storage.
    Quote Originally Posted by the_real_seebs View Post
    2. Massive efficiency boost for frequently-requested data.
    I think one can argue about this point quite a bit. Remember receiving stuff from storage or a different player generates the same bandwidth for the player requesting the information. I don't really see a "massive" boost to be honest. If you see efficiency problems with requestig data, then your Addon has a suboptimal design.

    push vs. poll

    Now don't get me wrong, I'm not against a library for synchronization. One just has to measure whether the serverside storage is really the best solution for a given problem, simply because it is so limited. And with limited I don't mean the hardware limitations but the 2048 bytes available to players.

    The morale of the story: avoid using the server storage if not absolutely necessary and all other solutions would be overly complicated, unmaintainable or bandwidth eating monsters.

  13. #13
    RIFT Community Ambassador the_real_seebs's Avatar
    Join Date
    Jan 2011
    Posts
    16,859

    Default

    Quote Originally Posted by Imhothar View Post
    1. What data can be so crucial, that it must be present on every machine?
    2. If you don't have any Addon than you don't have any data in your storage, because only the player himself can modify the player storage.
    Log in, once, run an RP addon, save data to storage. Now delete all your addons. Your data will still be in storage, because it persists between sessions.

    As to what's that crucial... I might not mind having something like an ignore list or alt list maintained in a way that preserved it even when I hop between machines.

    I think one can argue about this point quite a bit. Remember receiving stuff from storage or a different player generates the same bandwidth for the player requesting the information. I don't really see a "massive" boost to be honest. If you see efficiency problems with requestig data, then your Addon has a suboptimal design.
    It's a factor of two difference, pretty much exactly.

    Say a request is 64 bytes and the answer is 1k. If I request it from the server, total bandwidth is 1088 bytes. If I request it from a player, I send the server 64 bytes, the server sends the player 64 bytes, then the player sends 1024 bytes to the server, which sends it on to me, for exactly twice the bandwidth.

    Now say I upload my RP bio, and 500 players look at it. I upload 1k, 500 people download 1k, is a lot less bandwidth than me uploading 500k and 500 people downloading 1k.

    Now don't get me wrong, I'm not against a library for synchronization. One just has to measure whether the serverside storage is really the best solution for a given problem, simply because it is so limited. And with limited I don't mean the hardware limitations but the 2048 bytes available to players.

    The morale of the story: avoid using the server storage if not absolutely necessary and all other solutions would be overly complicated, unmaintainable or bandwidth eating monsters.
    My plan was to design the library so it would hide the question of whether server storage was being used, and then that one library could try to figure out how to optimize use of available storage.
    You can play WoW in any MMO. You don't have to play WoW in RIFT. Oh, and no, RIFT is not a WoW clone. Not having fun any more? Learn to play, noob! I don't speak for Riftui, but I moderate stuff there. Just came back? Welcome back! Here's what's changed. (Updated for 2.5!)

  14. #14
    Sword of Telara DoomSprout's Avatar
    Join Date
    Apr 2011
    Posts
    876

    Default

    I came to the same conclusion as Imhothar tbh. Guild storage and messaging have definite uses, player storage seems less useful. To query another player's storage, they need to be online, where a message would work without the limitation on data size. Using server storage as a cache would also result in very expensive cache misses.

    This is purely my thoughts, but in my designs I decided getting data from another player is best handled through messaging, and storing your own data is best handled using saved variables, except in the case that you need that data available on multiple machines.

    Once example where it *would* be useful is the bio data mentioned by Seebs. You want your character's bio to persist across all machines, so in this instance saved variables aren't that useful. I think there are relatively few examples of this though, and in most cases player storage should probably be avoided. It's quite simply too small (and therefore precious) to use if it's not absolutely necessary.

  15. #15
    RIFT Community Ambassador the_real_seebs's Avatar
    Join Date
    Jan 2011
    Posts
    16,859

    Default

    Is the cache miss really expensive though?

    My thought is, as long as misses are cheap, checking that should be practical.
    You can play WoW in any MMO. You don't have to play WoW in RIFT. Oh, and no, RIFT is not a WoW clone. Not having fun any more? Learn to play, noob! I don't speak for Riftui, but I moderate stuff there. Just came back? Welcome back! Here's what's changed. (Updated for 2.5!)

+ Reply to Thread
Page 1 of 2 1 2 LastLast

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts