+ Reply to Thread
Page 1 of 2 1 2 LastLast
Results 1 to 15 of 20
Like Tree1Likes

  Click here to go to the first Rift Team post in this thread.   Thread: Item Type

  1. #1
    Plane Touched
    Join Date
    Feb 2012
    Posts
    228

    Default Item Type

    Not sure if this has been asked before, if so please redirect me to the relevant threads.

    I'm trying to write an AH addon and I need to group similar items to get proper statistics for them (average unitPrice, volume of auctions...)

    Grouping by item name isn't a good idea, as it wouldn't work if the user switches locales and also because two items with the same name can have different rarities, which I'd like to group into different categories.

    I'd rather group by item type, which works amazingly well with crafting materials and similar items, but fails on equipment because the same item can have different types depending on the modifiers applied to it (runes, dyes...). For example this item has two different types because one has a rune and the other hasn't:

    ["Nimble Deathbringer Brigandine"] =
    {
    "I52B19BD61CD8962E,4224A3AE1BB6857A,224E5C0614A54B 60,,,0000004800000064,,",
    "I52B19BD61CD8962E,592EA5D4B6FDC604,224E5C0614A54B 60,,,0000004800000064,r531D12F64D843332,"
    }

    If I use the first chunk of the item type (text before the first comma), I'd lose rarity info as in this one:

    ["Noble Helm of the Defender"] =
    {
    "I6F7F7A602C4E3923,A1231A408F9FF310,,0A1F25CD50418 CE7,000000200000003C,,,", -- rare
    "I6F7F7A602C4E3923,F34DB02E5F14EE7D,,0A1F25CD50418 CE7,000000200000001E,,," -- uncommon
    }

    Also, I've found items with the first chunk of the item type and their rarities being equal but having different names (can provide examples later).

    I've also found items that, being the same in game (same name, same stats, same rarity) have different types, as this one:

    ["Zealous Tysumi's Treatise on Arcane Travel"] =
    {
    "I2D39EFCC5572ECC0,BA4B9D318B6B351F,32D0737F396621 D3,,,000000320000003C,,", -- epic
    "I2D39EFCC5572ECC0,EC2D500E5E61BC45,32D0737F396621 D3,,,0000003B00000064,," -- epic
    }

    Do we have any other way to do this rather than using the item type? If not, do we know what each chunk of the item type string means in order to write a "normalizing" function?

  2. #2
    Rift Disciple
    Join Date
    Mar 2011
    Posts
    130

    Default

    I haven't tackled this one yet on my end, but my initial idea was to "match" items ids and rarity in a separate table and cross-reference this table before executing calculations when a "similar" item is found.

    I don't know if this will work or not, but that's the best I got so far. My fear is that I would end up with a huge saved variables just to maintain that list.

  3. #3
    Plane Touched
    Join Date
    Feb 2012
    Posts
    228

    Default

    I've done some research on the eight parts that compose the item type and found this:

    1) Base Item ID

    The first part of determines the base type of the item. For example, in the following list of item types:

    Code:
    Cunning Darkcloth Shroud (epic):                   I5F6C21881471A11A,3714E7ED85A68CD7,07655CCB42C52FCB,,,000000320000003C,r2F25D3437A3C489F,
    Clever Darkcloth Shroud (epic):                    I5F6C21881471A11A,4C980695C716CE78,6EBEEFF87FA2FD2A,,,0000004800000064,r2F25D3437A3C489F,
    Darkcloth Shroud (epic):                           I5F6C21881471A11A,5C3D4D21922D1B81,,,,,,
    
    Brigand's Shoulderpads of the Stalwart (rare):     I617CBDFB65632909,876281F28EB92AE7,,2331D5BC0129900B,000000190000003C,,,
    Brigand's Shoulderpads of the Sinister (uncommon): I617CBDFB65632909,892B8CCA282B5778,,5B1DEB526157C7B0,000000190000001E,,,
    Both groups share the same first part of the item type, so they are of the same base type (Darkcloth Shroud and Brigand's Shoulderpads) but with different modifiers applied to them.

    This part is present in every item type.

    2) Unknown

    I can't guess what the second part of the item type is, but it changes every time any other part changes, and if you change any part to a different value, the Inspect.Item.Detail call fails, so I'd say it's some kind of hash or checksum.

    Edit: Changing the eighth value to any random value doesn't make the function call to fail, neither changes this second part of the item type. However, the type returned by Inspect.Item.Detail has the eighth value emptied again.

    This part is present in every item type.

    3) Augment ID

    The third part of the item type determines the augment applied to the item (and the prefix applied to the item name). Each augment seems to have a fixed value. For example, in the following list of items:

    Code:
    Nimble Deathbringer (epic):              I5CD0EF602E6910BB,FC014176029B6ECB,224E5C0614A54B60,,,0000004800000064,r2ACE12CA705875B9,
    Nimble Deathbringer (epic):              I5CD0EF602E6910BB,5EE520EADE7B1BB4,224E5C0614A54B60,,,0000004800000064,,
    Nimble Carmintium Greatsword (uncommon): I682C411032B85CA7,AA2104B450F64E51,224E5C0614A54B60,,,0000002B0000001E,,
    
    Savage Spined Compound Bow (rare):       I7272C4DC214E5AC5,88096A3A43AE1803,3D593CEB751E1269,,,0000002D0000001E,,
    Savage Deathbringer (epic):              I5CD0EF602E6910BB,790E969EA7A81A21,3D593CEB751E1269,,,000000320000003C,,
    
    Accurate Deathbringer (epic):            I5CD0EF602E6910BB,626635D11518D1DE,068D89496E33F686,,,0000003900000064,,
    Accurate Spined Compound Bow (rare):     I7272C4DC214E5AC5,22B08ADAC2907DBB,068D89496E33F686,,,000000300000003C,,
    Each group shares the same third part of the item type, and they have the same augment prefix.

    This part of the item type seems to only be present on augmented items, but I'll do further research to confirm both that every augmented item has a third part, and that every item with a third part is augmented.

    4) Random Modifier ID

    The fourth part of the item type determines the random modifier or "natural augment" of the item (and the suffix applied to the item name). Each random modifier seems to have a fixed value. For example, in the following list of items:

    Code:
    Officer's Greaves of the Warfront (rare):       I743360B637A00E91,46215FCACDECEA1E,,7935A02E255409D3,0000002A0000003C,,,
    Officer's Greaves of the Warfront (uncommon):   I743360B637A00E91,1D6C137BA4345A62,,7935A02E255409D3,0000002A0000001E,,,
    Marksman's Tunic of the Warfront (rare):        I752F38024666FD40,28E8F9B673A9F74F,,7935A02E255409D3,000000310000003C,,,
    
    Marksman's Tunic of the Stalwart (rare):        I752F38024666FD40,2F8E9411D614A25B,,2331D5BC0129900B,000000310000003C,,,
    Renegade's Shoulderpads of the Stalwart (rare): I756A7D4016BDB058,1BCF5FC0EF888754,,2331D5BC0129900B,000000290000003C,,,
    Both groups have the same fourth part of the item type, and the same random modifier suffix.

    This part of the item type seems to only be present on items with a random modifier, but I'll do further research to confirm it.

    Also, I haven't found any item with both the third and the fourth part present at the same time (which makes sense, as items with augment are craftable items and items with random modifier are random drops).

    5) Random Modifier "Power Level"

    The fifth part of the item type seems to determine the "power level" of the random modifier, as similar items with a higher fifth part have better stats.

    I've researched this part more than the others as some Planar essences have unusual values for it (in fact the Inspect.Item.Detail call fails on them) and found we can divide this part in two DWORDs and each one have a different meaning:

    The lower DWORD determines the rarity of the item:
    0x0000001E: uncommon
    0x0000003C: rare
    0x00000064: epic

    The higher DWORD is the required level + 2, for example:
    0x00000025 = 37 -> Required level 35
    0x00000034 = 52 -> Required level 50

    As usually, some examples:

    Code:
    Veteran's Hood of the Stalwart (uncommon, reqLvl 35): I7A107F660BEFBC5E,78588AB66351E7E7,,2331D5BC0129900B,000000250000001E,,,
    Scholar's Robe of the Mythical (uncommon, reqLvl 35): I76E60EC86D7406D2,85C8163EF9113400,,522DF7630EE801CD,000000250000001E,,,
    Veteran's Hood of the Tenacious (rare, reqLvl 35):    I7A107F660BEFBC5E,ED96579A967FC4B6,,61EACC89726E9C7A,000000250000003C,,,
    Veteran's Hood of the Sinister (epic, reqLvl 35):     I7A107F660BEFBC5E,F014C6996E642931,,5B1DEB526157C7B0,0000002500000064,,,
    Ascendant Boots of the Sinister (epic, reqLvl 50):    I7696FF2D0D844A96,31CFE326E6B4F31F,,5B1DEB526157C7B0,0000003400000064,,,
    As I said before, some Planar Essences have unusual values:

    Code:
    Eternal Faestone of the Tactician (uncommon):   I7F772CD50FED2A07,49A107449FFF2676,,61D3F28B075B4A89,FFFFFFFFFFFFFFFF0000001E,,,
    Eternal Shadestone of the Tactician (uncommon): I7EE10E1A19751391,CD499DD734B74F5A,,4FD27CD669BA19AD,FFFFFFFFFFFFFFFF0000001E,,,
    Eternal Shadestone of the Core (uncommon):      I7EE10E1A19751391,FBD302205F9C623F,,25389D3754409AA4,FFFFFFFFFFFFFFFF0000001E,,,
    They have a third DWORD and, moreover, their fourth parts don't match even when two of them have the same random modifier (of the Tactician).

    These item types give an error when used to call Inspect.Item.Detail, but if you strip out the third DWORD they return their usual values:

    Code:
    Inspect.Item.Detail("I7F772CD50FED2A07,49A107449FFF2676,,61D3F28B075B4A89,FFFFFFFFFFFFFFFF0000001E,,,") -> error
    Inspect.Item.Detail("I7F772CD50FED2A07,49A107449FFF2676,,61D3F28B075B4A89,FFFFFFFF0000001E,,,") -> ok, but the type returned has the third DWORD again
    This fifth part of the item type seems to only be present on items with a random modifier (that is, their fourth part also has a value), but I'll do further research to confirm it.

    6) Augment "Power Level"

    The sixth part of the item type seems to determine the "power level" of the augment applied when the item was crafted, as similar items with higher sixth part have better stats.

    As in the fifth part, you can divide it in two DWORDs that seem to have a meaning.

    The lower DWORD determines the rarity of the augment applied:
    0x0000001E: uncommon augment
    0x0000003C: rare augment
    0x00000064: epic augment

    The higher DWORD is the level of the augment applied + 2:
    0x00000012 = 18 -> Augment level 16
    0x0000002D = 45 -> Augment level 43
    Some rare augments and all of the epic exceed level 50 when using the above formula, so either rare and epic items follow a different one, or there are different augment levels at level 50.

    Some examples:
    Code:
    Tysumi's Treatise on Arcane Travel:         I2D39EFCC5572ECC0,60155E8E9F45D89C,,,,,,
    Zealous Tysumi's Treatise on Arcane Travel: I2D39EFCC5572ECC0,BA4B9D318B6B351F,32D0737F396621D3,,,000000320000003C,,
    Zealous Tysumi's Treatise on Arcane Travel: I2D39EFCC5572ECC0,EC2D500E5E61BC45,32D0737F396621D3,,,0000003B00000064,,
    - Respective stats:
      critSpell = 10, endurance = 17, focus = 13, intelligence = 23, powerSpell = 214, wisdom = 23
      critSpell = 14, endurance = 17, focus = 13, intelligence = 23, powerSpell = 214, wisdom = 28
      critSpell = 15, endurance = 17, focus = 13, intelligence = 23, powerSpell = 214, wisdom = 29
    
    Leggings of the Hunter:                     I3460ED866550EA68,B62826387CED5300,,,,,,
    Keen Leggings of the Hunter:                I3460ED866550EA68,BBBB0F7F67CE99A6,1C090799708101E2,,,000000320000003C,,
    Keen Leggings of the Hunter:                I3460ED866550EA68,EF8F90FC062928BA,1C090799708101E2,,,0000003B00000064,,
    - Respective stats:
      armor = 506, critAttack = 21, dexterity = 28, endurance = 17,                  strength = 20
      armor = 506, critAttack = 21, dexterity = 33, endurance = 17, powerAttack = 3, strength = 20
      armor = 506, critAttack = 21, dexterity = 35, endurance = 17, powerAttack = 4, strength = 20
    This sixth part of the item type seems to only be present on items with an augment applied (that is, their third part also has a value), but I'll do further research to confirm it.

    7) Rune ID

    The seventh part of the item type, if present, determines the rune applied to the item.

    I haven't researched if it has any additional meaning, but two items with the same seventh part have the same rune applied.

    8) Unused?

    I haven't found any item type with a value in its eighth part.

    In fact you can assign whatever value you want to it and Inspect.Item.Detail will buy it as a valid item type, but the type field of the returned table will have it emptied again.

    TL;DR

    Item types returned by Inspect.Item.Detail return a string composed by eight parts delimited by commas. My guess on which the meaning of each part is:

    1.- Base Item Type: Base name
    2.- Unknown: Probably a checksum that prevent us from forging item types
    3.- Augment applied: Prefix on the name, and stats affected
    4.- Random modifier applied: Suffix on the name, and stats affected
    5.- Random modifier power: Amount each stat affected by the random modifier is increased
    6.- Augment power: Amount each stat affected by the augment is increased
    7.- Rune: Rune applied to the item
    8.- Unused

    Corrigendum

    1.- In my earlier post I said the dye of an item changed its item type, but I've done some tests and that isn't true: the dye applied to an item doesn't change the item type.

    2.- In my earlier post I said the two Zealous Tysumi's Treatise on Arcane Travel had the same stats, which isn't true, as shown in the examples of the sixth part.
    Last edited by Baanano; 02-15-2012 at 12:38 PM. Reason: Updated

  4.   Click here to go to the next Rift Team post in this thread.   #4
    Rift Team
    Join Date
    Oct 2010
    Posts
    927

    Default

    Quote Originally Posted by Baanano View Post
    As I said before, some Planar Essences have unusual values:

    Code:
    Eternal Faestone of the Tactician (uncommon):   I7F772CD50FED2A07,49A107449FFF2676,,61D3F28B075B4A89,FFFFFFFFFFFFFFFF0000001E,,,
    Eternal Shadestone of the Tactician (uncommon): I7EE10E1A19751391,CD499DD734B74F5A,,4FD27CD669BA19AD,FFFFFFFFFFFFFFFF0000001E,,,
    Eternal Shadestone of the Core (uncommon):      I7EE10E1A19751391,FBD302205F9C623F,,25389D3754409AA4,FFFFFFFFFFFFFFFF0000001E,,,
    They have a third DWORD and, moreover, their fourth parts don't match even when two of them have the same random modifier (of the Tactician).

    These item types give an error when used to call Inspect.Item.Detail, but if you strip out the third DWORD they return their usual values:
    Whoops. Good catch, we'll get that fixed.

  5. #5
    Plane Touched
    Join Date
    Feb 2012
    Posts
    228

    Default

    Zorba, I won't ask you how wrong I am with my assumptions but I'd like to ask you a couple of things:

    When I call Inspect.Item.Detail(itemType) inside a long loop, at some point it starts throwing errors for valid itemTypes, and then it continues throwing errors for some minutes even if I call it at a slower pace.

    Is it being queued as happens with auction house full scans? If so, can that queue status be queried?

    I know I can circumvent this by calling the function with a huge table, but which would be a sensible limit? Sometimes the game crash with just 5,000 item types, while others it works with more than 100,000 (even though it takes too long).

    Would I be queued if I called Inspect.Item.Detail in a loop with, lets say, 5,000 item types splitted in 50 calls of 100 item types each?


    On another note, could you add an exactText filter for Command.Auction.Scan for when you only want "Mana Potion" and not "Greater Mana Potion"?

  6.   Click here to go to the next Rift Team post in this thread.   #6
    Rift Team
    Join Date
    Oct 2010
    Posts
    927

    Default

    Quote Originally Posted by Baanano View Post
    Zorba, I won't ask you how wrong I am with my assumptions but I'd like to ask you a couple of things:
    I'll answer without you asking :V From what I recall, you've correctly analyzed the whole thing. Some of the magic numbers in there ("rarity" and "requiredLevel") do, as I recall, have more values they can take on - I'm kind of surprised the required-level is so consistent, to be honest - but you've got the basic pattern down.

    Quote Originally Posted by Baanano View Post
    When I call Inspect.Item.Detail(itemType) inside a long loop, at some point it starts throwing errors for valid itemTypes, and then it continues throwing errors for some minutes even if I call it at a slower pace.
    I'm not quite sure what's going on here, to be honest. Where are you getting the item types from exactly?

    The issue is that the client is not aware of most items, it has to retrieve them from the server. If it doesn't have an item available then it will give you an error and then quietly send a request out to the server, and with luck, the data will be available the second time you request information. Now, that request is throttled, but there's no way to retrieve its status (somewhat intentionally, to prevent data mining) and it results in massive slowdown, not errors.

    If you can provide an example addon to demonstrate the behavior I'll check out what's going on.

    All that said, if it turns out that we're burning a lot of bandwidth on item requests, we may have to throttle it more severely. We recommend pulling item data down only when you actually need it - if you're making an addon that lets you, say, search over all items, include a local database with your addon, don't pull it all down from the server for every single user

    The actual pattern of how to call Inspect.Item.Detail() should be irrelevant, the throttling is a per-item throttle.

    Quote Originally Posted by Baanano View Post
    On another note, could you add an exactText filter for Command.Auction.Scan for when you only want "Mana Potion" and not "Greater Mana Potion"?
    I wouldn't expect this to be showing up soon, but we've had requests to add this to the main AH interface. If we do end up adding it to that, we'll also add the right hooks for the auction API to use it.

  7. #7
    Plane Touched
    Join Date
    Feb 2012
    Posts
    228

    Default

    I get the item types from a saved variable table, where I add the new item types I get each time I perform an auction house scan, so they should be valid item types as they were supplied by the game itself.

    I'm trying to minimize the amount of variables I need to store in that table to reduce its size, and if I could be sure that Inspect.Item.Detail would always return the item details when called with a valid item type, then I wouldn't have to store things like itemName, vendorPrice, itemIcon... as I could call Inspect.Item.Detail every time I needed them.

    Then I found the item types for some planar essences aren't valid and cause the function to fail. This made me wonder if there were other item types that would cause it to fail, so I tried to run something like this:

    Code:
    local function CheckItem(id)
    	local ok = pcall(Inspect.Item.Detail, id)
    	return ok
    end
    
    for i = 1, #tab do -- tab is the table with the item types (there were about 7000 item types)
    	if not CheckItem(tab[i]) then
    		print(i)
    	end
    end
    Event though it eated all my CPU, the output was something like:

    23
    57
    134
    135
    136
    137
    ... and then every index until the end of the table

    If I executed that code again it would print every index from 1 to #tab every time I tried, so I supposed Inspect.Item.Detail was queued.

    I ended wrapping the call in a coroutine and hooking it to Event.System.Update.Begin to slow the pace and then it only gave errors for the item types of the planar essences.

    I plan to test the item types I've parsed from the huge Items.xml from your FTP (the Discoveries file) into a lua table of about 4 MB, as I think some of them aren't valid item types, so I could post the exact code I run and the parsed lua table later today.
    Last edited by Baanano; 02-16-2012 at 07:33 AM.

  8.   Click here to go to the next Rift Team post in this thread.   #8
    Rift Team
    Join Date
    Oct 2010
    Posts
    927

    Default

    Quote Originally Posted by Baanano View Post
    If I executed that code again it would print every index from 1 to #tab every time I tried, so I supposed Inspect.Item.Detail was queued.

    I ended wrapping the call in a coroutine and hooking it to Event.System.Update.Begin to slow the pace and then it only gave errors for the item types of the planar essences.
    Once things are cached, it will happily return all those items each time you ask for 'em. I'm not entirely sure when it clears the cache, but you would at the very least have to return to character select. You may have to close the client entirely. /reloadui definitely wouldn't clear the cache.

  9. #9
    Plane Touched
    Join Date
    Feb 2012
    Posts
    228

    Default

    That would explain why, after an AH scan, Inspect.Item.Detail worked with over 100,000 item types (which really are the sample I got from the AH repeated 200 times), as they would have been cached during the AH scan, but after some crashes the function failed with just those same 5,000 item types loaded from SavedVars, as the cache would have been cleared when the client was restarted.

    I've been wondering since the last week if item details were on the client files or queried to the server each time you needed them, and planned to unplug my connection and call Inspect.Item.Detail before disconnection to see if it returned a value (client side) or not (server side). Now I know they're stored server side but cached in the client during the session, but it stills seems fun enough to try

    As of the code, actually it wasn't as simple as I posted earlier, but I'll do my best to make it fail and post the actual code and item types used so you can try to reproduce it.

  10.   Click here to go to the next Rift Team post in this thread.   #10
    Rift Team
    Join Date
    Oct 2010
    Posts
    927

    Default

    Quote Originally Posted by Baanano View Post
    As of the code, actually it wasn't as simple as I posted earlier, but I'll do my best to make it fail and post the actual code and item types used so you can try to reproduce it.
    If it's doing what you just described, then no worries about giving me reproducible code, since that's how it's meant to work

  11. #11
    Plane Touched
    Join Date
    Feb 2012
    Posts
    228

    Default

    Yes, everything makes sense now: the function isn't really being queued, it just fails, as you said, when the client hasn't cached the item yet. It could be interesting to add it to the documentation, though

    I've indeed done my research and found interesting results (at least for me). All the tests were performed using LuaPad and a lua table loaded as a global variable named ItemTable. I cleared the cache before each test.

    I can't attach the actual file as it's too large to upload to the forum, but instead I've attached the XSLT file Items.xslt.txt used to generate it. It was applied to the Items.xml of Discoveries from the FTP with date 10-Feb-2012. If anybody wants the actual file, ask for it and I'll upload it somewhere.

    The test were commented while running them. Beware of the wall of text!

    Test 1.1: Check how the cache works
    Code:
    local function CheckItem(id)
    	local ok, msg = pcall(Inspect.Item.Detail, id)
    	return ok
    end
    
    for i,id in ipairs(ItemTable) do
    	if not CheckItem(id) then 
    	 	print("Stopped on item #" .. i .. ": " .. id)
    	 	break 
    	end
    end
    First time it fails on every item type but advances until it stops on the 65th
    On /reloadui it immediately stops on the 65th
    On log out / relog it immediately stops on the 65th
    On exit / relog it starts again failing from the first until it stops on the 65th
    On crash it starts again failing from the first until it stops on the 65th

    Test 1.2: Silly check on how fast is item data retrieval from the server
    Code:
    local function CheckItem(id)
    	local ok, msg = pcall(Inspect.Item.Detail, id)
    	return ok
    end
    
    for i,id in ipairs(ItemTable) do
    	if not CheckItem(id) then 
    		if not CheckItem(id) then 
    			print("Stopped on item #" .. i .. ": " .. id)
    			break 
    		end
    	end
    end
    After clearing the cache it fails on every item type until it stops on the 65th, so item data retrieval from the server isn't that fast :P

    Conclusions on Test 1:
    1.- The 65th item type isn't valid. Update: Confirmed, it's a planar essence.
    2.- As Zorba said, Inspect.Item.Detail fails when called the first time if the item type is not cached, but item data is retrieved and the next time it doesn't fail
    3.- In order to clear the cache, making the client crash is faster than exiting/relogging as you only need to push a button instead of having to retype your password, waiting while the launcher checks for updates, selecting character...

  12. #12
    Plane Touched
    Join Date
    Feb 2012
    Posts
    228

    Default

    Sorry for answering to myself, but trying to write everything in one post gives me a web server error.

    Test 2.1: Precache 500 item types, then see if it stops only on the non-valid item types until the 501, when it should start stopping on every item type
    Code:
    local function CheckItem(id)
    	local ok, msg = pcall(Inspect.Item.Detail, id)
    	return ok
    end
    
    if not continue then 
    	print("Precaching 500 items")
    	continue = 1
    	for i=1,500 do
    		local id = ItemTable[i]
    		CheckItem(id)
    	end
    	print("Precache completed")
    end
    
    for i = continue,#ItemTable do
    	local id = ItemTable[i]
    	if not CheckItem(id) then 
    		print("Stopped on item #" .. i .. ": " .. id)
    		continue = i + 1
    		break 
    	end
    end
    It takes some time to precache, then stops on #2. I thought it would stop on #1, but it seems the precache time has been long enough to receive data from #1
    Then it stops on: 65, 257, 365, 397, 421, 454. These must be non-valid item types. Update: Confirmed, they are planar essences
    After that it stops on: 501, 503, 504, 505, 506, 507, 508, 509, 510, 511, 513, 514, 517, 518, 519, 520...
    Why hasn't it stopped on 502, 512, 515 and 516? Are there repeated item types in the Items.xml file from Discoveries?

    Test 2.2: Are item types 502, 512, 515 and 516 repeated?
    Code:
    local checkIndices = {502, 512, 515, 516}
    for _,checkIndex in ipairs(checkIndices) do
    	local itemType = ItemTable[checkIndex]
    	local repeated = false
    	for i = 1, checkIndex - 1 do
    		if ItemTable[i] == itemType then
    			print("#" .. checkIndex .. " is a repetition of #" .. i)
    	 	 	repeated = true
    			break
    		end
    	end
    	if not repeated then
    	 	print("#" .. checkIndex .. " isn't repeated")
    	end
    end
    None of them is repeated. Then why are they already cached? What are they?
    502: Vanquisher's Amulet of the Mythical
    512: Resolute Carmintium Warhammer
    515: Nimble Wool Waistband
    516: Treachureous Chromite Sabatons
    They don't look like anything I'd have in my inventory, perhaps they're cached because somebody that passed near me has them?
    Update: Gone to the middle of Lake of Solace, cleared cache, repeating Test 2.1... Same result, except this time it actually stopped on #1 the first time
    So, why are they cached?

    Test 2.3: Are Base Item IDs (first part of the item type) of item types 502, 512, 515 and 516 repetitions of earlier item types?
    Code:
    local function split(str, pattern)
    	local tab = {}
    	local val
    	while string.find(str, pattern) do
    		val, str = string.match(str, "(.-)"..pattern.."(.*)")
    	 	table.insert(tab, val)
    	end
    	table.insert(tab, str)
    	return tab
    end
    
    local checkIndices = {502, 512, 515, 516}
    for _,checkIndex in ipairs(checkIndices) do
    	local itemType = ItemTable[checkIndex]
    	local baseItemID = split(itemType, ",")[1]
    	local repeated = false
    	for i = 1, checkIndex - 1 do
    		local otherBaseItemID = split(ItemTable[i], ",")[1]
    		if baseItemID == otherBaseItemID then
    			print("#" .. checkIndex .. " is a partial repetition of #" .. i)
    			print("#" .. checkIndex .. ": " .. itemType)
    			print("#" .. i .. ": " .. ItemTable[i])
    	 	 	repeated = true
    			break
    		end
    	end
    	if not repeated then
    	 	print("#" .. checkIndex .. " isn't partially repeated")
    	end
    end
    #502 is a partial repetition of #166
    #512 is a partial repetition of #183
    #515 is a partial repetition of #318
    #516 is a partial repetition of #6
    So if I cache a Base Item ID, I'd have any other item type with the same Base Item ID cached?

    Test 2.4: Cache Base Item ID of #6, see if #516 is cached too
    Code:
    local function CheckItem(id)
    	local ok, msg = pcall(Inspect.Item.Detail, id)
    	return ok
    end
    
    local checkIndices = {6, 516}
    for _,checkIndex in ipairs(checkIndices) do
    	local itemType = ItemTable[checkIndex]
    	if not CheckItem(itemType) then 
    	 	print("Stopped on item #" .. checkIndex .. ": " .. itemType)
    	 	break 
    	end
    end
    print("Done")
    First run stops on item #6 (as expected)
    Second run doesn't stop, so it seems to confirm that when a Base Item ID is cached, every other item with the same ID is cached too

    Test 2.5: How much could I reduce the cardinality of the huge table with every discovered item by only caching Base Item IDs?
    Code:
    local function getBaseItemID(itemType)
    	local val = string.match(itemType, "(.-),")
    	return val
    end
    
    local reducingDict = {}
    local reducedItemTable = {}
    for _,itemType in ipairs(ItemTable) do
    	local baseItemID = getBaseItemID(itemType)
    	if not reducingDict[baseItemID] then
    		reducingDict[baseItemID] = true
    		table.insert(reducedItemTable, baseItemID)
    	end
    end
    
    print(#ItemTable .. " reduced to " ..#reducedItemTable)
    53067 reduced to 17687
    The item types needed to cache would be reduced to a third of the original table, but would it really work?

    Test 2.6: Would the reduction on Test 2.5 work? This one will be a trial of patience...
    Code:
    local function fixItemTypeAndGetBaseItemID(itemType)
    	itemType = string.gsub(itemType, "FFFFFFFFFFFFFFFF0", "FFFFFFFF0")
    	local val = string.match(itemType, "(.-),")
    	return itemType, val
    end
    
    local function CheckItem(id)
    	local ok, msg = pcall(Inspect.Item.Detail, id)
    	return ok
    end
    
    if not reducedItemTable then
    	local reducingDict = {}
    	reducedItemTable = {}
    	fixedItemTable = {}
    	for _,itemType in ipairs(ItemTable) do
    		local fixedItemType, baseItemID = fixItemTypeAndGetBaseItemID(itemType)
    		table.insert(fixedItemTable, fixedItemType)
    		if not reducingDict[baseItemID] then
    			reducingDict[baseItemID] = true
    			table.insert(reducedItemTable, fixedItemType)
    		end
    	end
    	print("Item table reduced to " .. #reducedItemTable)
    end
    
    if not lastItemIndex then
    	lastItemIndex = 1
    end
    
    if lastItemIndex <= #reducedItemTable then
    	local count = 0
    	repeat
    		CheckItem(reducedItemTable[lastItemIndex])
    		lastItemIndex = lastItemIndex + 1
    		count = count + 1
    	until count >= 500 or lastItemIndex > #reducedItemTable
    	if lastItemIndex <= #reducedItemTable then
    		local clicksLeft = math.ceil((#reducedItemTable - lastItemIndex) / 500)
    		print("Precaching will be completed with " .. clicksLeft .. " more clicks")
    	else
    		print("Precaching completed, ready to crash!")
    	end
    elseif not continue then
    	local startTime = Inspect.Time.Real()
    	Inspect.Item.Detail(fixedItemTable)
    	local endTime = Inspect.Time.Real()
    	print("Congratulations! You've succesfully cached the full list of known items (" .. #fixedItemTable .. ") and survived!")
    	print("Time spent on retrieving the details of every known item: " .. (math.floor((endTime - startTime) * 100) / 100) .. " seconds.")
    	continue = 1
    	print("Now, keep clicking to check which item types are broken...")
    else
    	local stopped = false
    	for i = continue,#fixedItemTable do
    		local id = fixedItemTable[i]
    		if not CheckItem(id) then 
    			print("Stopped on item #" .. i .. ": " .. id)
    			continue = i + 1
    			stopped = true
    			break 
    		end
    	end
    	if continue > #fixedItemTable then continue = 1 end
    	if not stopped then print("Every item type works :S") end
    end
    After about half an hour clicking and clicking the run button... Every discovered item type works and you can cache all of them!

    Conclusions on Test 2:
    The fact that caching a Base Item Type ID allows you to expect a reliable result for every item type that shares that first part is a good new:

    We can, on Addon load, start a coroutine that silently caches every item type stored in the addon database. Reducing the number of calls to the Inspect.Item.Detail needed to one per different Base Item Type ID will allow us to reduce the time needed to fill the cache.

    As the Addon I'd like to develop is for the Auction House, the cardinality of the its database will surely be much lower than the full list scanned on the last test and, due to the fact that most items you'd use on it will be already cached due to you having them on your inventory or having just seen them in an auction scan, it shouldn't take too long to fill the cache.

    This will allow me to reduce the size of the saved variables file by removing any field I can ask to Inspect.Item.Detail whenever I need it.



    I won't do further tests on item types as I already have everything I need: a criteria for grouping similar items, and how to reliably call Inspect.Item.Detail.

    As stack size for auction items will be avalaible soon, it's time to focus on developing the addon.

    Thank you for all your help, Zorba.
    Last edited by Baanano; 02-16-2012 at 05:55 PM.

  13. #13
    Champion
    Join Date
    Jun 2011
    Posts
    561

    Default

    Quote Originally Posted by ZorbaTHut View Post
    All that said, if it turns out that we're burning a lot of bandwidth on item requests, we may have to throttle it more severely. We recommend pulling item data down only when you actually need it - if you're making an addon that lets you, say, search over all items, include a local database with your addon, don't pull it all down from the server for every single user
    Fully understandable approach. However if every addon would use it's own individual database wouldn't that mean that the memory usage would explode client side? Or would you say that a couple of addons each using X MB of memory for it's database isn't a big deal?

    In case addon memory usage would eventually be a potential client side problem it would make sense for addon authors to combine forces and build a common library.

    Cheers
    N.

  14.   Click here to go to the next Rift Team post in this thread.   #14
    Rift Team
    Join Date
    Oct 2010
    Posts
    927

    Default

    Quote Originally Posted by Naifu View Post
    Fully understandable approach. However if every addon would use it's own individual database wouldn't that mean that the memory usage would explode client side? Or would you say that a couple of addons each using X MB of memory for it's database isn't a big deal?
    Rift frequently uses over a gigabyte, a few extra megs isn't likely to be a big problem. A shared database would certainly be preferable, however.

    If people start regularly making huge clientside databases then I've got a few ideas to help cut down on memory usage, but I'm probably not gonna get around to implementing them unless it's clear they're needed.

  15. #15
    RIFT Community Ambassador the_real_seebs's Avatar
    Join Date
    Jan 2011
    Posts
    16,859

    Default

    I used to have >300MB of active memory for addons in WoW.

    It had only a marginal performance impact.
    You can play WoW in any MMO. You don't have to play WoW in RIFT. Oh, and no, RIFT is not a WoW clone. Not having fun any more? Learn to play, noob! I don't speak for Riftui, but I moderate stuff there. Just came back? Welcome back! Here's what's changed. (Updated for 2.5!)

+ Reply to Thread
Page 1 of 2 1 2 LastLast

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts