Page 1 of 2 12 LastLast
Results 1 to 20 of 31

Thread: J.E. Sawyer on FO3/NV save game-related slowdowns

  1. #1
    But best title ever! Vault Fossil
    WorstUsernameEver's Avatar
    Join Date
    May 2010
    Posts
    1,564
    Total Downloaded
    0

    J.E. Sawyer on FO3/NV save game-related slowdowns

    Should you have played a game from Bethesda this generation you'll probably be familiar with slowdowns related to the increasing size of save-game files, a problem which wasn't fixed even in the recently published Skyrim. Since the problem was present in New Vegas too, project director J.E. Sawyer from Obsidian took some time to answer to some questions about it on his Formspring account, and explained why the bug couldn't be fixed in a patch:
    Question from one of lesser knowledge... Is there a save game size that you would call "normal" for F:NV? Mine is 14mb, and i'm having intense lag in multiple sections of the Mojave/ all DLC lands. Just curious if that is/could be an issue!

    That can easily be a big problem, especially if you're on the PS3. The longer you play a character, the more bit differences on objects (characters, pencils on tables, containers, etc.) get saved off and carried around in memory. I think we've seen save games that are pushing 19 megs, which can be really crippling in some areas.

    ...

    If it's an issue why hasn't it been resolved? The pace at which game sellers basically dish an alpha or beta game to the marketplace is sickening. New V stopped being "beta" in mid-2011. I know as a dev my workplace wasn't AAA dishing but we had integrity.

    Since you're a developer, you should understand the implication of what I wrote. It's an engine-level issue with how the save game data is stored off as bit flag differences compared to the placed instances in the main .esm + DLC .esms. As the game modifies any placed instance of an object, those changes are stored off into what is essentially another .esm. When you load the save game, you're loading all of those differences into resident memory.

    It's not like someone wrote a function and put a decimal point in the wrong place or declared something as a float when it should have been an int. We're talking about how the engine fundamentally saves off and references data at run time. Restructuring how that works would require a large time commitment. Obsidian also only had that engine for a total of 18 months prior to F:NV being released, which is a relatively short time to understand all of the details of how the technology works.

    ...

    Hey Josh, different user, but same kind of 'lesser knowledge' bloke. What's a bit difference and how does it determine a save file?

    http://en.wikipedia.org/wiki/Bit_field

    It's a compact way to store data. The bit differences in this case are just flags set up to mark what data has changed (i.e., are different) from what's in the core .esm.

    Let's say that I, as a designer, set up a creature in an area. I set all of the character's statistics and gear and save it in the master FalloutNV.esm file that gets loaded into the game. You, the player, run through the area and shoot that dude. You loot him of his gear and put a shovel in his inventory because you are wacky.

    The game needs a way to mark that his a) position B) health c) inventory d) some other stuff has changed on him. It does that by marking what fields have changed (by setting individual bits) and then indexing the individual (changed) values for reference later.

    When you load the save game, it loads up all of the bit fields marking changes in your save game. When the individual objects load, it applies the indexed changes to those objects. That way, when you come back to the area you left two nights ago, the character is still sprawled out where you left him, naked, with a shovel in his inventory.

    Individual bits of data are tiny, but there are thousands upon thousands of objects in F:NV, each one containing numerous data fields that could potentially be changed in your save game. Over time, it adds up.

    ...

    Is the inflating save file just an issue for the PS3 (I've seen lots of lag/crash complaints from PS3 users) or does it happen on all platforms? I'm just wondering if other platforms handle it better than the PS3.

    As with Fallout 3 and Skyrim, the problems are most pronounced on the PS3 because the PS3 has a divided memory pool.

    ...

    (Same fellow you just answered) So....basically, every time I manipulate an object, it fluctuates the save file up or down? I've noticed files can get rather huge, is there no way this could have been greatly diminished in a game as big as New Vegas?

    It almost always goes up. Some areas will reset contents after three (game) days, but a lot of stuff lingers. Additionally, we also have to deal with "persistent references". These are objects that are immediately loaded with the game because we need to be able to reference them anywhere/everywhere in the world -- even if the player is nowhere near the object. Characters are the most common example. All of the companions need to be able to move around the world even when they are not in your current area, so they are all persistent references.

    All object data (excluding art assets like .nifs and audio assets [VO]) for persistent references is loaded at all times, so that's more-or-less a permanent chunk of resident memory. The number of persistent references invariably goes up with each DLC, so as the number of DLCs increases, the system has less and less memory available. Of course, the player's save game file only gets bigger and bigger, since he or she is going through more or more areas manipulating an increasingly large number of objects.

    This is why some of our later patches actually removed content from the core game (e.g. Primm). Even though we had balanced the memory footprint for the core game, DLC content was pushing down the available resources.

    ...

    "divided memory pool"?

    The Xbox 360 has a unified memory pool: 512 megs of RAM usable as system memory or graphics memory. The PS3 has a divided memory pool: 256 megs for system, 256 for graphics. It's the same total amount of memory, but not as flexible for a developer to make use of.

    ...

    But why did the patch remove content from the PC? Most PCs nowadays have 2-4GB of RAM, plus 500MB-1GB of GPU memory. So I doubt the DLCs negatively impacted most PCs to justify content removal...

    If we had generated .esms per-platform, that would have been a crazy nightmare for a lot of reasons. A slightly less risky approach could have been to script the removal of assets using the IsPC/IsXbox/IsPS3 functions, but that also introduces its own host of potential problems, especially if objects are attempting to reference something as the script removes it.

    We ran into a small but non-trivial number of crashes in F:NV involving persistent references attempting to interact with an object as the player transitions out of his or her current area. E.g. Chief Hanlon attempts to sit in a chair. The player leaves the area, the chair Hanlon wants to sit in is unloaded, and the game crashes.

    ...

    Hypothetically, if I were to play through FNV without opening any non-necessary containers (e.g. never opening an ammo case) or touching any non-necessary objects (e.g. never moving a pencil) would the game be more stable?

    By some margin, yes, but randomized loot I believe is generated on the area's first load, which also applies to the equipment in many characters' (e.g. Fiends') inventories.
    Thanks GameBanshee.
    YOU WANT DEAL?

  2. #2
    I should set a custom title Vault Senior Citizen

    Join Date
    Apr 2003
    Posts
    1,059
    Total Downloaded
    0

    Techno Cul De Sac Retorical Amusment

    Techno Cul De Sac Rhetorical Amusement



    Quote Originally Posted by J.E. Sawyer
    ...
    If we had generated .esms per-platform, that would have been a crazy nightmare for a lot of reasons. A slightly less risky approach could have been to script the removal of assets using the IsPC/IsXbox/IsPS3 functions, but that also introduces its own host of potential problems, especially if objects are attempting to reference something as the script removes it.
    ...
    What other games are hitting this memory allocation ceiling?

    Will the short term answer mean less interactive STUFF, (less environmental targets, less EXPLODING HEADS), could this start an interactive Stuff techno war between AAA title pushers?
    Could collecting pencils or exploding heads became console achievements?
    How many collected rocks in the FO's?

    So consoles don't have the TECHNOLOGY for grabbing all the present day video game FAT LOOT without maxing out the hardware / software?

    Could this trauma of save game size truncating the cornacopia of FAT LOOT, and or crash, resonate into the next marketing frenzy for console hardware?

    Imagine, PC players fingering the console half gig of ram choke point as " ' this is why we can't have nice things! ' "

    Which console purveyor will be the first to close the FAT LOOT lack gap?




    4too

  3. #3
    I should set a custom title Sonny, I Watched the Vault Bein' Built!

    Join Date
    Feb 2008
    Posts
    2,113
    Total Downloaded
    0
    It looks like filling your game with junk not only negatively impacts gameplay but the stability and efficiency of the game. If the item doesn't have any purpose then don't allow the player to interact with it.

    "Pessimism is, in brief, playing the sure game. You cannot lose at it; you may gain. It is the only view of life in which you can never be disappointed. Having reckoned what to do in the worst possible circumstances, when better arise, as they may, life becomes child's play."
    ~Thomas Hardy

    "Idealism is what precedes experience; cynicism is what follows."
    ~David T. Wolf

  4. #4
    I should set a custom title It Wandered In From the Wastes

    Join Date
    Oct 2010
    Posts
    159
    Total Downloaded
    0
    If the item doesn't have any purpose then don't allow the player to interact with it.
    Tell that shit to the Deus Ex fans.

  5. #5
    A right proper chap. Sonny, I Watched the Vault Bein' Built!
    Alphadrop's Avatar
    Join Date
    Aug 2008
    Location
    Blighty
    Posts
    2,402
    Total Downloaded
    0
    Quote Originally Posted by gumbarrel
    If the item doesn't have any purpose then don't allow the player to interact with it.
    Tell that shit to the Deus Ex fans.
    Deus Ex can get away with it though as it has a smaller amount of areas and less loot.

    Rage is a modern example where pretty much nothing apart from actors and vehicles moves and those are loaded with the area and scripts rather than kept as a reference (seen by how everything respawns when you change area) which is why the game runs so smooth on consoles even if it was criticised that you couldn't knock over a can.

    256 (well sort of) is a horribly low amount of ram to try something as big as a persistant freeroam game on, no wonder the ps3 versions always have it bad.
    ''You have the right to remain inconspicuous."

  6. #6
    I should set a custom title Vault Senior Citizen

    Join Date
    Apr 2003
    Posts
    1,059
    Total Downloaded
    0

    Pardon My Mime

    Pardon My Mime




    Quote Originally Posted by UncannyGarlic
    ...If the item doesn't have any purpose then don't allow the player to interact with it.




    The prevailing business model for marketing to the consumer culture requires ...








    4too

  7. #7
    Happy New Year! Vault Senior Citizen
    Richwizard's Avatar
    Join Date
    Jan 2010
    Location
    The Old West
    Posts
    1,314
    Total Downloaded
    0
    Sawyer's answers make sense. I have 6GB of RAM in my PC, but my video card only has 768mb. I do have an occasional graphics glitch because of this and the fact that my graphics settings are maxed out. As the game progresses and the save files get bigger, it can get worse. The simple solution to this problem is to have enough memory to process the files efficiently. On a console, this is more of a problem.

    @Sabirah - Don't feed the trolls.


  8. #8
    Still kind of new First time out of the vault

    Join Date
    Dec 2011
    Posts
    10
    Total Downloaded
    0
    let us get this straighted out one more time.

    the game that ships with the vanilla esps/esms will have fixed positions you want ot keep for when the user wants to start a new game.

    but for the save file we are still looking at a 3d vector, 3 floats for every object static or animated static or kick around /drag debris.

    having less items that can be moved around does mean less needs ot technically be saved. but that isn't a good excuse for why the game would be slow as you are playing it. you save the most current position regardless of where it is in its animation. only things acted upon in masses causes potential slow downs in RT.

    the game is not littered enough to cause size issues in the case of load/save or speed issues in the case of just walking around and shit.

  9. #9
    Super Moderator Lived Through the Heat Death
    Lexx's Avatar
    Join Date
    Apr 2005
    Location
    Germany
    Posts
    8,457
    Total Downloaded
    0
    Well, Sawyer never said that it's the only problem, hm? I don't get why you are jumping so up and down on it. Can't be changed now anyway.

  10. #10
    I should set a custom title It Wandered In From the Wastes

    Join Date
    Feb 2010
    Posts
    141
    Total Downloaded
    0
    Quote Originally Posted by Richwizard
    Sawyer's answers make sense. I have 6GB of RAM in my PC, but my video card only has 768mb. I do have an occasional graphics glitch because of this and the fact that my graphics settings are maxed out. As the game progresses and the save files get bigger, it can get worse. The simple solution to this problem is to have enough memory to process the files efficiently. On a console, this is more of a problem.

    @Sabirah - Don't feed the trolls.
    Supposedly the engine's really retarded about how it handles memory in general though, hence the need for the LAA Executables.

    Also IIRC the engine is missing some pretty basic features that were finally added in Skyrim (occlusion culling, for instance) which puts way more pressure on the game. I might be wrong about this. But I'm pretty sure all the LOD Terrain is loaded all the time.

  11. #11
    I should set a custom title Still Mildly Glowing

    Join Date
    Oct 2008
    Posts
    246
    Total Downloaded
    0
    Well the original Fallouts had savegames that would get huge (humongous!) I remember loading times of an eternity...

    Of course, as old as the engine was at least it didn't move objects around (sometimes actually flying around) every step you took through the gameworld, and save their new positions.

  12. #12
    Administrator Brother None's Avatar
    Join Date
    Apr 2003
    Location
    Leiden, the Netherlands
    Posts
    37,358
    Total Downloaded
    0
    I used to jump on top of tables and run around knocking tableware around, thinking it funny. Now I realize I was breaking the game. I'm so sorry, game!

  13. #13
    I should set a custom title Sonny, I Watched the Vault Bein' Built!
    TheWesDude's Avatar
    Join Date
    Feb 2005
    Posts
    2,078
    Total Downloaded
    0
    and finally the truth comes out.


    beth really sucks at making games and never "pushed the envelope" of technology.


    their engine really limited what they could do. not to mention that consoles are the biggest problem.


    and seriously sony... 256 megs of ram???? wtf is up with that.
    Trivia Dragon
    -==(UDIC)==-

    [MYTH]Murzin

  14. #14
    I should set a custom title Vault Fossil
    Stanislao Moulinsky's Avatar
    Join Date
    Jul 2009
    Posts
    1,596
    Total Downloaded
    0
    Quote Originally Posted by NVExpert
    let us get this straighted out one more time.

    the game that ships with the vanilla esps/esms will have fixed positions you want ot keep for when the user wants to start a new game.

    but for the save file we are still looking at a 3d vector, 3 floats for every object static or animated static or kick around /drag debris.

    having less items that can be moved around does mean less needs ot technically be saved. but that isn't a good excuse for why the game would be slow as you are playing it. you save the most current position regardless of where it is in its animation. only things acted upon in masses causes potential slow downs in RT.

    the game is not littered enough to cause size issues in the case of load/save or speed issues in the case of just walking around and shit.
    Low RAM and shitty costant streaming from the HD from what I gathered. Apparently running the game with a SSD reduces dramatically stuttering.
    I don't know the key to success, but the key to failure is trying to please everybody.
    Bill Cosby

  15. #15
    Propane custodian Vault Fossil
    valcik's Avatar
    Join Date
    Dec 2008
    Location
    Slovakia
    Posts
    1,675
    Total Downloaded
    0
    Quote Originally Posted by TheWesDude
    beth really sucks at making games and never "pushed the envelope" of technology.
    Yup. Mr. Howard has deserved a Master Recyclator achievement.

  16. #16
    I should set a custom title Vault Senior Citizen

    Join Date
    Jun 2007
    Posts
    1,230
    Total Downloaded
    0
    That can easily be a big problem, especially if you're on the PS3. The longer you play a character, the more bit differences on objects (characters, pencils on tables, containers, etc.) get saved off and carried around in memory. I think we've seen save games that are pushing 19 megs, which can be really crippling in some areas.
    How amusing, doesn't/didn't skyrim have the same problem on ps3, too?
    Weird, considering that it's a brand new engine!
    True, fusion cells shouldn't just explode, but then again, I don't really care. Exploding cars are fun.
    FO3 target audience is FUN!

  17. #17
    But best title ever! Vault Fossil
    WorstUsernameEver's Avatar
    Join Date
    May 2010
    Posts
    1,564
    Total Downloaded
    0
    Pete Hines denied that the Skyrim issue is the same on Twitter:
    @NotSkyrSpoilers Josh Sawyer did not work on Skyrim nor this engine and his comments don't reflect how the current tech works.

    ...

    @Elkirak No, it isn't true. He brings up issues we solved long ago.
    Not flattering that they basically have a similar issue in their all-new engine though (and I'm giving him the benefit of doubt and assuming he's saying the truth and that in Skyrim the issue is a different one and not a legacy of Morrowind/Oblivion/Fallout 3/Fallout: New Vegas).
    YOU WANT DEAL?

  18. #18
    I should set a custom title Sonny, I Watched the Vault Bein' Built!

    Join Date
    Dec 2008
    Location
    Video game land
    Posts
    2,105
    Total Downloaded
    0
    I wouldn't be surprised if it turned out he was lying, knowing Bethesda's reputation with telling the truth. The problem sounds very similar to the one people are getting in Skyrim.

    Quote Originally Posted by Ausir
    The Enclave should ditch their power armor project and just make armor out of hillbilly hides.

  19. #19
    D'yusooportdenoooos? Lived Through the Heat Death
    Walpknut's Avatar
    Join Date
    Dec 2010
    Posts
    8,409
    Total Downloaded
    0
    So it seems getting one of those "Clean up the wasteland" mods on your game is a very good idea, all that junk not only is it useless it apparently also hurts perfomance.

    Quote Originally Posted by Kilus View Post
    Quote Originally Posted by whirlingdervish View Post
    fighting raiders recap: literally stabbing a guy in the butt 5 times without him noticing. RADIANT(TM)
    Maybe he was enjoying that.

  20. #20
    A right proper chap. Sonny, I Watched the Vault Bein' Built!
    Alphadrop's Avatar
    Join Date
    Aug 2008
    Location
    Blighty
    Posts
    2,402
    Total Downloaded
    0
    Well apart from the problems mostly being on consoles, PC's have enough RAM to get away with it.
    ''You have the right to remain inconspicuous."

Page 1 of 2 12 LastLast

Bookmarks

Posting Permissions