Morrowind modding for smarties, part 14, Horror Show: When Masters and Identifiers change
PREREQUISITES:
 
1. you know/care about what problems may happen trying to update mods having TR_Mainland.esm as master to the new version, with TR changing identifiers, and you are looking for a way to fix problems without redoing a big part of the dependant mod(s) changes to references (e.g. moved/added/deleted windows, rocks, plants...) in Construction Set
2. you know how to use Mash updaters (see also here) from a Mash 84 source version/with installed Python 2.7x setup.
 
In this case, you may be interested on how to make mash updaters compatible with TR changed identifiers, as changed identifiers are something standard Mash updaters code can't handle.
 
It is a complex procedure, but if you are going to have a lot of mods with a lot of references broken, IMO still better try this than fixing things in Construction Set.
 
First, be sure you have File\Lock Times ON in Mash, it is important to keep your mod files timestamps (= mod loading order) through the changes.

Make a backup copy of Mopy\mosh.py somewhere (or, better, make a backup of your entire Morrowind folder somewhere).
 
Change mosh.py this way (or get it from here):
 
     def getObjectMaps(self,toMod):
        """Return a dictionary of ObjectMaps with fromMod key for toMod."""
        if self.objectMaps == None: self.loadObjectMaps()
        subset = {}
#begin abot +++ for TR identifiers change
        fromOldIdToNewIdFile = 'Translation.txt' # TR id changes file
        fileName = "..\\Data Files\\" + fromOldIdToNewIdFile  # look for it in Data Files first
        remap = os.path.exists(fileName)
        if not remap:
            fileName = "..\\" + fromOldIdToNewIdFile # then in Morrowind main folder
            remap = os.path.exists(fileName)
        if remap:
            try:
                fromIdToId = {}
                with open(fileName, 'r') as f:
                    for line in f:
                        if not line.startswith('#'): # skip comment lines
                            vk = line.strip().split(':')
                            if len(vk) == 2: # verify we got a couple
                                k, v = vk[0],vk[1]
                                fromIdToId[k] = v
                if f: f.close()
            except EOFError:
                remap = false
                if f: f.close()
#end abot +++
        for key in self.objectMaps.keys():
            if key[1] == toMod:
                subset[key[0]] = self.objectMaps[key] # standard Mash identifier remapping
#begin abot +++ for TR identifiers change
                if remap:
                    if key[0] in fromIdToId:
                        subset[fromIdToId[key[0]]] = self.objectMaps[key] # try mapping also the new changed identifier
#end abot +++
        return subset
 
Basically, these changes try to use both the Translation.txt file provided by TR release and the generated Tamriel_DatN.esm (reference-compatible with the new Tamriel_Data.esm, but including also objects scraped in the new version but still present in the old version) to allow Mash updaters procedure to remap references correctly even for objects discarded/replaced in new Tamriel_Data.esm version.
So be sure to have latest released (by me if available or by TR) Translation.txt in your Data Files folder so tweaked Mash can use information from there.

I have made a tes3cmd script package to generate a Tamriel_DatN.esm you can use as starting point for your Mash updaters procedure.
 
To use the tool you need:
old versions of TR masters (the Tamriel_Data.esm, TR_Mainland.esm versions you are currently using in your loading list before updating)
new/just downloaded versions of TR masters RENAMED AS Tamriel_DataNew.esm, TR_MainlanN.esm
 
once you have the package files in your Data Files folder, double clicking Tamriel_DatNgen.bat should generate your Tamriel_DatN.esm file for mash updaters/compatible with previous release
(you can review contents of Tamriel_DatN.esm in the out.txt file)
 
When Tamriel_DatN.esm has been generated Tamriel_DataNew.esm should be no more needed.

From Mash, copy the timestamp (3rd field in top right column) from the old Tamriel_Data.esm to the new Tamriel_DatN.esm and save, so they have the same timestamp, it will make things easier to find.

From Mash, right click the new Tamriel_DatN.esm, updaters, clear the updaters window if not empty, add the old Tamriel_Data.esm. This is the base updater you can use to change mods/saves to the new Tamriel_DatN.esm
 
Second step is the updater for TR_Mainland.
Get the new/just released version named as e.g. TR_MainlanN.esm,
right click it in Mash and change its master in the right colum masters list from Tamriel_Data.esm to Tamriel_DatN.esm.

Right click the new TR_MainlanN.esm, updaters, clear the updaters window if not empty, add the old TR_Mainland.esm.

From Mash, copy the timestamp (3rd field in top right column) from the old TR_Mainland.esm to the new TR_MainlanN.esm and save, so they have the same timestamp, it will make things easier to find.

You should do this for ALL other mods in your loading list having Tamriel_Data.esm as master, so you need a way to easily find/list them e.g. to look for *.es? files containing the Tamriel_Data.esm string.

It is simple to do it e.g. from a file manager like Total Commander (shareware works)/Double Commander or from a search tool like GrepWin).
Alternatively (or if you want to restrict the search only to mods used in your current loading list) you could use latest tes3cmd alpha version (or my update) providing the --active parameter, with a command line e.g. like this:
tes3cmd.exe dump --active --type TES3 --match "Tamriel_Data.esm" --list > out.txt
to output the list of loaded mods having Tamriel_Data.esm as master to out.txt text file.
As an example, these are the mods having Tamriel_Data.esm as master in my current loading list:
ArynxTR1809.esp
HousepodPackSO1809.esp
JoinAllHouses+ClansTR1809.esp
MCA - TR Addon1809.esp
Open MournholdTR.esp
PC_Stirk_MN.esp
Silgrad TowerTRFM.esp
Sky_Main1903MN.esp
TR_Mainland.esm
TR_Preview.esp
TR_Travels_(Preview_and_Mainland).esp
TR_signs+fixes1809.esp
abotBoatsTR1809.esp
abotGuardsTR1809.esp
abotRiverStridersTR1809.esp
abotSiltStridersTR1809+scenic.esp
abotTRWaterSound1809.esp
abotWaterLifeTRaddon1809.esp
abotWindowsGlowTR1809.esp

From this list, we can skip TR_Mainland.esm, TR_Preview.esp, TR_Travels_(Preview_and_Mainland).esp  because we will be using the new released versions

If the mod you are changing has also TR_Mainland.esm as master, also change it from TR_Mainland.esm to TR_MainlanN.esm, and so on...

so, if you have e.g. a abotSiltStridersTR1809.esp mod, you should:

  1. right click it from Mash, duplicate as e.g. abotSiltStridersTR1912.esp
  2. click the new abotSiltStridersTR1912.esp, change its masters from old Tamriel_Data.esm, TR_Mainland.esm to new Tamriel_DatN.esm, TR_MainlanN.esm, press Save button
    3.  

You should do this for ALL mods in your loading list having Tamriel_Data.esm as master

AFTER the masters have been changed using Mash updaters, you can use the TR_FilePatcher_v2.jar tool if needed (e.g. if you get many object not found error due to changed identifiers), just remember to remove the dependency from Tamriel_Data.esm/TR_Mainland.esm that the tool is re-injecting.
I suggest to use it only if needed/you get error messages while playing because it may be difficult to run, it is very slow, and it will add back dependency on Tamriel_Data.esm.

 

After having this done with all mods depending on TR masters, you should do the same with the save(s) you want to keep on playing.

so, if you have a saved game  to update, you should change from Mash ALL the save masters from e.g.
Tamriel_Data.esm to Tamriel_DatN.esm,
TR_Mainland.esm to TR_MainlanN.esm,
abotSiltStridersTR1809.esp to abotSiltStridersTR1912.esp
... and so on for every updated master
Remember to press the save button after you have changed all the masters to apply the update

These are more or less the needed steps to be able to keep playing correctly from previous saves without references deleted/misassigned.

I suggest you try the process from a backup Morrowind installation folder until you are confident with all this (or make a backup of your Morrowind folder you can easily restore).

There may be other conflicts/changes unrelated to references that still need fixing/patching obviously (e.g. a mod adding practice dummies/archery targets to a TR cell which has changed name in new TR version, the new cell name used from GetPCCell script command, old mod is forcing the old name on load, result=crash) , but that can't really be done automatically as possible mod lists combinations are great, you will have to fix them the usual way e.g. changing the cell name in the conflicting mod.

Note:  I know I'm writing this mostly for myself, hard to remember steps between TR releases

Note2: wow, you read everything? you deserve this
But wait, we still have a bunch of mods and latest saves depending on weird-named Tamriel_DatN.esm, TR_MainlanN.esm. Albeit not necessary to play, it would be nice to have them to use standard names again after being updated.

That's why we used weird names, because we changed a single letter from the standard name, keeping the same name lenght, so it is much easier to use any hexadecimal editor to directly replace that letter in any of the files e.g from Tamriel_DatN.esm back to Tamriel_Data.esm.
You can do it both for mods and the few save files fast using something like HxD editor e.g. for mods opening all *1912*.es? files and replacing Tamriel_DatN.esm string with Tamriel_Data.esm string, TR_MainlanN.esm string with TR_Mainland.esm string.
You can replace the master names also from Mash if you prefer, but using the hex editor is usually faster.

Once done with all the replacement in mods and edited saves, we can just copy Tamriel_DatN.esm over Tamriel_Data.esm, and TR_MainlanN.esm over TR_Mainland.esm

Then it would probably be a good time to run tes3cmd multipatch, and Mash\Repair all on your few updated saves.
Hopefully Mash "Repair All" will find several bad references in the save, but very few in .esp/.esp mods.
In case of mod problems, go to the mods tab, right click the mod with bad references and do  "Repair Refs"
(you will probably better check the mod in CS and fix these missing references in this case e.g. if the mod was replacing a tree, that tree change may be no more there).

Another thing you could do at this point is to edit the rules in your mlox\mlox_user.txt
so they use the new mod names, e.g. replace the string "1809"  with "1912",
then you can run MLOX and see if it makes sense to change something in the loading order.