Ships magazines error

[donaldseadog]

One more random, cleared magazine of scarab A TEL and added 12.9mm mg, in GE get the old 127mm but slightly dofferent, looking in csv the nearest occurrence is about 16 slots away.

 

[donaldseadog]

Import seems to import in to PE the incorrect mag items that are seen in GE, so I guess it is working but a bit funny as I wonder if there are 'extra' items: checking against the last about 10 classes with magazine content the csv list seems to progress to the last class (Scrarab A) at item #551, the final ~ 30 items seem misplaced or redundant?

Is it worth going thru the entire 0 - 581 items checking agains the classes in order to see if they fit some where?

[broncepulido]

Incredible work and incredible outcome!!! (no pattern of the errors, as I did suspect!).

Thanks Donald.

[TonyE]

Thanks much Don, I'm hoping to finally take a good look this weekend.

[donaldseadog]

4 hours ago, broncepulido said:

Incredible work and incredible outcome!!! (no pattern of the errors, as I did suspect!).

Thanks Donald.

It's been a great opportunity for me to get back to using the pe, I've mostly ignored it for 10 years or so. And I've seen some wish list items too 😁

I did have a thought while cleaning my teeth last night about lack of obvious pattern and might have another spin tonight if TE doesn't solve the riddle before hand. I'm thinking maybe that by using mostly the first ship class (to make the process of testing easy) I disturbed the pattern and I have to try ship classes further down the class id list.

[donaldseadog]

16 minutes ago, donaldseadog said:

It's been a great opportunity for me to get back to using the pe, I've mostly ignored it for 10 years or so. And I've seen some wish list items too 😁

I did have a thought while cleaning my teeth last night about lack of obvious pattern and might have another spin tonight if TE doesn't solve the riddle before hand. I'm thinking maybe that by using mostly the first ship class (to make the process of testing easy) I disturbed the pattern and I have to try ship classes further down the class id list.

And infact looking at my screen shots, where I used ship class down the list both error items were same as 20 magazine items before, and in the case of all the jfk as it is at top if list it kept looking at first listed mag item.

So there might be the pattern, a shift of 20 (but 20 is a funny number?)

I'm off to sydney today and my wife drives the first half so I might play with this on my laptop (which can be used in tablet mode, very cool).

[donaldseadog]

If anyone else wants to check my guess that the magic number is 20, you need to install my latest toolbox or lazgui-extn to look at unit magazines, you need a simple test scenario with maybe a few different ship class units, strip out all magazine items, add one item to each magazine and give it 'silly' quantity and\or reload values. Export db.

Run game with no db specified (so it uses the exported one) and look at magazine content.

In pe use search feature to find weapon id for the mag items and then look in the csv file (see TEs early post for name but look at the magEx.csv file) to see if it appears 20 items before the item with the silly values.

Does that all make sense?

HINT use a small db, the early CW I used exports from pe in about 3-5 secs, the latest HCDB2 for me takes it seems like a couple of minutes.

 

 

 

 

 

 

[donaldseadog]

(In car with crew control in operation)

Here's a very interesting one, "Unknown Weapon" means (I think, I'd need to check the dll code to be sure) a weapon ID number not in the loaded database?

[TonyE]

I didn't get much time into this tonight.  

I am only doing Exports from the PE.

1. Exported DB, copied annex_a1.csv and annex_magsEx.csv to a separate location for later comparison as annex_a1_Export_01.csv and annex_magsEx_Export_01.csv.

2. Exported DB again without changing anything in the PE.  Copied annex_a1 and mags again this time as Export_02.

3. Compared the csv files, no changes.  So this at least means the export isn't totally random.

4. Added one item to the first ship's magazine (an ordnance pallet to the end of the magazine list entries for ship id 2048 wielingen.

5. Exported DB, copied a1 and mags to Export_03.

6. Compared the 02 files to the 03 files, the renumbering in a1 makes total sense, magazine inserted and magazine numbers adjusted.  magsEx did not make sense, ordering seems all messed up.

 

[donaldseadog]

Along those lines, I imported a new (simple) database, then exported without change.

It appears that the annex_mags.csv file is what is imported, and annex_magsEx.csv is what is exported, so I compared the two after the first export (and saved the annex_mags.csv as annex_mags_0.csv.

Comparing the two I can see the mixed up entries, I looked thru the PE and the mix ups seem to all be within a single platform, ie each platform still has the same items but in a different order.

I then did a second import/ export cycle and the two files seemed as they had been after the first export.

 

[donaldseadog]

Next I tried a small edit to the magazine, I did this twice, both to the 6th platform, once to the last mag item once to the first mag item. (I didn't look in the game to see the result). Both times the change showed up in the export csv but when I imported again it was back to the original mag item.

The screen shot shows (left to right) the last export, the first export, the last import (done after the last export), the first import. The selected lines are the item that was edited in PE.

Late now but I'll write up a test scenario for this DB and see what is in commondb.res.

(EDIT) OK, I wrote a test scenario to look at the magazine in toolbox unit report, there was no change to the magazine (Explaining the import 'resetting' the data).

Edited November 8, 2023 by donaldseadog
update with more info

[TonyE]

Amazing work on this issue by many but especially @donaldseadog!  

From what we've all learned, last night I blew off my normal responsibilities and applied some brain power to the PE.  

I made two modifications to the PE which are testing well for donaldseadog and myself.  He tested more thoroughly than me.

1. PE exports to annex_mags.csv again instead of the errant annex_magsEx.csv file.  I introduced the bug when switching from MS Access Export Specifications to VBA code to create the .csv files.  That was driven by data size limitations and more importantly a Microsoft patch that broke export specifications.

2. PE sorts magazines differently when re-numbering them.  This is what is intended to ensure the ordering of magazines doesn't go awry.  The sorting is now by Annex, then tPlatformMagazine.ID, then tPlatformMagazine.SortID (SortID was previously not part of this qOMagazineRenumberList ORDER BY clause).

If nothing nasty shows up in additional testing, I'll release it with the next build.  The bummer is that this won't fix existing corruption of magazines, it only prevents new corruption on future exports.

[TonyE]

The annex_magsEx.csv bug was introduced in 2022.009 so it is possible databases/pfData files that haven't had an import since then don't have the corruption.  I say 'possible' since the sorting issue has existed since the PE was created.

HCE - 2022.009 - 2022-07-29
===========================
- Chg:00000 PE No longer uses the MS Access Import and Export Specifications,
             imports and exports are now done with file reader & writer and
             DAO.recordset.  This works around Microsoft breaking the import
             and export spec functionality.  As a side effect, it makes it
             easier to modify the structure of the database.

[broncepulido]

48 minutes ago, TonyE said:

Amazing work on this issue by many but especially @donaldseadog!  

From what we've all learned, last night I blew off my normal responsibilities and applied some brain power to the PE.  

I made two modifications to the PE which are testing well for donaldseadog and myself.  He tested more thoroughly than me.

1. PE exports to annex_mags.csv again instead of the errant annex_magsEx.csv file.  I introduced the bug when switching from MS Access Export Specifications to VBA code to create the .csv files.  That was driven by data size limitations and more importantly a Microsoft patch that broke export specifications.

2. PE sorts magazines differently when re-numbering them.  This is what is intended to ensure the ordering of magazines doesn't go awry.  The sorting is now by Annex, then tPlatformMagazine.ID, then tPlatformMagazine.SortID (SortID was previously not part of this qOMagazineRenumberList ORDER BY clause).

If nothing nasty shows up in additional testing, I'll release it with the next build.  The bummer is that this won't fix existing corruption of magazines, it only prevents new corruption on future exports.

I'm not sure, do you mean from the next PE iteration it will not arbitrary changes on the scripted magazines from that point?

[TonyE]

2 hours ago, broncepulido said:

I'm not sure, do you mean from the next PE iteration it will not arbitrary changes on the scripted magazines from that point?

Correct, upon first export with the updated PE you may find the magazines assigned to a platform may be sorted differently than before but after that the ordering should be consistent and of course assigned to the correct platform in the game as in the PE.