Kuat Drive Yards
  2113 active members
  281 are online

Year

19

Day

323

Time

10:24:21

Guest
Login
snewsgnewsmessagegeneralfactioninventorycombatroom
Year 15 Day 102 20:00
Otto Mustard

Two questions:

1. An update was made in the past week or so to the export of Raw materials from inventory. Amongst other changes, the Sector/System/Planet, etc. and all X/Y coordinates were removed from the exported file. Can/will this location information be restored to the exports? The location information is quite valuable in sorting out quantities of RMs in various systems.

2. I didn't see anything about this change in the Sim News. Is there another location where change logs are posted?

Thanks,
Otto


Year 15 Day 103 7:38
This one appears to be an interim exploit fix related to This Simnews, and some bug reports.

I would guess that this was done as a stopgap measure while they figure out the best way to permit export of location information for owners, but not for those who aren't able to view the location of the particular entity, without screwing up the columns in the export.

I agree that the location info is extremely useful. I strongly believe they'll eventually return, as long as the exploit can be effectively prevented.


Year 15 Day 103 7:49
It was the simplest way to remove the exploit at the moment. If and when a better solution is found to stop said exploit, the feature may be returned. But until now theres not much we can do about it.


Year 15 Day 104 2:28
Ben Camden

Would it be possible to utilize the same (or similar) code that gives us the inventory screen, to ensure that the exports only gives the information they're supposed to give? Or would that be very mean to the server?


____________

gold1.gifRGC-3.pnggold2.gif
Senior Governor Ben Camden
Regional Government
Year 15 Day 104 6:39
Varlathotep Al`Jormungandr Ragnvaldansson

Ben, I'd guess that on the back end they actually do pull the same information (though this assumption is very likely incorrect). Chances are the only difference is the interface masks out information while the export files appear to be more of a data dump from the database itself.


____________

That's got to smarts.
Year 15 Day 105 3:44
Ben Camden

That's why I asked Var.
There is some function to do it in place, however given that it usually takes several seconds to get an inventory, I don't imagine doing the same thing with up to 50 times the amount of data would be fast, nor light on the server.

But maybe a complicated query like that could be made as a low priority thing in the background (depending on the DB used), and put into a temporary bloated table for later download... However that might just require a change of the entire export process >_


____________

gold1.gifRGC-3.pnggold2.gif
Senior Governor Ben Camden
Regional Government
Year 15 Day 105 5:48
Varlathotep Al`Jormungandr Ragnvaldansson

Inevitably the more stuff you're querying the longer it takes. The problem would definitely be with larger inventories like government inventories where they've got thousands upon thousands of entities. It's not exactly something you can just "throw to the background" because the database server itself is already a separate entity than the actual Combine server the game is run off of plus PHP waits for the MySQL(i) function to return a value before continuing on with the script.

The biggest problem would be a mix of the getting the data then removing the locational data from only the entities that you're not supposed to see. You have to remember that this would not only have to query the entity table but what ever table stores access data as well to check to see if you're actually able to access the location.

As for the later downloading thing, I'm not sure how practical that would be.

While it'd be nice to have the locational data pulled out with the export function, it really (at this time) isn't practical and would most definitely cause more stress on the servers than its worth.


____________

That's got to smarts.
Year 15 Day 105 8:00
Ben Camden

Yeah, so PHP ruins my idea, oh well. If it can't initiate a query, give a text message to the user and then just let the query complete in the background my idea wouldn't work...


____________

gold1.gifRGC-3.pnggold2.gif
Senior Governor Ben Camden
Regional Government
Year 15 Day 105 8:26
Varlathotep Al`Jormungandr Ragnvaldansson

I wouldn't say it really "ruins" the idea. It just makes it more difficult and more hacky to implement. There appears to be some way to run queries that way from PHP (at least it APPEARS to, I'm not 100% sure if it actually DOES) but it requires additional extensions from PHP that Combine might not have.


____________

That's got to smarts.
Year 15 Day 106 2:07
Ben Camden

Yeah, and further thinking along the lines have gotten me to a possible solution. though it would entail people having to wait up to an hour for exports longer than 100 entities - not to mention probably a good deal of the exporting procedures re-done >_<

It would, however, involve a new link in our inventory menu, a few new tables (probably up to 10 for all entity types), a separate database user for exports using mysql-governor to limit its resource consumption and a cron job to slowly do exports (and clean up).

The "only" extension needed here would be the mysql-governor - assuming we're using a mysql version compatible with it (assuming swc is using mysql even).


____________

gold1.gifRGC-3.pnggold2.gif
Senior Governor Ben Camden
Regional Government
Year 15 Day 106 4:29
Varlathotep Al`Jormungandr Ragnvaldansson

That honestly sounds like far more work than its worth. At that point why not just set up your own page crawler using automation software that'll go through your inventory and make a csv of it? Or if it's a faction inventory just have a few recruits copy out all of the data and drop it into excel.

Truthfully I don't think any option that requires new php extensions would be the best idea.


____________

That's got to smarts.
Year 15 Day 106 7:27
Ben Camden

Well, page crawlers would be a no-go on SWC, as far as I know. When we're not allowed to use them on the galaxy map, I seriously doubt we are on inventory.

And the purpose is limited to spreading out the larger, complicated exports over time. It might even mean dropping the 5000 limit as well.
Mysql-governor also isn't a php extension, it's a mysql extension (not that it's that much of a difference, it's one more thing to have installed and configured on the server).

It is probably a lot of work though, that I'm 99.99 % certain of.


____________

gold1.gifRGC-3.pnggold2.gif
Senior Governor Ben Camden
Regional Government
Year 15 Day 106 7:57
Varlathotep Al`Jormungandr Ragnvaldansson

Yeah, having to actually install an additional extension would most likely be a no go. My thought was lowering the number of records it exported down to something like 1000 entities at a time that way the server would have plenty of time to get the list, mask it and then send the file.

I don't think running it in the background at this point is really even an option since there's a lot more to it than just "send this query to the back, create this file and allow it to be downloaded". They'd have to consider the security of the file created, the resources these queries will use and still the part of the system masking the results will be PHP so it's not exactly like they could throw that part to the background without doing some strange things.


____________

That's got to smarts.
Year 15 Day 115 19:42
Otto Mustard

Looks like it's been fixed.

Thanks!