Author SHA1 Message Date
Jylpah ec696d6946 add pyutils 2022-10-29 17:47:33 +03:00
Jylpah 1f41901042 update 9.2 2022-10-09 00:18:30 +03:00
Jylpah d275914b32 Update .gitignore 2022-08-18 09:44:39 +03:00
Jylpah 9f8c82f434 Stream client support 2022-08-18 09:42:55 +03:00
Jylpah 3a8f065f35 Update 9.2 2022-08-18 09:42:32 +03:00
Jylpah 0cac200f06 Update 9.1 2022-07-14 11:25:38 +03:00
Jylpah 342b90bcf9 SSL -> TLS 2022-06-09 20:48:50 +03:00
Jylpah a9845b6fbb Update 9.0 2022-06-08 17:25:06 +03:00
Jylpah ee36e9192f Update 8.10 2022-04-30 21:43:24 +03:00
Jylpah 5d5821564e update 8.9 2022-03-24 15:44:48 +02:00
Jylpah 3c6f9dcae3 Merge branch 'master' of https://github.com/Jylpah/blitz-tools 2022-03-24 15:41:26 +02:00
Jylpah 433e2295c2 Update 8.9 2022-03-24 15:41:22 +02:00
Jylpah 3dc9533c6a Update maps.json 2022-03-24 15:41:15 +02:00
Jylpah 83456ae150 Add updates 8.5 - 8.8 2022-02-16 17:48:01 +02:00
Jylpah 0c25c781a9 Update 8.8 2022-02-16 17:41:50 +02:00
Jylpah 244ae314d8 Update 8.8 2022-02-16 17:40:17 +02:00
Jylpah 33b0efb48d Update 8.6 2022-01-22 22:27:34 +02:00
Jylpah 574fb95f00 Update 8.6 2021-12-20 10:15:47 +02:00
Jylpah fdc5db966e Update installation and usage instructions 2021-11-24 10:28:32 +02:00
Jylpah 65e4fd83d8 Update installation instructions 2021-11-24 09:48:39 +02:00
Jylpah e9451c7f5d Merge pull request #34 from Jylpah/stable-filters-update2
Fix --filters and improve performance when filtering
2021-11-23 21:42:45 +02:00
Jylpah fa65872ad2 --filters: Bug fix and performance improvement 2021-11-23 21:42:09 +02:00
Jylpah b2f37ebcf5 Merge branch 'stable-filter-update' into stable-filters-update2 2021-11-23 18:44:09 +02:00
Jylpah d323f11662 Update analyze_wotb_replays.py 2021-11-23 18:38:49 +02:00
Jylpah d847a394a9 Add allies/enemies categorizations: xxx_damage_dealt, xxx_wins, process_battle_results(): Fix bug filter(s) 2021-11-23 18:20:11 +02:00
Jylpah 8514e9db17 Update analyze_wotb_replays.py 2021-11-23 18:08:24 +02:00
Jylpah 521986adf1 Update 8.5 2021-11-23 18:01:17 +02:00
9 changed files with 1002 additions and 251 deletions

No files matched your search

Executable
+15
View File
@@ -0,0 +1,15 @@
__pycache__/
BlitzApp
tmp/
ssl/
*.ini
*.swp
*.code-workspace
*.sqlite3
.vscode/
*.sqlite3-journal
.gitattributes
export*
*.csv
!/stats/*.csv
!*.py
+3
View File
@@ -0,0 +1,3 @@
[submodule "pyutils"]
path = pyutils
url = https://github.com/Jylpah/pyutils.git
+25 -23
View File
@@ -1,5 +1,5 @@
# blitz-tools # blitz-tools
Python scripts to analyze WoT Blitz replays and stats. All the scripts require [Python 3.7](https://www.python.org/downloads/) or higher. I am using 3.8 myself so you may have to change the Python interpreter version in the scripts' first line. Python scripts to analyze WoT Blitz replays and stats. All the scripts require [Python 3.7](https://www.python.org/downloads/) or later.
### Replays ### Replays
* [upload_wotb_replays.py](upload_wotb_replays.py): An asynchronous replay uploader. * [upload_wotb_replays.py](upload_wotb_replays.py): An asynchronous replay uploader.
@@ -13,46 +13,48 @@ Python scripts to analyze WoT Blitz replays and stats. All the scripts require [
* [blitzutils.py](blitzutils.py): A utils class for various Blitz related to functions. *Other scripts use this*, so you need to place it to the same folder with the other scripts * [blitzutils.py](blitzutils.py): A utils class for various Blitz related to functions. *Other scripts use this*, so you need to place it to the same folder with the other scripts
### Data ### Data
* [tanks.json](tanks.json): Tankopedia in WG JSON format extracted from the Blitz app files with [extract_tankopedia.py]](extractTankopedia.py]). Contains only limited information required by the scripts. * [tanks.json](tanks.json): Tankopedia in WG JSON format extracted from the Blitz app files with [extract_tankopedia.py](extractTankopedia.py). Contains only limited information required by the scripts.
* [maps.json](maps.json): Blitz map names extracted from the Blitz app files with [extract_tankopedia.py]](extract_tankopedia.py]). * [maps.json](maps.json): Blitz map names extracted from the Blitz app files with [extract_tankopedia.py](extract_tankopedia.py).
### Quick instructions ### Quick instructions
1. Install [Python 3.8](https://www.python.org/downloads/) on to your computer/laptop 1. Install [Python 3.8](https://www.python.org/downloads/) or later to your computer/laptop
2. Install the required Python modules: motor, aiohttp, aioconsole, aiofiles, aiosqlite, beautifulsoup4, lxml, xmltodict, progress 2. Install the required Python modules (see [requirements.txt](requirements.txt): motor, aiohttp, aioconsole, aiofiles, aiosqlite, beautifulsoup4, lxml, xmltodict, progress
``` ```
# Remember to run the command with a Python interpreter >= 3.7 python -m pip install -r requirements.txt
python -m pip install motor aiohttp aioconsole aiofiles aiosqlite beautifulsoup4 lxml xmltodict progress
``` ```
3. Download all the files including the [maps.json](maps.json) and [tanks.json](tanks.json) files. **The recommened way** is to install the scipts via `git clone`: [GitHub Desktop](https://desktop.github.com/) is an excellent GUI for git. This makes it easy to receive the updates via `git pull`. 3. Clone the repository. [GitHub Desktop](https://desktop.github.com/) is an excellent GUI for `git`. This makes it easy to receive the updates via `git pull`.
4. Upload your replays.
``` ```
# Windows, run directly from Command Prompt or Powershell git clone https://github.com/Jylpah/blitz-tools.git
# Check the options ```
4. Upload your replays with `upload_wotb_replays.py`
```
## Check the options
# Windows Command Prompt or Powershell
python.exe .\upload_wotb_replays.py --help python.exe .\upload_wotb_replays.py --help
# Syntax # Linux or Windows Subsystem for Linux (WSL)
python.exe .\upload_wotb_replays.py REPLAY_FILE.wotbreplay ./upload_wotb_replays.py --help
# Upload replays (Windows)
python.exe .\upload_wotb_replays.py REPLAY_FILE.wotbreplay DIR_WITH_REPLAYS
# example # example
python.exe .\upload_wotb_replays.py 20200223_0122__jylpah_T-54_grossberg.wotbreplay python.exe .\upload_wotb_replays.py 20200223_0122__jylpah_T-54_grossberg.wotbreplay
# Linux
./upload_wotb_replays.py --help
``` ```
5. Analyze replays. 5. Analyze replays with `analyze_wotb_replays.py`
``` ```
# Windows, run directly from Command Prompt or Powershell ## Check options
# Windows Command Prompt or Powershell
python.exe .\analyze_wotb_replays.py --help python.exe .\analyze_wotb_replays.py --help
# Syntax # Syntax (Windows)
python.exe .\analyze_wotb_replays.py REPLAY_FILE.wotbreplay.json python.exe .\analyze_wotb_replays.py REPLAY_FILE.wotbreplay.json [DIR_WITH_REPLAYS ...]
# Example # Example (Windows)
python.exe .\analyze_wotb_replays.py 20200223_0122__jylpah_T-54_grossberg.wotbreplay.json python.exe .\analyze_wotb_replays.py 20200223_0122__jylpah_T-54_grossberg.wotbreplay.json
# Linux # Linux or Windows Subsystem for Linux (WSL)
./analyze_wotb_replays.py --help ./analyze_wotb_replays.py --help
``` ```
+185 -110
View File
@@ -29,6 +29,8 @@ STAT_TANK_BATTLE_MIN = 100
BATTLE_TIME_BUCKET = 3600*24*14 BATTLE_TIME_BUCKET = 3600*24*14
OPT_EXPORT_REPLAY_DIR = 'export_replays' OPT_EXPORT_REPLAY_DIR = 'export_replays'
OPT_EXPORT_CSV_FILE = 'export.csv'
OPT_EXPORT_JSON_FILE = 'export.json'
RE_SRC_IS_DB = re.compile(r'^DB:') RE_SRC_IS_DB = re.compile(r'^DB:')
@@ -205,7 +207,11 @@ class BattleCategorizationList():
'battle_duration' : [ 'Battle Duration', 'bucket', [ x*60 for x in range(0,8) ], 'int' ], 'battle_duration' : [ 'Battle Duration', 'bucket', [ x*60 for x in range(0,8) ], 'int' ],
'distance_travelled' : [ 'Distance Driven', 'bucket', [ x*250 for x in range(0,20) ], 'int' ], 'distance_travelled' : [ 'Distance Driven', 'bucket', [ x*250 for x in range(0,20) ], 'int' ],
'allies_wins' : [ 'Allies WR', 'bucket', [ 0, .35, .45, .50, .55, .65], '%' ], 'allies_wins' : [ 'Allies WR', 'bucket', [ 0, .35, .45, .50, .55, .65], '%' ],
'allies_battles' : [ 'Player Battles', 'bucket', [ 0, 500, 1000, 2500, 5e3, 10e3, 15e3, 25e3], 'int' ],
'allies_damage_dealt' : [ 'Player Avg Dmg', 'bucket', [ 0, 500, 1000, 1250, 1500, 1750, 2000, 2500], 'int' ],
'enemies_wins' : [ 'Enemies WR', 'bucket', [ 0, .35, .45, .50, .55, .65], '%' ], 'enemies_wins' : [ 'Enemies WR', 'bucket', [ 0, .35, .45, .50, .55, .65], '%' ],
'enemies_battles' : [ 'Player Battles', 'bucket', [ 0, 500, 1000, 2500, 5e3, 10e3, 15e3, 25e3], 'int' ],
'enemies_damage_dealt' : [ 'Player Avg Dmg', 'bucket', [ 0, 500, 1000, 1250, 1500, 1750, 2000, 2500], 'int' ],
'player_name' : [ 'Player', 'string', 25 ], 'player_name' : [ 'Player', 'string', 25 ],
'protagonist' : [ 'account_id', 'number' ], 'protagonist' : [ 'account_id', 'number' ],
'tank_name' : [ 'Tank', 'string', 25], 'tank_name' : [ 'Tank', 'string', 25],
@@ -222,10 +228,41 @@ class BattleCategorizationList():
'top_tier' 'top_tier'
] ]
## Player stats based categorizations
_categorizations_stats = [ group + '_' + measure for group in [ 'player', 'allies', 'enemies'] for measure in ['wins', 'battles', 'damage_dealt' ]]
@classmethod
def get_categorizations_all(cls, cats: list = None) -> list:
try:
if cats == None:
return list(cls._categorizations.keys())
else:
return list(set(cls._categorizations.keys()) & set(cats))
except Exception as err:
bu.error(exception=err)
return None
@classmethod @classmethod
def get_categorizations_all(cls): def get_categorizations_stats(cls, cats=None, stats_cats=None):
return cls._categorizations.keys() try:
if cats == None:
return cls._categorizations_stats
else:
if stats_cats == None:
return cats
player_stat_filters = set(cls._categorizations_stats)
cats = set(cls.get_categorizations_all(cats))
if stats_cats:
cats = cats & player_stat_filters
else:
cats = cats - player_stat_filters
bu.debug(', '.join(list(cats)))
return list(cats)
except Exception as err:
bu.error(exception=err)
return None
@classmethod @classmethod
@@ -360,6 +397,10 @@ class BattleCategorizationList():
return BattleCategorizationListIterator(self) return BattleCategorizationListIterator(self)
def len(self):
return len(self.categorizations)
def get_categorization(self, cat: str): def get_categorization(self, cat: str):
"""Return a BattleCategorization for a category""" """Return a BattleCategorization for a category"""
try: try:
@@ -368,6 +409,7 @@ class BattleCategorizationList():
bu.error(exception=err) bu.error(exception=err)
return None return None
def record_result(self, result: dict): def record_result(self, result: dict):
try: try:
for cat in self.categorizations_list: for cat in self.categorizations_list:
@@ -1378,7 +1420,7 @@ class ErrorCatchingArgumentParser(argparse.ArgumentParser):
## main() ------------------------------------------------------------- ## main() -------------------------------------------------------------
async def main(argv): async def main(argv):
global wg, wi, WG_APP_ID, OPT_MODE_DEFAULT global wg, wi, WG_APP_ID, OPT_MODE_DEFAULT, OPT_EXPORT_CSV_FILE, OPT_EXPORT_JSON_FILE
# set the directory for the script # set the directory for the script
current_dir = os.getcwd() current_dir = os.getcwd()
os.chdir(os.path.dirname(sys.argv[0])) os.chdir(os.path.dirname(sys.argv[0]))
@@ -1391,8 +1433,8 @@ async def main(argv):
OPT_HIST = False OPT_HIST = False
OPT_STAT_FUNC = StatFunc.get_default() OPT_STAT_FUNC = StatFunc.get_default()
OPT_WORKERS_N = 10 OPT_WORKERS_N = 10
OPT_EXPORT_CSV_FILE = 'export.csv' # OPT_EXPORT_CSV_FILE = 'export.csv'
OPT_EXPORT_JSON_FILE = 'export.json' # OPT_EXPORT_JSON_FILE = 'export.json'
#WG_ACCOUNT = None # format: nick@server, where server is either 'eu', 'ru', 'na', 'asia' or 'china'. #WG_ACCOUNT = None # format: nick@server, where server is either 'eu', 'ru', 'na', 'asia' or 'china'.
# China is not supported since WG API stats are not available there # China is not supported since WG API stats are not available there
@@ -1404,8 +1446,8 @@ async def main(argv):
## VERY unlikely you have a DB set up ## VERY unlikely you have a DB set up
DB_SERVER = 'localhost' DB_SERVER = 'localhost'
DB_PORT = 27017 DB_PORT = 27017
DB_SSL = False DB_TLS = False
DB_CERT_REQ = ssl.CERT_NONE DB_CERT_REQ = False
DB_AUTH = 'admin' DB_AUTH = 'admin'
DB_NAME = 'BlitzStats' DB_NAME = 'BlitzStats'
DB_USER = 'mongouser' DB_USER = 'mongouser'
@@ -1457,14 +1499,14 @@ async def main(argv):
OPT_DB = configDB.getboolean('opt_DB', OPT_DB) OPT_DB = configDB.getboolean('opt_DB', OPT_DB)
DB_SERVER = configDB.get('db_server', DB_SERVER) DB_SERVER = configDB.get('db_server', DB_SERVER)
DB_PORT = configDB.getint('db_port', DB_PORT) DB_PORT = configDB.getint('db_port', DB_PORT)
DB_SSL = configDB.getboolean('db_ssl', DB_SSL) DB_TLS = configDB.getboolean('db_tls', DB_TLS)
DB_CERT_REQ = configDB.getint('db_ssl_req', DB_CERT_REQ) DB_CERT_REQ = configDB.getboolean('db_tls_req', DB_CERT_REQ)
DB_AUTH = configDB.get('db_auth', DB_AUTH) DB_AUTH = configDB.get('db_auth', DB_AUTH)
DB_NAME = configDB.get('db_name', DB_NAME) DB_NAME = configDB.get('db_name', DB_NAME)
DB_USER = configDB.get('db_user', DB_USER) DB_USER = configDB.get('db_user', DB_USER)
DB_PASSWD = configDB.get('db_password', DB_PASSWD) DB_PASSWD = configDB.get('db_password', DB_PASSWD)
DB_CERT = configDB.get('db_ssl_cert_file', DB_CERT) DB_CERT = configDB.get('db_tls_cert_file', DB_CERT)
DB_CA = configDB.get('db_ssl_ca_file', DB_CA) DB_CA = configDB.get('db_tls_ca_file', DB_CA)
except (KeyError, configparser.NoSectionError) as err: except (KeyError, configparser.NoSectionError) as err:
bu.error(exception=err) bu.error(exception=err)
@@ -1510,7 +1552,7 @@ async def main(argv):
#### Connect to MongoDB. You are unlikely to have this set up... #### Connect to MongoDB. You are unlikely to have this set up...
bu.debug('DB_SERVER: ' + DB_SERVER) bu.debug('DB_SERVER: ' + DB_SERVER)
bu.debug('DB_PORT: ' + str(DB_PORT)) bu.debug('DB_PORT: ' + str(DB_PORT))
bu.debug('DB_SSL: ' + "True" if DB_SSL else "False") bu.debug('DB_TLS: ' + "True" if DB_TLS else "False")
bu.debug('DB_AUTH: ' + DB_AUTH) bu.debug('DB_AUTH: ' + DB_AUTH)
bu.debug('DB_NAME: ' + DB_NAME) bu.debug('DB_NAME: ' + DB_NAME)
@@ -1518,7 +1560,7 @@ async def main(argv):
db = None db = None
if args.db: if args.db:
try: try:
client = motor.motor_asyncio.AsyncIOMotorClient(DB_SERVER,DB_PORT, authSource=DB_AUTH, username=DB_USER, password=DB_PASSWD, ssl=DB_SSL, ssl_cert_reqs=DB_CERT_REQ, ssl_certfile=DB_CERT, tlsCAFile=DB_CA) client = motor.motor_asyncio.AsyncIOMotorClient(DB_SERVER,DB_PORT, authSource=DB_AUTH, username=DB_USER, password=DB_PASSWD, tls=DB_TLS, tlsAllowInvalidCertificates=DB_CERT_REQ, tlsCertificateKeyFile=DB_CERT, tlsCAFile=DB_CA)
db = client[DB_NAME] db = client[DB_NAME]
bu.debug('Database connection initiated') bu.debug('Database connection initiated')
except Exception as err: except Exception as err:
@@ -1548,98 +1590,78 @@ async def main(argv):
args.account_id = await wg.get_account_id(args.account) args.account_id = await wg.get_account_id(args.account)
bu.debug('WG account_id: ' + str(args.account_id)) bu.debug('WG account_id: ' + str(args.account_id))
try: # try:
BattleCategory.set_fields(args.mode) BattleCategory.set_fields(args.mode)
replayQ = asyncio.Queue(maxsize=1000) replayQ = asyncio.Queue(maxsize=1000)
reader_tasks = [] reader_tasks = []
# Make replay Queue # Make replay Queue
scanner_task = asyncio.create_task(mk_replayQ(replayQ, args, db)) scanner_task = asyncio.create_task(mk_replayQ(replayQ, args, db))
bu.set_counter('Reading replays', 10) bu.set_counter('Reading replays', 10)
# Start tasks to process the Queue # Start tasks to process the Queue
for i in range(OPT_WORKERS_N): for i in range(OPT_WORKERS_N):
reader_tasks.append(asyncio.create_task(replay_reader(replayQ, i, args))) reader_tasks.append(asyncio.create_task(replay_reader(replayQ, i, args)))
bu.debug('ReplayReader ' + str(i) + ' started') bu.debug('ReplayReader ' + str(i) + ' started')
bu.debug('Waiting for the replay scanner to finish') bu.debug('Waiting for the replay scanner to finish')
await asyncio.wait([scanner_task]) await asyncio.wait([scanner_task])
bu.finish_progress_bar() bu.finish_progress_bar()
# bu.debug('Scanner finished. Waiting for replay readers to finish the queue') # bu.debug('Scanner finished. Waiting for replay readers to finish the queue')
await replayQ.join() await replayQ.join()
await asyncio.sleep(0.1) await asyncio.sleep(0.1)
bu.debug('Replays read. Cancelling Readers and analyzing results') bu.debug('Replays read. Cancelling Readers and analyzing results')
for task in reader_tasks: for task in reader_tasks:
task.cancel() task.cancel()
await asyncio.sleep(0.1) await asyncio.sleep(0.1)
results = [] results = []
players = set() players = set()
replays = dict() replays = dict()
for res in await asyncio.gather(*reader_tasks): for res in await asyncio.gather(*reader_tasks):
results.extend(res[0]) results.extend(res[0])
players.update(res[1]) players.update(res[1])
replays.update(res[2]) replays.update(res[2])
if len(players) == 0:
raise Exception("No players found to fetch stats for. No replays found?")
if args.min != None:
results = filter_min_replays_by_player(results, args.min)
if len(players) == 0: # Filter based on non-stats filters
raise Exception("No players found to fetch stats for. No replays found?") results = filter_results(results, args.filters, stats_filters=False)
players = get_players(results)
(player_stats, stat_id_map) = await process_player_stats(players, OPT_WORKERS_N, args, db)
# Filter based on stats filters
results = filter_results(results, args.filters, stats_filters=True)
bu.debug(str(len(replays)) + ' replays to process')
replays = filter_replays(replays, results)
bu.debug('Number of player stats: ' + str(len(player_stats)))
teamresults = calc_team_stats(results, player_stats, stat_id_map, args)
if args.min != None: blt_cat_list = process_battle_results(teamresults, args)
results = filter_min_replays_by_player(results, args.min) bu.verbose_std('WR = ' + StatFunc.get_title())
players = get_players(results) blt_cat_list.print_results()
replays = filter_replays(replays, results)
histograms = None
if args.hist:
histograms = process_player_dist(results, player_stats, stat_id_map)
print_player_dist(histograms)
if args.json:
await export_json(args, blt_cat_list, histograms)
if args.csv:
await export_csv(args, blt_cat_list, histograms)
if args.replays:
await export_replays(replays)
(player_stats, stat_id_map) = await process_player_stats(players, OPT_WORKERS_N, args, db) bu.debug('Finished. Cleaning up..................')
bu.debug('Number of player stats: ' + str(len(player_stats))) # except Exception as err:
teamresults = calc_team_stats(results, player_stats, stat_id_map, args) # bu.error(exception=err)
blt_cat_list = process_battle_results(teamresults, args)
bu.verbose_std('WR = ' + StatFunc.get_title())
blt_cat_list.print_results()
histograms = None
if args.hist:
histograms = process_player_dist(results, player_stats, stat_id_map)
print_player_dist(histograms)
if args.json:
res = blt_cat_list.get_results_json()
if histograms != None:
res_hist = dict()
for histogram in histograms.keys():
res_hist[histogram] = histograms[histogram].get_results()
res['histograms'] = res_hist
if args.outfile == '-':
export_file = OPT_EXPORT_JSON_FILE
else:
export_file = args.outfile
await bu.save_JSON(export_file, res, pretty=False) ## Excel does not understand pretty JSON
bu.verbose_std('Results exported to: ' + export_file)
if args.csv:
rows = blt_cat_list.get_results_list()
if args.hist:
rows.append('')
rows.append(['Player Histograms'])
for histogram in histograms.values():
rows.append('')
rows.extend(histogram.get_results_list())
if args.outfile == '-':
export_file = OPT_EXPORT_CSV_FILE
else:
export_file = args.outfile
with open(export_file, 'w') as csvfile:
# creating a csv writer object
csvwriter = csv.writer(csvfile)
csvwriter.writerows(rows)
bu.verbose_std('Results exported to: ' + export_file)
if args.replays:
await export_replays(replays)
bu.debug('Finished. Cleaning up..................')
except Exception as err:
bu.error(exception=err)
except UserWarning as err: except UserWarning as err:
bu.warning(str(err)) bu.warning(str(err))
except asyncio.CancelledError: except asyncio.CancelledError:
@@ -1655,6 +1677,48 @@ async def main(argv):
return None return None
async def export_csv(args : argparse.Namespace, blt_cat_list: BattleCategorizationList, histograms: dict = None):
"""Export results into a CSV file"""
try:
rows = blt_cat_list.get_results_list()
if args.hist:
rows.append('')
rows.append(['Player Histograms'])
for histogram in histograms.values():
rows.append('')
rows.extend(histogram.get_results_list())
if args.outfile == '-':
export_file = OPT_EXPORT_CSV_FILE
else:
export_file = args.outfile
with open(export_file, 'w') as csvfile:
# creating a csv writer object
csvwriter = csv.writer(csvfile)
csvwriter.writerows(rows)
bu.verbose_std('Results exported to: ' + export_file)
except Exception as err:
bu.error(exception=err)
async def export_json(args : argparse.Namespace, blt_cat_list: BattleCategorizationList, histograms: dict = None):
"""Export results into a JSON file"""
try:
res = blt_cat_list.get_results_json()
if histograms != None:
res_hist = dict()
for histogram in histograms.keys():
res_hist[histogram] = histograms[histogram].get_results()
res['histograms'] = res_hist
if args.outfile == '-':
export_file = OPT_EXPORT_JSON_FILE
else:
export_file = args.outfile
await bu.save_JSON(export_file, res, pretty=False) ## Excel does not understand pretty JSON
bu.verbose_std('Results exported to: ' + export_file)
except Exception as err:
bu.error(exception=err)
async def help_extended(db : motor.motor_asyncio.AsyncIOMotorDatabase = None, parser: argparse.ArgumentParser = None): async def help_extended(db : motor.motor_asyncio.AsyncIOMotorDatabase = None, parser: argparse.ArgumentParser = None):
"""Help for --filters""" """Help for --filters"""
if parser != None: if parser != None:
@@ -1808,6 +1872,7 @@ def print_player_dist(histograms: dict):
print('') print('')
hist.print() hist.print()
def process_battle_results(results: dict, args : argparse.Namespace): def process_battle_results(results: dict, args : argparse.Namespace):
"""Process replay battle results""" """Process replay battle results"""
try: try:
@@ -1815,8 +1880,7 @@ def process_battle_results(results: dict, args : argparse.Namespace):
cats = BattleCategorizationList.get_categorizations(args) cats = BattleCategorizationList.get_categorizations(args)
blt_cat_list= None blt_cat_list= None
if args.filters != None: # remove if?
results = filter_results(results, args.filters)
if len(results) > 0: if len(results) > 0:
blt_cat_list = BattleCategorizationList(cats) blt_cat_list = BattleCategorizationList(cats)
for result in results: for result in results:
@@ -1827,7 +1891,7 @@ def process_battle_results(results: dict, args : argparse.Namespace):
blt_cat_list.calc_results() blt_cat_list.calc_results()
else: else:
warning_txt = 'No replays to process ' warning_txt = 'No replays to process '
if args.filter != None: if args.filters != None:
warning_txt = warning_txt + 'after filtering' warning_txt = warning_txt + 'after filtering'
bu.warning(warning_txt) bu.warning(warning_txt)
@@ -1836,21 +1900,28 @@ def process_battle_results(results: dict, args : argparse.Namespace):
return blt_cat_list return blt_cat_list
def filter_results(results: list, filter_json : str) -> bool: def filter_results(results: list, filters_json : str, stats_filters=None) -> bool:
"""Filter replays based on battle category filters""" """Filter replays based on battle category filters"""
try: try:
filters = json.loads(filter_json) if filters_json == None:
return results
filters = json.loads(filters_json)
bu.debug(str(filters)) bu.debug(str(filters))
if type(filters) != dict: if type(filters) != dict:
bu.error('invalid --filter arguments: ' + str(filter)) bu.error('invalid --filter arguments: ' + str(filter))
sys.exit(1) sys.exit(1)
filter_cats = BattleCategorizationList(filters.keys()) filter_cats = BattleCategorizationList.get_categorizations_stats(filters.keys(), stats_filters)
filter_cats = BattleCategorizationList(filter_cats)
if filter_cats.len() == 0:
bu.debug('Nothing to filter')
return results
bu.verbose('Filters applied:') bu.verbose('Filters applied:')
for category in filter_cats: for category in filter_cats:
cat = category.category_key cat = category.category_key
categorization = filter_cats.get_categorization(cat) categorization = filter_cats.get_categorization(cat)
bu.debug('Category: ' + cat + ' filters type: ' + str(type(filters)) + ' filters: ' + str(filters)) bu.debug('Category: ' + cat + ' filters: ' + str(filters))
filters[cat] = categorization.get_filter_categories(filters[cat]) filters[cat] = categorization.get_filter_categories(filters[cat])
bu.verbose(' ' + cat + ' = ' + ','.join(filters[cat])) bu.verbose(' ' + cat + ' = ' + ','.join(filters[cat]))
bu.verbose('') bu.verbose('')
@@ -1870,8 +1941,12 @@ def filter_results(results: list, filter_json : str) -> bool:
res.append(result) res.append(result)
except Exception as err: except Exception as err:
bu.error(exception=err) bu.error(exception=err)
if len(res) == 0:
raise UserWarning('No replays left after filtering')
bu.verbose_std('Replays after filtering: ' + str(len(res))) bu.verbose_std('Replays after filtering: ' + str(len(res)))
return res return res
except UserWarning:
raise
except Exception as err: except Exception as err:
bu.error(exception=err) bu.error(exception=err)
return None return None
@@ -1946,9 +2021,9 @@ async def process_player_stats(players, N_workers: int, args : argparse.Namespac
def remap_stat_id(stat_id_map: dict, stat_id_remap: dict): def remap_stat_id(stat_id_map: dict, stat_id_remap: dict):
bu.debug('Remapping stat_ids: stat_id_map: ' + str(len(stat_id_map)) + ' remap needed: ' + str(len(stat_id_remap))) bu.debug('Remapping stat_ids: stat_id_map: ' + str(len(stat_id_map)) + ' remap needed: ' + str(len(stat_id_remap)))
for stat_id in stat_id_map: for stat_id in stat_id_map:
# bu.debug(stat_id) # bu.debug(stat_id)
if stat_id_map[stat_id] in stat_id_remap.keys(): if stat_id_map[stat_id] in stat_id_remap.keys():
stat_id_map[stat_id] = stat_id_remap[stat_id_map[stat_id]] stat_id_map[stat_id] = stat_id_remap[stat_id_map[stat_id]]
return stat_id_map return stat_id_map
@@ -2424,7 +2499,7 @@ async def mk_readerQ_item(replay_json, filename : str = None, _id: str = None) -
try: try:
if wi.chk_JSON_replay(replay_json): if wi.chk_JSON_replay(replay_json):
if not '_id' in replay_json: if not '_id' in replay_json:
if _id != None: if _id == None:
_id = wi.read_replay_id(replay_json) _id = wi.read_replay_id(replay_json)
replay_json['_id'] = _id replay_json['_id'] = _id
else: else:
+12 -6
View File
@@ -3,6 +3,7 @@
# Extract tankopedia data in WG API JSON format from Blitz app files (Android APK unzipped) # Extract tankopedia data in WG API JSON format from Blitz app files (Android APK unzipped)
from os import path
import sys, argparse, json, os, inspect, aiohttp, asyncio, aiofiles, re, logging, time, xmltodict, collections, configparser import sys, argparse, json, os, inspect, aiohttp, asyncio, aiofiles, re, logging, time, xmltodict, collections, configparser
import blitzutils as bu import blitzutils as bu
from blitzutils import WG from blitzutils import WG
@@ -11,9 +12,11 @@ logging.getLogger("asyncio").setLevel(logging.DEBUG)
FILE_CONFIG = 'blitzstats.ini' FILE_CONFIG = 'blitzstats.ini'
BLITZAPP_STRINGS='/assets/Data/Strings/en.yaml' # BLITZAPP_STRINGS='/assets/Data/Strings/en.yaml'
BLITZAPP_VEHICLES_DIR='/assets/Data/XML/item_defs/vehicles/' # BLITZAPP_VEHICLES_DIR='/assets/Data/XML/item_defs/vehicles/'
BLITZAPP_VEHICLE_FILE='/list.xml' BLITZAPP_STRINGS='Data/Strings/en.yaml'
BLITZAPP_VEHICLES_DIR='Data/XML/item_defs/vehicles/'
BLITZAPP_VEHICLE_FILE='list.xml'
wg = None wg = None
@@ -104,8 +107,11 @@ async def main(argv):
async def extract_tanks(blitz_app_base : str, nation: str): async def extract_tanks(blitz_app_base : str, nation: str):
tanks = [] tanks = []
list_xml_file = blitz_app_base + BLITZAPP_VEHICLES_DIR + nation + BLITZAPP_VEHICLE_FILE # WG has changed the location of Data directory - at least in steam client
if not os.path.isfile(list_xml_file): if path.isdir(path.join(blitz_app_base, 'assets')):
blitz_app_base = path.join(blitz_app_base, 'assets')
list_xml_file = path.join(blitz_app_base, BLITZAPP_VEHICLES_DIR ,nation ,BLITZAPP_VEHICLE_FILE)
if not path.isfile(list_xml_file):
print('ERROR: cannot open ' + list_xml_file) print('ERROR: cannot open ' + list_xml_file)
return None return None
bu.debug('Opening file: ' + list_xml_file + ' (Nation: ' + nation + ')') bu.debug('Opening file: ' + list_xml_file + ' (Nation: ' + nation + ')')
@@ -132,7 +138,7 @@ async def read_user_strs(blitz_app_base : str) -> dict:
"""Read user strings to convert map and tank names""" """Read user strings to convert map and tank names"""
tank_strs = {} tank_strs = {}
map_strs = {} map_strs = {}
filename = blitz_app_base + BLITZAPP_STRINGS filename = path.join(blitz_app_base, BLITZAPP_STRINGS)
bu.debug('Opening file: ' + filename + ' for reading UserStrings') bu.debug('Opening file: ' + filename + ' for reading UserStrings')
try: try:
async with aiofiles.open(filename, 'r', encoding="utf8") as f: async with aiofiles.open(filename, 'r', encoding="utf8") as f:
+2 -1
View File
@@ -18,6 +18,7 @@
"fort": "Fort Despair", "fort": "Fort Despair",
"grossberg": "Dynasty's Pearl", "grossberg": "Dynasty's Pearl",
"himmelsdorf": "Himmelsdorf", "himmelsdorf": "Himmelsdorf",
"holland": "Molendijk",
"holmeisk": "Wasteland", "holmeisk": "Wasteland",
"iceworld": "Everfrost", "iceworld": "Everfrost",
"idle": "Yukon", "idle": "Yukon",
@@ -56,4 +57,4 @@
"skit": "Naval Frontier", "skit": "Naval Frontier",
"test": "World of Ducks", "test": "World of Ducks",
"tutorial": "Proving Grounds" "tutorial": "Proving Grounds"
} }
Submodule
+1
Submodule pyutils added at 1eb17e8402.
+94 -86
View File
@@ -1,88 +1,96 @@
# WoT Blitz releases # WoT Blitz releases
| Release | Date | Notes | Inaccurate? | | Release | Date | Notes |
|:-------:|:--------:|-------------|-----| |:-------:|:--------:|-------------|
| 8.4 | 2021-10-20 | Polish heavy tanks | | | 9.1 | 2022-07-14 | Releases updated via DLC, graphics updates for Object 140, Patton 46 |
| 8.3 | 2021-09-16 | FV4005, FV4022, VK 91.01 (P) and WZ-121 legendary camos, -5 deg gun depression to WZ-120 | | | 9.0 | 2022-06-08 | Japanies heavies, Castilla graphics update, new garage |
| 8.2 | 2021-08-18 | Ghost Factory map, XP/credits nerf for inactive players, new store UI | | | 8.10 | 2022-04-28 | New battle results screen, new "Operation" mechanics |
| 8.1 | 2021-07-15 | Armor highlighting updated | | | 8.9 | 2022-03-23 | "Free" camo resupply, VK 72.01 supplies |
| 8.0 | 2021-06-08 | American experimental heavies (Yoh-line), PBR camos | | | 8.8 | 2022-02-16 | Molendijk map |
| 7.9 | 2021-04-27 | Mines rework, new tutorial, colorblind mode, JgPz E100 legendary camo | | | 8.7 | 2022-01-19 | Soviet Light Tanks |
| 7.8 | 2021-03-24 | Mid-tier balance changes, Grille line new consumables | | | 8.6 | 2021-12-15 | Joining Light and Medium Tank classes in MM |
| 7.7 | 2021-02-24 | Czechoslovakian medium tanks | | | 8.5 | 2021-11-17 | New national crew voices, economy update |
| 7.6 | 2021-01-27 | AMC 13 90, TOG II buffs, free replay camera | | | 8.4 | 2021-10-20 | Polish heavy tanks |
| 7.5 | 2020-12-09 | ATGM removal, Batchat, Kpz70, T-2020 buff | | | 8.3 | 2021-09-16 | FV4005, FV4022, VK 91.01 (P) and WZ-121 legendary camos, -5 deg gun depression to WZ-120 |
| 7.4 | 2020-11-12 | TD buffs, Clan missions | | | 8.2 | 2021-08-18 | Ghost Factory map, XP/credits nerf for inactive players, new store UI |
| 7.3 | 2020-10-08 | Swedish HT line | | | 8.1 | 2021-07-15 | Armor highlighting updated |
| 7.2 | 2020-08-19 | Balance changes, added tier X ammo capa, new aiming reticles | | | 8.0 | 2021-06-08 | American experimental heavies (Yoh-line), PBR camos |
| 7.1 | 2020-07-15 | New Yukon map, IS-6, KV-4, Patton 48 armor buff, Grille DPM buff, SP IC, AMZ 13 75, M41 Bulldog mobility buff | | | 7.9 | 2021-04-27 | Mines rework, new tutorial, colorblind mode, JgPz E100 legendary camo |
| 7.0 | 2020-06-17 | British LTs, graphics improvements, ATGM nerf, Sheridan nerf, map fixes | | | 7.8 | 2021-03-24 | Mid-tier balance changes, Grille line new consumables |
| 6.10 | 2020-04-29 | T92E1 nerf, Tier V-IX heavy HP buff, Battle Pass, map fixes | | | 7.7 | 2021-02-24 | Czechoslovakian medium tanks |
| 6.9 | 2020-03-25 | Japanese TDs, Tier X heavy HP buff | | | 7.6 | 2021-01-27 | AMC 13 90, TOG II buffs, free replay camera |
| 6.8 | 2020-02-19 | ATGM nerf, Hellas map | | | 7.5 | 2020-12-09 | ATGM removal, Batchat, Kpz70, T-2020 buff |
| 6.7 | 2020-01-15 | ATGM US Lights | | | 7.4 | 2020-11-12 | TD buffs, Clan missions |
| 6.6 | 2019-12-18 | T-62A + Leo buffs | | | 7.3 | 2020-10-08 | Swedish HT line |
| 6.5 | 2019-11-21 | New UI | | | 7.2 | 2020-08-19 | Balance changes, added tier X ammo capa, new aiming reticles |
| 6.4 | 2019-10-16 | Halloween map, mid-tier HT buff... | | | 7.1 | 2020-07-15 | New Yukon map, IS-6, KV-4, Patton 48 armor buff, Grille DPM buff, SP IC, AMZ 13 75, M41 Bulldog mobility buff |
| 6.3 | 2019-09-18 | European line | | | 7.0 | 2020-06-17 | British LTs, graphics improvements, ATGM nerf, Sheridan nerf, map fixes |
| 6.2 | 2019-08-14 | New MM | | | 6.10 | 2020-04-29 | T92E1 nerf, Tier V-IX heavy HP buff, Battle Pass, map fixes |
| 6.1 | 2019-07-10 | Autoloader buff | | | 6.9 | 2020-03-25 | Japanese TDs, Tier X heavy HP buff |
| 6.0 | 2019-06-05 | UK TD line #2 | | | 6.8 | 2020-02-19 | ATGM nerf, Hellas map |
| 5.10 | 2019-04-24 | new missions | | | 6.7 | 2020-01-15 | ATGM US Lights |
| 5.9 | 2019-03-27 | New shell tracers, minor fixes | | | 6.6 | 2019-12-18 | T-62A + Leo buffs |
| 5.8 | 2019-02-21 | Chinese TD line | | | 6.5 | 2019-11-21 | New UI |
| 5.7 | 2019-01-16 | Avatars | | | 6.4 | 2019-10-16 | Halloween map, mid-tier HT buff... |
| 5.6 | 2018-12-05 | Normandy map | | | 6.3 | 2019-09-18 | European line |
| 5.5 | 2018-11-13 | Low-tier clean-up | | | 6.2 | 2019-08-14 | New MM |
| 5.4 | 2018-10-10 | MM "rigging" adjusted | | | 6.1 | 2019-07-10 | Autoloader buff |
| 5.3 | 2018-09-12 | MM "rigging" started, Newbie queue added | | | 6.0 | 2019-06-05 | UK TD line #2 |
| 5.2 | 2018-08-22 | Faust map | | | 5.10 | 2019-04-24 | new missions |
| 5.1 | 2018-07-31 | New MM: vehicle class matching | | | 5.9 | 2019-03-27 | New shell tracers, minor fixes |
| 5.0 | 2018-06-30 | Vineyeards map update, Spare Parts killed | | | 5.8 | 2019-02-21 | Chinese TD line |
| 4.10 | 2018-05-24 | Alpenstadt map | | | 5.7 | 2019-01-16 | Avatars |
| 4.9 | 2018-04-25 | Chinese heavy tanks | | | 5.6 | 2018-12-05 | Normandy map |
| 4.8 | 2018-03-28 | Choice of battle modes, crew skills accelerator | | | 5.5 | 2018-11-13 | Low-tier clean-up :-( |
| 4.7 | 2018-02-28 | New minimap | | | 5.4 | 2018-10-10 | MM "rigging" adjusted |
| 4.6 | 2018-01-31 | Chinese nation | | | 5.3 | 2018-09-12 | MM "rigging" started, Newbie queue added |
| 4.5 | 2017-12-19 | Dynasty's Pearl map | | | 5.2 | 2018-08-22 | Faust map |
| 4.4 | 2017-11-09 | New British tanks | | | 5.1 | 2018-07-31 | New MM: vehicle class matching |
| 4.3 | 2017-10-10 | Level 7 screw skills | | | 5.0 | 2018-06-30 | Vineyeards map update, Spare Parts removed ! |
| 4.2 | 2017-09-12 | French LTs, Naval Frontier map | | | 4.10 | 2018-05-24 | Alpenstadt map |
| 4.1 | 2017-08-16 | New markers | | | 4.9 | 2018-04-25 | Chinese heavy tanks |
| 4.0 | 2017-07-13 | Rating battles | Y | | 4.8 | 2018-03-28 | Choice of battle modes, crew skills accelerator |
| 3.10 | 2017-06-15 | Fixes | Y | | 4.7 | 2018-02-28 | New minimap |
| 3.9 | 2017-05-19 | Military Honor, Mayan Ruins | Y | | 4.6 | 2018-01-31 | Chinese nation |
| 3.8 | 2017-04-24 | Spare Parts, New 9-slot Equipment system | Y | | 4.5 | 2017-12-19 | Dynasty's Pearl map |
| 3.7 | 2017-03-22 | Balance changes, Tournaments | Y | | 4.4 | 2017-11-09 | New British tanks |
| 3.6 | 2017-02-16 | Vehicle Enrichment | Y | | 4.3 | 2017-10-10 | Level 7 screw skills |
| 3.5 | 2017-01-13 | Boosters, Dead rail update | Y | | 4.2 | 2017-09-12 | French LTs, Naval Frontier map |
| 3.4 | 2016-11-28 | Canyon map, USA T57 line | | | 4.1 | 2017-08-16 | New markers |
| 3.3 | 2016-11-01 | Updated Mines map | | | 4.0 | 2017-07-13 | Rating battles (date uncertain) |
| 3.2 | 2016-10-04 | Copperfield update | | | 3.10 | 2017-06-15 | Fixes (date uncertain) |
| 3.1 | 2016-09-03 | Clan tournaments, Yamato Harbor map | Y | | 3.9 | 2017-05-19 | Military Honor, Mayan Ruins (date uncertain) |
| 3.0 | 2016-07-20 | Supremacy mode, Falls Creek update | Y | | 3.8 | 2017-04-24 | Spare Parts, New 9-slot Equipment system (date uncertain) |
| 2.11 | 2016-06-28 | Vineyeards map, Oasis Palms update, German paper TDs | | | 3.7 | 2017-03-22 | Balance changes, Tournaments (date uncertain) |
| 2.10 | 2016-06-07 | Rockfield map update, Tank birthdays, high-pen gun for IS-6 | | | 3.6 | 2017-02-16 | Vehicle Enrichment (date uncertain) |
| 2.9 | 2016-05-03 | Training rooms, +/-1 Matchmaker | Y | | 3.5 | 2017-01-13 | Boosters, Dead rail update (date uncertain) |
| 2.8 | 2016-04-04 | New Russian TD line, new consumables system | Y | | 3.4 | 2016-11-28 | Canyon map, USA T57 line |
| 2.7 | 2016-03-07 | Mirage map | Y | | 3.3 | 2016-11-01 | Updated Mines map |
| 2.6 | 2016-02-10 | German LTs, shooting through obstacles | Y | | 3.2 | 2016-10-04 | Copperfield update |
| 2.5 | 2016-01-11 | Camouflages, Clan supply | Y | | 3.1 | 2016-09-03 | Clan tournaments, Yamato Harbor map (date uncertain) |
| 2.4 | 2015-12-09 | Additional Japanese tanks, windows version | Y | | 3.0 | 2016-07-20 | Supremacy mode, Falls Creek update (date uncertain) |
| 2.3 | 2015-11-12 | Canal map | Y | | 2.11 | 2016-06-28 | Vineyeards map, Oasis Palms update, German paper TDs |
| 2.2 | 2015-10-05 | Desert Sands map update | Y | | 2.10 | 2016-06-07 | Rockfield map update, Tank birthdays, high-pen gun for IS-6 |
| 2.1 | 2015-09-07 | UK TDs, Clan Chat | Y | | 2.9 | 2016-05-03 | Training rooms, +/-1 Matchmaker (date uncertain) |
| 2.0 | 2015-08-05 | Clans, Oasis Palms updated | | | 2.8 | 2016-04-04 | New Russian TD line, new consumables system (date uncertain) |
| 1.11 | 2015-07-13 | KV-2, KV-13, Sherman Jumbo | Y | | 2.7 | 2016-03-07 | Mirage map (date uncertain) |
| 1.10 | 2015-06-11 | Port Bay map | Y | | 2.6 | 2016-02-10 | German LTs, shooting through obstacles (date uncertain) |
| 1.9 | 2015-05-13 | New German medium tanks (Leo) | Y | | 2.5 | 2016-01-11 | Camouflages, Clan supply (date uncertain) |
| 1.8 | 2015-04-14 | Game tutorial, Bot map for new players | Y | | 2.4 | 2015-12-09 | Additional Japanese tanks, windows version (date uncertain) |
| 1.7 | 2015-02-16 | US Turreted TDs | Y | | 2.3 | 2015-11-12 | Canal map (date uncertain) |
| 1.6 | 2015-01-19 | Castilla map, UK HT line | Y | | 2.2 | 2015-10-05 | Desert Sands map update (date uncertain) |
| 1.5 | 2014-12-01 | UK Nation | Y | | 2.1 | 2015-09-07 | UK TDs, Clan Chat (date uncertain) |
| 1.4 | 2014-10-29 | German HT line (Maus) | Y | | 2.0 | 2015-08-05 | Clans, Oasis Palms updated |
| 1.3 | 2014-10-02 | USSR IS-4 line, Lost Temple, Black Goldville maps | Y | | 1.11 | 2015-07-13 | KV-2, KV-13, Sherman Jumbo (date uncertain) |
| 1.2 | 2014-08-20 | Oasis Palms map | Y | | 1.10 | 2015-06-11 | Port Bay map (date uncertain) |
| 1.1.1 | 2014-07-23 | Fixes :-) | | | 1.9 | 2015-05-13 | New German medium tanks (Leo) (date uncertain) |
| 1.0 | 2014-06-26 | Worldwide launch | | | 1.8 | 2015-04-14 | Game tutorial, Bot map for new players (date uncertain) |
| 0.9 | 2014-05-05 | Nordic soft launch | | | 1.7 | 2015-02-16 | US Turreted TDs (date uncertain) |
| 1.6 | 2015-01-19 | Castilla map, UK HT line (date uncertain) |
| 1.5 | 2014-12-01 | UK Nation (date uncertain) |
| 1.4 | 2014-10-29 | German HT line (Maus) (date uncertain) |
| 1.3 | 2014-10-02 | USSR IS-4 line, Lost Temple, Black Goldville maps (date uncertain) |
| 1.2 | 2014-08-20 | Oasis Palms map (date uncertain) |
| 1.1.1 | 2014-07-23 | Fixes :-) |
| 1.0 | 2014-06-26 | Worldwide launch |
| 0.9 | 2014-05-05 | Nordic soft launch |
+665 -25
View File
File diff suppressed because it is too large. Load diff