feature(zms): add new league middlewares and other models and tools. by hiha3456 · Pull Request #458 · opendilab/DI-engine

hiha3456 · 2022-08-26T07:29:07Z

Description

add LeagueCoordinator, LeagueLearnerCommunicator, StepLeagueActor, BattleStepCollector, battle_inferencer, battle_rolloutor, with their corresponding tests.
add BattleTransitionList to gather transitions and cut trajectories for league environment
add dataclass of actor data and learner model
add an attribute return_original_data into EnvSupervisor
add BattleContext into context.py
add EventEnum and add feature in event_loop so that we can add customized string in chosen events.
add utils sparse_logging to logging in a sparse frequency
add my_pickle_loads inside ding/framework/parallel.py so that we could transfer a cuda tensor to a pure-cpu node without bug
add old Storage and FileStorage class used in old dev-league branch inside ding/framework/storage
change a bit player
add sl_branch inside starcraft_player
add old ding/league/v2/base_league.py used in old dev-league branch
add steve, change upgo, vtrace in ding/rl_utils
add detach_grad, flatten in ding/torch_utils/data_helper.py
add l2_distance in ding/torch_utils/metric.py
add GLU2, GatedConvResBlock, scatter_connection_v2, AttentionPool, lstm ding/torch_utils/network/, and make some changes in networks
add read_yaml_config in ding/utils/default_helper.py
and other changes...

Related Issue

TODO

delete old checkpoints saved by LeagueLearnerCommunicator, because it can consume disk storage very quickly
When we use multiple envs inside env_manager(or other variants), in some cases only some of the envs work properly, so for the return timesteps of step(action), I hugely recommend that, all the EnvManagers need to return the timesteps in format dict instead of list. As far as I know, the return of EnvSupervisor and BaseEnvManagerV2 is dict, and the returns subprocesEnvManager and BaseEnvManager is list.
The BattleCollector now could not handle the case when the policy has intermediate state, for example, the policy of SC2(which in DI-star) maintain a huge amount of intermediate state. In this case, each env should maintain number of players policies, which is not the case in current BattleCollector. The current BattleCollector can only handle this kind of policy when EnvManager has only one environment.
add middleware of teacher model.

Check List

merge the latest version source branch/repo, and resolve all the conflicts
pass style check
pass all the tests

…v-league-lxl

…gine into dev-league-lxl

…ansitionList, make battle_rolloutor_for_distar run in dict and list cases

…ector

ding/rl_utils/steve.py

ding/torch_utils/data_helper.py

ding/torch_utils/metric.py

ding/torch_utils/network/activation.py

ding/utils/data/collate_fn.py

ding/envs/env_manager/env_supervisor.py

PaParaZz1 · 2022-08-27T03:50:01Z

ding/framework/context.py

        self.keep('train_iter', 'last_eval_iter')
+
+
+class BattleContext(Context):


add overview comments

ding/utils/tests/test_sparse_logging.py

ding/utils/sparse_logging.py

PaParaZz1 · 2022-09-14T05:12:17Z

ding/torch_utils/network/scatter_connection.py

        return output
+
+
+def scatter_connection_v2(shape, project_embeddings, entity_location, scatter_dim, scatter_type='add'):


can we merge v2 to original ScatterConnection

PaParaZz1 · 2022-09-14T05:13:24Z

ding/torch_utils/network/activation.py

+        return x
+
+
+def build_activation2(activation):


we should merge this function into original version

…at because it is useless

PaParaZz1 · 2022-09-23T03:33:15Z

ding/framework/middleware/league_coordinator.py

+    from ding.framework import Task, Context
+    from ding.league.v2 import BaseLeague
+    from ding.league.player import PlayerMeta
+    from ding.league.v2.base_league import Job


on import from 2-level directory, like from ding.league.v2 import BaseLeague, Job

PaParaZz1 · 2022-09-23T03:34:51Z

ding/framework/middleware/league_coordinator.py

+        sleep(1)
+        log_every_sec(
+            logging.INFO, 600, "[Coordinator {}] running jobs {}".format(task.router.node_id, self._running_jobs)
+        )


maybe we should add state_dict/load_state_dict method for coordinator?

PaParaZz1 · 2022-09-23T03:39:47Z

ding/framework/middleware/tests/mock_for_test.py

+    def get_job_info(self, player_id):
+        self.get_job_info_cnt += 1
+        other_players = [i for i in self.active_players_ids if i != player_id]
+        another_palyer = random.choice(other_players)


PaParaZz1 · 2022-09-23T03:43:46Z

ding/framework/parallel.py

+    return unpickler.load()
+
+
+def my_pickle_loads(msg):


move this function to ding/utils

PaParaZz1 · 2022-09-23T03:48:14Z

ding/framework/middleware/functional/actor_data.py

+
+@dataclass
+class PlayerModelInfo:
+    get_new_model_time: float


why two time here, maybe we can simplify them

PaParaZz1 · 2022-09-23T03:52:32Z

ding/framework/middleware/functional/collector.py

+        # If we don't have Trajectory(t-1), i.e. the length of the whole episode is smaller than unroll_len,
+        # we fill up the trajectory with the first element of episode.
+        return_episode = []
+        i = 0


why init i here

PaParaZz1 · 2022-09-23T03:53:52Z

ding/framework/middleware/functional/collector.py

+                    initial_elements.append(trajectory[0])
+                trajectory = initial_elements + trajectory
+            if self._last_step_fn:
+                last_step = deepcopy(trajectory[-1])


why deepcopy here

PaParaZz1 · 2022-09-23T03:57:01Z

ding/framework/middleware/league_actor.py

+            task.router.node_id
+        )
+
+        ctx.n_episode = self.cfg.policy.collect.n_episode


PaParaZz1 · 2022-09-23T03:57:18Z

ding/framework/middleware/league_actor.py

+            time_begin = time.time()
+            collector(ctx)
+
+            if ctx.job_finish is True:


if ctx.job_finish

PaParaZz1 · 2022-09-23T03:57:51Z

ding/framework/middleware/league_actor.py

+                )
+            )
+
+            gc.collect()


why call gc here

hiha3456 and others added 30 commits May 28, 2022 20:34

change a bit

b404024

change a bit

ffcc50b

change a bit

a55fe17

change a bit

8084dae

change a bit

da46b22

change a bit

65b8762

change position of policy_resetter, battle_rolloutor, battle_inferencer

ad9632c

polish stdim & infonce loss

192e6e4

polish EventEnum

46ee7cf

Merge branch 'main' of https://github.com/opendilab/DI-engine into de…

654eb9f

…v-league-lxl

polish EventEnum

48ddc09

simplify the code

474524e

for multiple policies

9d91063

change a bit

6ca4ae2

fix style

2d85ccf

revert to drop the streaming type code

157b9b3

fix codecov

a13b5e6

Merge branch 'main' of https://github.com/opendilab/DI-engine

9a25055

fix import

ecd0119

add readme

4d54abf

drop useless codes

65525c1

change rolloutor

a1b0908

move on_league_job in call

68191a9

change __call__

59f2a6e

solve conflicts

2010965

solve conflicts

1382b7f

Merge branch 'dev-distar-actor' of https://github.com/opendilab/DI-en…

b546f8a

…gine into dev-league-lxl

add league coordinator

cfabb47

reformatting

0ff864d

Merge branch 'main' into dev-distar-actor

4cf60f9

zhumengshen added 10 commits August 24, 2022 11:02

change import of collector.py

3f21d3c

drop out useless mocks

37b6bc7

make test of coordinator pass

de8b6eb

feature(zms): add test_league_learner_communicator.py

e63d710

change a bit

30e6249

change file name

9dc8770

update test_handle_step_exception.py

117ae79

update test of BattleTransitionList, and add last_step_fn in BattleTr…

f66248d

…ansitionList, make battle_rolloutor_for_distar run in dict and list cases

uupdate tests; actor, collector, functional collector

8cf4a82

remove one todo

9af8168

hiha3456 requested a review from PaParaZz1 August 26, 2022 07:29

hiha3456 marked this pull request as draft August 26, 2022 07:30

zhumengshen added 5 commits August 26, 2022 07:46

reformat

6221024

reformat

16281d9

fix bug

fb83c8e

reformat; add last_step_fn in entry of LeagueActor and BattleStepColl…

bc7b477

…ector

update tests

7fc8354

PaParaZz1 requested changes Aug 27, 2022

View reviewed changes

PaParaZz1 added the enhancement New feature or request label Aug 27, 2022

delete useless files

3e01213

PaParaZz1 requested changes Sep 14, 2022

View reviewed changes

PaParaZz1 marked this pull request as ready for review September 21, 2022 10:50

hiha3456 added 5 commits September 22, 2022 07:49

add unittest of flatten and detach_grad

0a8be2e

add comments about the difference between GLU2 and GLU

b3d39f9

add unittest of parameter "dim" in default_collate; remove dim from c…

4f09857

…at because it is useless

reformat

54ea00e

usee pytest of test_sparse_logging

ef64a7e

PaParaZz1 requested changes Sep 23, 2022

View reviewed changes

change format of comment of sparse_logging

543f1ec

PaParaZz1 mentioned this pull request Apr 13, 2023

Roadmap for DI-engine #548

Open

		self.keep('train_iter', 'last_eval_iter')


		class BattleContext(Context):

		return output


		def scatter_connection_v2(shape, project_embeddings, entity_location, scatter_dim, scatter_type='add'):

Conversation

hiha3456 commented Aug 26, 2022

Description

Related Issue

TODO

Check List

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants