The model-based reinforcement learning (MBRL) paradigm, which uses planning algorithms, has recently achieved unprecedented results in the area of DRL. These agents are quite complex and involve multiple components, factors that can create challenges for research. In this work, we propose a modular software architecture (our implementation can be found in https://github.com/GaspTO/Modular_MBRL) suited for these types of agents, which makes possible the implementation of different algorithms and for each component to be easily configured (such as different exploration policies, search algorithms...). We illustrate the use of this architecture by implementing several algorithms and experimenting with agents created using different combinations of these. We also suggest a new simple search algorithm called averaged minimax that achieved good results in this work. Our experiments also show that the best algorithm combination is problem-dependent.