.. index:: pair: page; Creating a new Engine from scratch .. _doxid-tutorial_engine_creation: Creating a new Engine from scratch ================================== As explained in :ref:`apposite section of the architecture overview `, engines are a core aspect of the NRP-core framework. They run the various (and possibly heterogeneous) components/modules of the simulations, with the Simulation Loop and TransceiverFunctions merely being a way to synchronize and exchange data therebetween. In other terms, the NRP-core facilitates communication between differing simulator types in order to integrate them into a single coherent simulation. We aim to achieve predictable behaviour even in cases where simulators with different execution schemes are deployed. This requires a strict engine interface, which will properly synchronize runtime and data exchange. The NRP has adopted a client-server approach to this problem, together with constraints in terms of synchronous communications. Each simulator runs in its own process, and acts as a server. The Simulation Loop manages synchronization, and accesses each engine as a client. Data exchange is facilitated via :ref:`DataPacks `. Therefore, a developer wishing to create a new engine **must supply five components** : * :ref:`Engine Server ` * :ref:`EngineClient ` * :ref:`ProcessLauncher ` * :ref:`Engine configuration schema ` * Only if required, custom :ref:`DataPack python bindings ` In the next sections we comment how to proceed to implement each of this components. The code samples in these sections are based on a bare bone example engine included in the folder *docs/example_engine*. Please note that this guide describes the steps needed to create an engine *from scratch*. It will take a considerable amount of development time, and the exact implementation will depend on the communication protocol and data structures of your choice. In order to learn how to base your new engine implementation in one of the provided engine templates see this guide: :ref:`Creating a new Engine from template ` Should you wish to integrate a simulator with a Python interface in NRP-core, we also supply a :ref:`PythonJSONEngine `, which can execute arbitrary Python scripts. .. _doxid-tutorial_engine_creation_1tutorial_engine_creation_directories: Directory tree ~~~~~~~~~~~~~~ We propose to structure source files of the new engine in the following way: .. ref-code-block:: cpp example_engine/ ├── cmake │   └── ProjectConfig.cmake.in ├── CMakeLists.txt ├── example_engine_server_executable │   ├── example_engine_server_executable.cpp │   ├── example_engine_server_executable.h │   └── main.cpp └── nrp_example_engine ├── config │   ├── cmake_constants.h.in │   ├── example_config.h │   └── example_config.json ├── engine_server │   ├── example_engine_server.cpp │   └── example_engine_server.h ├── nrp_client │   ├── example_engine_client.cpp │   └── example_engine_client.h └── python ├── example_engine_python.cpp └── __init__.py.in * root - root directory of the new engine, the right place to put your ``CMakeLists.txt`` * cmake - helper files for cmake * example_engine_server_executable - source files related to server executable * config - source files related to engine configuration * engine_server - source code of the server side of the engine * nrp_client - source code of the client side of the engine * python - Python module with Python wrappers for datapack classes .. _doxid-tutorial_engine_creation_1tutorial_engine_creation_cmake: Setting up CMake ~~~~~~~~~~~~~~~~ We use CMake to manage project compilation. Below is described the basic structure used in an Engine cmake configuration file in order to create all libraries and executables necessary for the new engine. The code samples are taken from docs/example_engine/CMakeLists.txt, which can be used as a template. Create basic variable definitions. These will be used in the code later on .. ref-code-block:: cpp set(PROJECT_NAME "NRPExampleEngine") set(HEADER_DIRECTORY "nrp_example_engine") set(NAMESPACE_NAME "${PROJECT_NAME}") set(LIBRARY_NAME "${PROJECT_NAME}") set(PYTHON_MODULE_NAME "example_engine") set(EXECUTABLE_NAME "NRPExampleServerExecutable") set(TEST_NAME "${PROJECT_NAME}Tests") set(ENV{NRP_ENGINE_LAUNCHERS} "${LIBRARY_NAME}.so;$ENV{NRP_ENGINE_LAUNCHERS}") set(LIB_EXPORT_NAME "${LIBRARY_NAME}Targets") set(LIB_CONFIG_NAME "${LIBRARY_NAME}Config") set(LIB_VERSION_NAME "${LIB_CONFIG_NAME}Version") List Cpp compile files. LIB_SRC_FILES should contain files required by the new :ref:`EngineClient ` and Engine Server, PYTHON_MODULE_SRC_FILES should contain files required for integrating datapacks into TransceiverFunctions, and EXEC_SRC_FILES should contain files required by the forked Engine Server process, in particular the source file containing the :ref:`main() ` function. .. ref-code-block:: cpp # List library build files set(LIB_SRC_FILES nrp_example_engine/engine_server/example_engine_server.cpp nrp_example_engine/nrp_client/example_engine_client.cpp ) # List of Python module build files set(PYTHON_MODULE_SRC_FILES nrp_example_engine/python/example_engine_python.cpp ) # List executable build files set(EXEC_SRC_FILES example_engine_server_executable/:ref:`main `.cpp example_engine_server_executable/example_engine_server_executable.cpp ) # List testing build files set(TEST_SRC_FILES ) Create configuration files. These files use CMake variables to insert compile-time information into the source code, mainly things such as the install location, library names, ... .. ref-code-block:: cpp ## Header configuration # General Header defines set(NRP_EXAMPLE_EXECUTABLE ${EXECUTABLE_NAME}) configure_file("nrp_example_engine/config/cmake_constants.h.in" "${CMAKE_CURRENT_BINARY_DIR}/include/${HEADER_DIRECTORY}/config/cmake_constants.h" @ONLY) # Python module dependencies configure_file("nrp_example_engine/python/__init__.py.in" "${CMAKE_CURRENT_BINARY_DIR}/src/__init__.py" @ONLY) Add a library target. This instructs CMake to create a library object containing the source files defined in LIB_SRC_FILES. In addition, it links the new library to ${NRP_GEN_LIB_TARGET}, which is NRPGeneralLibrary.so, the base NRP library. .. ref-code-block:: cpp ## NRPExampleEngineLibrary add_library("${LIBRARY_NAME}" SHARED ${LIB_SRC_FILES}) add_library(${NAMESPACE_NAME}::${LIBRARY_NAME} ALIAS ${LIBRARY_NAME}) target_compile_options(${LIBRARY_NAME} PUBLIC $<$,$>:${NRP_COMMON_COMPILATION_FLAGS}>) target_compile_options(${LIBRARY_NAME} PUBLIC $<$:-fconcepts>) set_target_properties(${LIBRARY_NAME} PROPERTIES PREFIX "") target_link_libraries(${LIBRARY_NAME} PUBLIC ${NRP_GEN_LIB_TARGET} NRPJSONEngineProtocol::NRPJSONEngineProtocol PRIVATE ) target_include_directories(${LIBRARY_NAME} BEFORE PUBLIC "$" "$" "$" PRIVATE ) Add a Python module target. With this, a new library will be created which can be used as a Python module. The proceeding install code will install the new module at the correct location, so that it can be accessed by :ref:`TransceiverFunctions `. .. ref-code-block:: cpp ## example_engine if(NOT ${PYTHON_MODULE_SRC_FILES} STREQUAL "") add_library(${PYTHON_MODULE_NAME} SHARED ${PYTHON_MODULE_SRC_FILES}) add_library(${NAMESPACE_NAME}::${PYTHON_MODULE_NAME} ALIAS ${PYTHON_MODULE_NAME}) target_compile_options(${PYTHON_MODULE_NAME} PRIVATE $<$,$>:${NRP_COMMON_COMPILATION_FLAGS}>) set_target_properties(${PYTHON_MODULE_NAME} PROPERTIES PREFIX "") target_include_directories(${PYTHON_MODULE_NAME} PUBLIC ) target_link_libraries(${PYTHON_MODULE_NAME} PUBLIC ${NAMESPACE_NAME}::${LIBRARY_NAME} ) endif() Add an executable target. This will compile a new executable which can be executed in a forked process to run an Engine Server along with a simulation. .. ref-code-block:: cpp ## NRPExampleServerExecutable if(NOT "${EXEC_SRC_FILES}" STREQUAL "") add_executable(${EXECUTABLE_NAME} ${EXEC_SRC_FILES}) target_link_libraries(${EXECUTABLE_NAME} ${LIBRARY_NAME}) endif() Add installation instructions. After compilation, this will instruct CMake on the correct location to install header files as well as all newly generated libraries, executables, and Python modules. .. ref-code-block:: cpp ## Installation set(INSTALL_CONFIGDIR "${CMAKE_INSTALL_LIBDIR}/cmake/${PROJECT_NAME}") # Install library files install(TARGETS ${LIBRARY_NAME} EXPORT ${LIB_EXPORT_NAME} LIBRARY DESTINATION ${NRP_PLUGIN_INSTALL_DIR} ARCHIVE DESTINATION ${NRP_PLUGIN_INSTALL_DIR} RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR} PUBLIC_HEADER DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}/${HEADER_DIRECTORY} ) # Install export target install(EXPORT ${LIB_EXPORT_NAME} DESTINATION ${INSTALL_CONFIGDIR} FILE "${LIB_EXPORT_NAME}.cmake" NAMESPACE "${NAMESPACE_NAME}::" ) # Install headers install(DIRECTORY "${HEADER_DIRECTORY}" "${CMAKE_CURRENT_BINARY_DIR}/include/${HEADER_DIRECTORY}" DESTINATION ${CMAKE_INSTALL_INCLUDEDIR} FILES_MATCHING PATTERN "*.h" PATTERN "*.hpp" ) # Install Python module if(TARGET ${PYTHON_MODULE_NAME}) install(TARGETS ${PYTHON_MODULE_NAME} DESTINATION "${PYTHON_INSTALL_DIR_REL}/${NRP_PYTHON_MODULE_NAME}/engines/${PYTHON_MODULE_NAME}") install(FILES "${CMAKE_CURRENT_BINARY_DIR}/src/__init__.py" DESTINATION "${PYTHON_INSTALL_DIR_REL}/${NRP_PYTHON_MODULE_NAME}/engines/${PYTHON_MODULE_NAME}") endif() # Install executable files if(TARGET ${EXECUTABLE_NAME}) install(TARGETS ${EXECUTABLE_NAME} RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR}) endif() # create cmake version and config files include(CMakePackageConfigHelpers) write_basic_package_version_file( "${CMAKE_CURRENT_BINARY_DIR}/${LIB_VERSION_NAME}.cmake" VERSION ${PROJECT_VERSION} COMPATIBILITY AnyNewerVersion ) configure_package_config_file("${CMAKE_CURRENT_LIST_DIR}/cmake/ProjectConfig.cmake.in" "${CMAKE_CURRENT_BINARY_DIR}/${LIB_CONFIG_NAME}.cmake" INSTALL_DESTINATION ${INSTALL_CONFIGDIR} ) .. _doxid-tutorial_engine_creation_1tutorial_engine_creation_engine_config: Creating an Engine configuration schema ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Engines should be configurable by users. Configuration is based on JSON documents, which are validated using `JSON schemas `__. This :ref:`page ` offers more details on configuration management in NRP-core. Every new engine configuration schema should be based on the provided basic configuration schema: .. code-block:: cpp json://nrp-core/engines/engine_base.json#/EngineBase The new engine schema can be afterwards placed into a separate JSON file in *config_schemas/engines/* folder, so it can be found at run time. Here is an example of how this might look like: .. _doxid-tutorial_engine_creation_1tutorial_engine_creation_engine_config_example: Example ------- .. ref-code-block:: cpp { "$schema": "http://json-schema.org/draft-07/schema#", "title": "Example config", "description": "Base Json Engine configuration schema", "$id": "#EngineExample", "allOf": [ { "$ref": "json://nrp-core/engines/engine_base.json#EngineBase" }, { "properties" : { "ServerAddress": { "type": "string", "default": "localhost:9002", "description": "Address from which the engine server sends/receives data" }, "RegistrationServerAddress": { "type": "string", "default": "localhost:9001", "description": "Address to which servers should register to" } } } ] } .. _doxid-tutorial_engine_creation_1tutorial_engine_creation_engine_config_linking: Linking configuration schema to the engine ------------------------------------------ To use the newly created schema, it has to be linked to the engine client. This is done by passing the schema URI as template argument to the base class of your new engine: .. ref-code-block:: cpp class ExampleEngineClient : public EngineClient A further explanation of how schema URIs are structured can be found in the section :ref:`Referencing schemas `. .. _doxid-tutorial_engine_creation_1tutorial_engine_creation_engine_datapack: DataPack data ~~~~~~~~~~~~~ DataPacks are object that facilitate exchange of data between transceiver functions and simulators. A more detailed description of datapacks can be found in this :ref:`page `. They consists of some :ref:`data structure `, used as payload in data exchanges between Engine servers and clients, and :ref:`metadata `, used to uniquely identify the datapack and relate it to a specific engine. There is in principle no restrictions in the type a :ref:`DataPack ` can store, as long as Engine client and server are able to exchange them over the wire. But since DataPacks also must be available inside of TransceiverFunctions, python bindings must be available for each data type used in DataPacks. To reduce the complexity of developping new engines, it is strongly recommended to use one of the data types for which NRP-core already provide python bindings. These are: nlohmann::json and protobuf messages. However, if you decide to use another data type you must implement Python bindings for it and make them available to NRP-core as a Python module. .. _doxid-tutorial_engine_creation_1tutorial_engine_creation_engine_client: Creating an EngineClient ~~~~~~~~~~~~~~~~~~~~~~~~ An :ref:`EngineClient ` is used by the Simulation Loop to interface with a simulator via an Engine Server. A communication protocol is required to facilitate data exchange. We provide a set of predefined protocol implementations here. In most cases, using one of these as a base template suffices and greatly reduces development efforts. A new engine client must inherit from the :ref:`EngineClient ` class. As such, it may look as shown below. A detailed function description can be found in :ref:`EngineClientInterface `. A set of methods need to be implemented in the new client class. These methods will be called by the Simulation Loop in various points of the loop. .. _doxid-tutorial_engine_creation_1tutorial_engine_creation_simulation_control_methods: Simulation control and state methods ------------------------------------ * :ref:`EngineClientInterface::initialize() ` - should perform all necessary steps (requests to the server) to initialize the simulation. The function is called before the simulation loop starts. * :ref:`EngineClient::runLoopStepCallback() ` - callback method called from :ref:`EngineClient::runLoopStepAsync() ` which should request the server to run a simulation step with specified timestep. The request is performed in a separate thread. This allows all engines to run their steps in parallel. :ref:`EngineClientInterface::runLoopStepAsyncGet() ` must be used to join the threads again. * :ref:`EngineClientInterface::shutdown() ` - should request the server to perform cleanup, before the server process is requested to terminate. * :ref:`EngineClientInterface::reset() ` - should request the server to reset the simulation. .. _doxid-tutorial_engine_creation_1tutorial_engine_creation_data_exchange_methods: Data exchange methods --------------------- * :ref:`EngineClientInterface::getDataPacksFromEngine() ` - may be used to request results of the latest step from the simulator. Received data should be deserialized into proper datapack types, which will be consumed by transceiver functions. The function will be called before runLoopStepAsync. * :ref:`EngineClientInterface::sendDataPacksToEngine() ` - may be used to pass relevant data, like reference values, to the simulator. Input to the functions will be a list of datapacks, (results of transceiver function execution). The datapacks need to be serialized into structures used by the communication protocol between engine client and server. The function will be called after runLoopStepAsync. .. _doxid-tutorial_engine_creation_1tutorial_engine_creation_simulation_spawning_methods: Simulation process spawning methods ----------------------------------- These methods are used by the process launcher. * :ref:`EngineClientInterface::engineProcStartParams() ` - should return all startup parameters of the simulation process. Related to :ref:`EngineProcStartParams ` config parameter. .. _doxid-tutorial_engine_creation_1tutorial_engine_creation_engine_client_example: Example ------- .. ref-code-block:: cpp #ifndef EXAMPLE_ENGINE_CLIENT_H #define EXAMPLE_ENGINE_CLIENT_H #include "nrp_example_engine/config/example_config.h" #include "nrp_general_library/engine_interfaces/engine_client_interface.h" #include "nrp_general_library/plugin_system/plugin.h" class ExampleEngineClient : public :ref:`EngineClient ` { public: ExampleEngineClient(:ref:`nlohmann::json ` &config, :ref:`ProcessLauncherInterface::unique_ptr ` &&launcher); void :ref:`initialize `() override; void :ref:`reset `() override; void :ref:`shutdown `() override; void :ref:`sendDataPacksToEngine `(const :ref:`datapacks_set_t ` &dataPacks) override; :ref:`datapacks_vector_t ` :ref:`getDataPacksFromEngine `(const :ref:`datapack_identifiers_set_t ` &datapackIdentifiers) override; const std::vector :ref:`engineProcStartParams `() const override; protected: :ref:`SimulationTime ` :ref:`runLoopStepCallback `(:ref:`SimulationTime ` timeStep) override; }; using ExampleEngineLauncher = ExampleEngineClient::EngineLauncher; :ref:`CREATE_NRP_ENGINE_LAUNCHER `(ExampleEngineLauncher); #endif // EXAMPLE_ENGINE_CLIENT_H .. _doxid-tutorial_engine_creation_1tutorial_engine_creation_python: Creating a Python module ~~~~~~~~~~~~~~~~~~~~~~~~ The Simulation Loop and engines are written in C++, but transceiver functions are written in Python. We need a way of wrapping C++ code with Python, particularly for datapack data types which will be used in TFs. This is done inside so called Python module. Most of the wrappers are already defined in the base Python module, but wrappers for new datapack types must be added. .. ref-code-block:: cpp namespace python = boost::python; :ref:`BOOST_PYTHON_MODULE `(PYTHON_MODULE_NAME) { // Import the base Python module python::import(PYTHON_MODULE_NAME_STR); } .. _doxid-tutorial_engine_creation_1tutorial_engine_creation_engine_proc_launcher: Creating a new ProcessLauncher ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The NRP runs multiple simulators. To keep their runtime environment separate, each simulator runs in its own process. At startup, the NRP forks an additional process for each engine. This is the purpose of the :ref:`ProcessLauncher `. Usually, developers can use the default launcher and won't have to implement their own. However, should the need arise, a developer can define his own :ref:`LaunchCommand `. We recommend using the :ref:`BasicFork ` class as a starting template, and modify it to fit the specific engine's needs. .. _doxid-tutorial_engine_creation_1tutorial_engine_creation_engine_server: Creating an Engine Server ~~~~~~~~~~~~~~~~~~~~~~~~~ An Engine Server runs in its own process, executes the simulation, and exchanges data with the Simulation Loop via the :ref:`EngineClient `. To interface with said client, a communication protocol is required. We provide a set of predefined protocol implementations here. If your simulator provides a dedicated server, you may use it directly, by specifying path to the executable in :ref:`EngineProcCmd ` config parameter. An example of engine using a server provided by the simulator is our nrp_nest_server_engine. If no dedicated server exists for your simulator, you will need to create it. Generally, the server must be able to handle requests from the following client methods: * initialize - initialize the simulation with parameters coming from the client * shutdown - shutdown the simulation * reset - reset the simulation * runLoopStepAsync - run step of the simulation with step duration requested by the client * getDataPacksFromEngine - return data from the last simulation step to the client * sendDataPacksToEngine - retrieve data for the next simulation step from the client The Engine Server must also define a :ref:`main() ` function to execute. Path to the executable should be specified in :ref:`EngineProcCmd ` config parameter.