Creating a new Engine from scratch

As explained in apposite section of the architecture overview, engines are a core aspect of the NRP-core framework. They run the various (and possibly heterogeneous) components/modules of the simulations, with the Simulation Loop and TransceiverFunctions merely being a way to synchronize and exchange data therebetween. In other terms, the NRP-core facilitates communication between differing simulator types in order to integrate them into a single coherent simulation. We aim to achieve predictable behaviour even in cases where simulators with different execution schemes are deployed. This requires a strict engine interface, which will properly synchronize runtime and data exchange.

The NRP has adopted a client-server approach to this problem, together with constraints in terms of synchronous communications. Each simulator runs in its own process, and acts as a server. The Simulation Loop manages synchronization, and accesses each engine as a client. Data exchange is facilitated via DataPacks. Therefore, a developer wishing to create a new engine must supply five components :

In the next sections we comment how to proceed to implement each of this components. The code samples in these sections are based on a bare bone example engine included in the folder docs/example_engine.

Please note that this guide describes the steps needed to create an engine from scratch. It will take a considerable amount of development time, and the exact implementation will depend on the communication protocol and data structures of your choice. In order to learn how to base your new engine implementation in one of the provided engine templates see this guide: Creating a new Engine from template Should you wish to integrate a simulator with a Python interface in NRP-core, we also supply a PythonJSONEngine, which can execute arbitrary Python scripts.

Directory tree

We propose to structure source files of the new engine in the following way:

example_engine/
├── cmake
│   └── ProjectConfig.cmake.in
├── CMakeLists.txt
├── example_engine_server_executable
│   ├── example_engine_server_executable.cpp
│   ├── example_engine_server_executable.h
│   └── main.cpp
└── nrp_example_engine
    ├── config
    │   ├── cmake_constants.h.in
    │   ├── example_config.h
    │   └── example_config.json
    ├── engine_server
    │   ├── example_engine_server.cpp
    │   └── example_engine_server.h
    ├── nrp_client
    │   ├── example_engine_client.cpp
    │   └── example_engine_client.h
    └── python
        ├── example_engine_python.cpp
        └── __init__.py.in

  • root - root directory of the new engine, the right place to put your CMakeLists.txt

  • cmake - helper files for cmake

  • example_engine_server_executable - source files related to server executable

  • config - source files related to engine configuration

  • engine_server - source code of the server side of the engine

  • nrp_client - source code of the client side of the engine

  • python - Python module with Python wrappers for datapack classes

Setting up CMake

We use CMake to manage project compilation. Below is described the basic structure used in an Engine cmake configuration file in order to create all libraries and executables necessary for the new engine. The code samples are taken from docs/example_engine/CMakeLists.txt, which can be used as a template.

Create basic variable definitions. These will be used in the code later on

set(PROJECT_NAME "NRPExampleEngine")
set(HEADER_DIRECTORY "nrp_example_engine")

set(NAMESPACE_NAME "${PROJECT_NAME}")

set(LIBRARY_NAME "${PROJECT_NAME}")
set(PYTHON_MODULE_NAME "example_engine")
set(EXECUTABLE_NAME "NRPExampleServerExecutable")
set(TEST_NAME "${PROJECT_NAME}Tests")
set(ENV{NRP_ENGINE_LAUNCHERS} "${LIBRARY_NAME}.so;$ENV{NRP_ENGINE_LAUNCHERS}")

set(LIB_EXPORT_NAME "${LIBRARY_NAME}Targets")
set(LIB_CONFIG_NAME "${LIBRARY_NAME}Config")
set(LIB_VERSION_NAME "${LIB_CONFIG_NAME}Version")

List Cpp compile files. LIB_SRC_FILES should contain files required by the new EngineClient and Engine Server, PYTHON_MODULE_SRC_FILES should contain files required for integrating datapacks into TransceiverFunctions, and EXEC_SRC_FILES should contain files required by the forked Engine Server process, in particular the source file containing the main() function.

# List library build files
set(LIB_SRC_FILES
    nrp_example_engine/engine_server/example_engine_server.cpp
    nrp_example_engine/nrp_client/example_engine_client.cpp
)

# List of Python module build files
set(PYTHON_MODULE_SRC_FILES
    nrp_example_engine/python/example_engine_python.cpp
)

# List executable build files
set(EXEC_SRC_FILES
    example_engine_server_executable/main.cpp
    example_engine_server_executable/example_engine_server_executable.cpp
)

# List testing build files
set(TEST_SRC_FILES
)

Create configuration files. These files use CMake variables to insert compile-time information into the source code, mainly things such as the install location, library names, …

## Header configuration

# General Header defines
set(NRP_EXAMPLE_EXECUTABLE ${EXECUTABLE_NAME})
configure_file("nrp_example_engine/config/cmake_constants.h.in" "${CMAKE_CURRENT_BINARY_DIR}/include/${HEADER_DIRECTORY}/config/cmake_constants.h" @ONLY)

# Python module dependencies
configure_file("nrp_example_engine/python/__init__.py.in" "${CMAKE_CURRENT_BINARY_DIR}/src/__init__.py" @ONLY)

Add a library target. This instructs CMake to create a library object containing the source files defined in LIB_SRC_FILES. In addition, it links the new library to ${NRP_GEN_LIB_TARGET}, which is NRPGeneralLibrary.so, the base NRP library.

## NRPExampleEngineLibrary
add_library("${LIBRARY_NAME}" SHARED ${LIB_SRC_FILES})
add_library(${NAMESPACE_NAME}::${LIBRARY_NAME} ALIAS ${LIBRARY_NAME})
target_compile_options(${LIBRARY_NAME} PUBLIC $<$<OR:$<CXX_COMPILER_ID:Clang>,$<CXX_COMPILER_ID:GNU>>:${NRP_COMMON_COMPILATION_FLAGS}>)
target_compile_options(${LIBRARY_NAME} PUBLIC $<$<CXX_COMPILER_ID:GNU>:-fconcepts>)

set_target_properties(${LIBRARY_NAME} PROPERTIES PREFIX "")

target_link_libraries(${LIBRARY_NAME}
    PUBLIC
        ${NRP_GEN_LIB_TARGET}
        NRPJSONEngineProtocol::NRPJSONEngineProtocol

    PRIVATE
)

target_include_directories(${LIBRARY_NAME} BEFORE
    PUBLIC
        "$<INSTALL_INTERFACE:${CMAKE_INSTALL_INCLUDEDIR}>"
        "$<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}>"
        "$<BUILD_INTERFACE:${CMAKE_CURRENT_BINARY_DIR}/include>"

    PRIVATE
)

Add a Python module target. With this, a new library will be created which can be used as a Python module. The proceeding install code will install the new module at the correct location, so that it can be accessed by TransceiverFunctions.

## example_engine
if(NOT ${PYTHON_MODULE_SRC_FILES} STREQUAL "")
    add_library(${PYTHON_MODULE_NAME} SHARED ${PYTHON_MODULE_SRC_FILES})
    add_library(${NAMESPACE_NAME}::${PYTHON_MODULE_NAME} ALIAS ${PYTHON_MODULE_NAME})
    target_compile_options(${PYTHON_MODULE_NAME} PRIVATE $<$<OR:$<CXX_COMPILER_ID:Clang>,$<CXX_COMPILER_ID:GNU>>:${NRP_COMMON_COMPILATION_FLAGS}>)
    set_target_properties(${PYTHON_MODULE_NAME} PROPERTIES PREFIX "")

    target_include_directories(${PYTHON_MODULE_NAME}
        PUBLIC
    )

    target_link_libraries(${PYTHON_MODULE_NAME}
        PUBLIC
            ${NAMESPACE_NAME}::${LIBRARY_NAME}
    )
endif()

Add an executable target. This will compile a new executable which can be executed in a forked process to run an Engine Server along with a simulation.

## NRPExampleServerExecutable
if(NOT "${EXEC_SRC_FILES}" STREQUAL "")
    add_executable(${EXECUTABLE_NAME} ${EXEC_SRC_FILES})
    target_link_libraries(${EXECUTABLE_NAME} ${LIBRARY_NAME})
endif()

Add installation instructions. After compilation, this will instruct CMake on the correct location to install header files as well as all newly generated libraries, executables, and Python modules.

## Installation

set(INSTALL_CONFIGDIR "${CMAKE_INSTALL_LIBDIR}/cmake/${PROJECT_NAME}")

# Install library files
install(TARGETS
        ${LIBRARY_NAME}
    EXPORT
        ${LIB_EXPORT_NAME}
    LIBRARY DESTINATION ${NRP_PLUGIN_INSTALL_DIR}
    ARCHIVE DESTINATION ${NRP_PLUGIN_INSTALL_DIR}
    RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR}

    PUBLIC_HEADER DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}/${HEADER_DIRECTORY}
)

# Install export target
install(EXPORT ${LIB_EXPORT_NAME}
    DESTINATION
        ${INSTALL_CONFIGDIR}
    FILE
        "${LIB_EXPORT_NAME}.cmake"
    NAMESPACE
        "${NAMESPACE_NAME}::"
)

# Install headers
install(DIRECTORY "${HEADER_DIRECTORY}" "${CMAKE_CURRENT_BINARY_DIR}/include/${HEADER_DIRECTORY}"
    DESTINATION
        ${CMAKE_INSTALL_INCLUDEDIR}
    FILES_MATCHING
        PATTERN "*.h"
        PATTERN "*.hpp"
)

# Install Python module
if(TARGET ${PYTHON_MODULE_NAME})
    install(TARGETS ${PYTHON_MODULE_NAME}
        DESTINATION "${PYTHON_INSTALL_DIR_REL}/${NRP_PYTHON_MODULE_NAME}/engines/${PYTHON_MODULE_NAME}")

    install(FILES "${CMAKE_CURRENT_BINARY_DIR}/src/__init__.py"
        DESTINATION "${PYTHON_INSTALL_DIR_REL}/${NRP_PYTHON_MODULE_NAME}/engines/${PYTHON_MODULE_NAME}")
endif()

# Install executable files
if(TARGET ${EXECUTABLE_NAME})
    install(TARGETS ${EXECUTABLE_NAME}
        RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR})
endif()

# create cmake version and config files
include(CMakePackageConfigHelpers)
write_basic_package_version_file(
    "${CMAKE_CURRENT_BINARY_DIR}/${LIB_VERSION_NAME}.cmake"
    VERSION ${PROJECT_VERSION}
    COMPATIBILITY AnyNewerVersion
)

configure_package_config_file("${CMAKE_CURRENT_LIST_DIR}/cmake/ProjectConfig.cmake.in"
    "${CMAKE_CURRENT_BINARY_DIR}/${LIB_CONFIG_NAME}.cmake"
    INSTALL_DESTINATION ${INSTALL_CONFIGDIR}
)

Creating an Engine configuration schema

Engines should be configurable by users. Configuration is based on JSON documents, which are validated using JSON schemas. This page offers more details on configuration management in NRP-core.

Every new engine configuration schema should be based on the provided basic configuration schema:

json://nrp-core/engines/engine_base.json#/EngineBase

The new engine schema can be afterwards placed into a separate JSON file in config_schemas/engines/ folder, so it can be found at run time.

Here is an example of how this might look like:

Example

{
    "$schema": "http://json-schema.org/draft-07/schema#",
    "title": "Example config",
    "description": "Base Json Engine configuration schema",
    "$id": "#EngineExample",
    "allOf": [
      { "$ref": "json://nrp-core/engines/engine_base.json#EngineBase" },
      {
        "properties" : {
          "ServerAddress": {
            "type": "string",
            "default": "localhost:9002",
            "description": "Address from which the engine server sends/receives data"
          },
          "RegistrationServerAddress": {
            "type": "string",
            "default": "localhost:9001",
            "description": "Address to which servers should register to"
          }
        }
      }
    ]
}

Linking configuration schema to the engine

To use the newly created schema, it has to be linked to the engine client. This is done by passing the schema URI as template argument to the base class of your new engine:

class ExampleEngineClient
        : public EngineClient<ExampleEngineClient, "json://nrp-core/engines/engine_example.json#/EngineExample">

A further explanation of how schema URIs are structured can be found in the section Referencing schemas.

DataPack data

DataPacks are object that facilitate exchange of data between transceiver functions and simulators. A more detailed description of datapacks can be found in this page.

They consists of some data structure, used as payload in data exchanges between Engine servers and clients, and metadata, used to uniquely identify the datapack and relate it to a specific engine. There is in principle no restrictions in the type a DataPack can store, as long as Engine client and server are able to exchange them over the wire. But since DataPacks also must be available inside of TransceiverFunctions, python bindings must be available for each data type used in DataPacks. To reduce the complexity of developping new engines, it is strongly recommended to use one of the data types for which NRP-core already provide python bindings. These are: nlohmann::json and protobuf messages. However, if you decide to use another data type you must implement Python bindings for it and make them available to NRP-core as a Python module.

Creating an EngineClient

An EngineClient is used by the Simulation Loop to interface with a simulator via an Engine Server. A communication protocol is required to facilitate data exchange. We provide a set of predefined protocol implementations here. In most cases, using one of these as a base template suffices and greatly reduces development efforts.

A new engine client must inherit from the EngineClient class. As such, it may look as shown below. A detailed function description can be found in EngineClientInterface.

A set of methods need to be implemented in the new client class. These methods will be called by the Simulation Loop in various points of the loop.

Simulation control and state methods

Data exchange methods

  • EngineClientInterface::getDataPacksFromEngine() - may be used to request results of the latest step from the simulator. Received data should be deserialized into proper datapack types, which will be consumed by transceiver functions. The function will be called before runLoopStepAsync.

  • EngineClientInterface::sendDataPacksToEngine() - may be used to pass relevant data, like reference values, to the simulator. Input to the functions will be a list of datapacks, (results of transceiver function execution). The datapacks need to be serialized into structures used by the communication protocol between engine client and server. The function will be called after runLoopStepAsync.

Simulation process spawning methods

These methods are used by the process launcher.

Example

#ifndef EXAMPLE_ENGINE_CLIENT_H
#define EXAMPLE_ENGINE_CLIENT_H

#include "nrp_example_engine/config/example_config.h"
#include "nrp_general_library/engine_interfaces/engine_client_interface.h"
#include "nrp_general_library/plugin_system/plugin.h"

class ExampleEngineClient
        : public EngineClient<ExampleEngineClient, ExampleConfigConst::EngineSchema>
{
    public:

        ExampleEngineClient(nlohmann::json &config, ProcessLauncherInterface::unique_ptr &&launcher);

        void initialize() override;

        void reset() override;

        void shutdown() override;

        void sendDataPacksToEngine(const datapacks_set_t &dataPacks) override;
        datapacks_vector_t getDataPacksFromEngine(const datapack_identifiers_set_t &datapackIdentifiers) override;

        const std::vector<std::string> engineProcStartParams() const override;

    protected:

        SimulationTime runLoopStepCallback(SimulationTime timeStep) override;
};

using ExampleEngineLauncher = ExampleEngineClient::EngineLauncher<ExampleConfigConst::EngineType>;

CREATE_NRP_ENGINE_LAUNCHER(ExampleEngineLauncher);


#endif // EXAMPLE_ENGINE_CLIENT_H

Creating a Python module

The Simulation Loop and engines are written in C++, but transceiver functions are written in Python. We need a way of wrapping C++ code with Python, particularly for datapack data types which will be used in TFs. This is done inside so called Python module. Most of the wrappers are already defined in the base Python module, but wrappers for new datapack types must be added.

namespace python = boost::python;

BOOST_PYTHON_MODULE(PYTHON_MODULE_NAME)
{
    // Import the base Python module
    python::import(PYTHON_MODULE_NAME_STR);
}

Creating a new ProcessLauncher

The NRP runs multiple simulators. To keep their runtime environment separate, each simulator runs in its own process. At startup, the NRP forks an additional process for each engine. This is the purpose of the ProcessLauncher. Usually, developers can use the default launcher and won’t have to implement their own. However, should the need arise, a developer can define his own LaunchCommand. We recommend using the BasicFork class as a starting template, and modify it to fit the specific engine’s needs.

Creating an Engine Server

An Engine Server runs in its own process, executes the simulation, and exchanges data with the Simulation Loop via the EngineClient. To interface with said client, a communication protocol is required. We provide a set of predefined protocol implementations here.

If your simulator provides a dedicated server, you may use it directly, by specifying path to the executable in EngineProcCmd config parameter. An example of engine using a server provided by the simulator is our nrp_nest_server_engine.

If no dedicated server exists for your simulator, you will need to create it. Generally, the server must be able to handle requests from the following client methods:

  • initialize - initialize the simulation with parameters coming from the client

  • shutdown - shutdown the simulation

  • reset - reset the simulation

  • runLoopStepAsync - run step of the simulation with step duration requested by the client

  • getDataPacksFromEngine - return data from the last simulation step to the client

  • sendDataPacksToEngine - retrieve data for the next simulation step from the client

The Engine Server must also define a main() function to execute. Path to the executable should be specified in EngineProcCmd config parameter.