General developer guide

This page is supposed to give an overview of the development process set up for the NRP Core repository.

Testing

The testing framework of our choice is Google Test. To execute our tests we use ctest, which is a test runner provided by CMake.

Please note that all commands in this section should be executed from the build directory!

By default, building of the tests is enabled. To disable it, add -DENABLE_TESTING=OFF to the cmake step of the build process.

The simplest way to run all available tests, after the build and installation process is complete:

make test

or, equivalently:

ctest

To run a single test or a group of tests you can use the ctest regex (-R) option (-VV runs the tests in verbose mode):

ctest -VV -R FTILoopTest.RunLoop  # Run a single test from the FTILoopTest group
ctest -VV -R FTILoopTest          # Run all tests from the FTILoopTest group

Every NRP Core module should have its own tests compiled into a single executable. The executable should be named ${PROJECT_NAME}Tests.

To find all test executables:

find . -name "*Tests"

which should give an output similar to this:

./nrp_engine_protocols/nrp_grpc_engine_protocol/NRPGRPCEngineProtocolTests
./nrp_engine_protocols/nrp_json_engine_protocol/NRPJSONEngineProtocolTests
./nrp_general_library/NRPGeneralLibraryTests
./nrp_gazebo_engines/nrp_gazebo_grpc_engine/NRPGazeboGrpcEngineTests
./nrp_gazebo_engines/nrp_gazebo_json_engine/NRPGazeboJSONEngineTests
./nrp_simulation/NRPCoreSimTests
./nrp_python_json_engine/NRPPythonJSONEngineTests
./nrp_nest_engines/nrp_nest_json_engine/NRPNestJSONEngineTests

To run a single executable, which contains all tests for the module:

./nrp_general_library/NRPGeneralLibraryTests

To run a single test you can use the gtest filtering capability:

nrp_general_library/NRPGeneralLibraryTests --gtest_filter=InterpreterTest.TestTransceiverFcnDataPacks

Generating and examining core dumps

Core dumps are files that contain the memory of a process at the moment of unusual termination (usually a segmentation fault). They can be really helpful whenever NRPCoreSim or engine crashes. Core dumps allow you to analyse the stacks of all threads of the process at the moment of the crash, which may give you some clues about the problem’s origins. Core dumps can also be generated manually, which may be useful whenever your process is hanging.

In order to have full stack information after the crash you should compile the project in debug mode:

cmake .. -DCMAKE_BUILD_TYPE=Debug -DCMAKE_INSTALL_PREFIX=/home/nrp/.local/nrp/

Bear in mind that this will disable optimizations and the code will be much slower.

By default, core dumps for non-package executables will not be generated on Ubuntu. To enable core dump generation in the working directory of the experiment:

ulimit -c unlimited
sudo sysctl -w kernel.core_pattern=core.%u.%p.%t

More information about core dump generation can be found under this link

To inspect a core dump:

gdb <executable_name> <core_file>

Where executable_name will be NRPCoreSim or one of the server executables (like NRPNestJSONExecutable or gzserver) and core_file is the name of the core dump that you want to inspect.

To see the stack (backtrace) of the offending thread type this while inside gdb:

(gdb) bt

In some cases it may be useful to see stacks of all threads:

(gdb) thread apply all bt

To generate core dump manually (for example when a process is hanging):

kill -ABRT <process_pid>

Logger usage

For logging we use the own wrapper for the fast thread-safe logger SpdLog.

In order to enable logging functionality, add to the code:

#include "nrp_general_library/utils/nrp_logger.h"

The logger has the following calls for printing the logs of corresponding severity:

NRPLogger::debug("debug message");
NRPLogger::debug("formatted string debug message {}", "Hello world!");
NRPLogger::info("info message");
NRPLogger::info("formatted decimal info message {0:d}", 22);
NRPLogger::warn("warn message");
NRPLogger::warn("formatted binary warn message {0:b}", 42);
NRPLogger::error("error message");
NRPLogger::error("formatted float error message {:03.2f}", 3.14);
NRPLogger::critical("critical message");
NRPLogger::critical("{:>30}", "right aligned critical message");

and, separately, trace level

NRP_LOGGER_TRACE("trace message");
NRP_LOGGER_TRACE("formatted string trace message {}", __FUNCTION__);

The macro NRP_LOGGER_TRACE can be totally voided if PRODUCTION_RELEASE is defined at compilation. This allows hiding all trace log calls (created by NRP_LOGGER_TRACE) from the compiled code.

Each logger can be initialized with explicitly or default parameters. This behaviour is determined by the NRPLogger constructor that is called. The settings can be defined explicitly in the following constructor:

NRPLogger(
    std::string loggerName,
    NRPLogger::level_t fileLogLevel,
    NRPLogger::level_t consoleLogLevel,
    std::string logDir,
    bool doSavePars = false);

The name of the logger, loggerName, is displayed in the log message and is appended to the log file name. The corresponding minimum log levels can be set for both file and console (fileLogLevel and consoleLogLevel). The parameter logDir specifies the location of the log files with respect to the working directory (or may be set as absolute path). The doSavePars flag allows this constructor to propagate the logger settings or consume them from the shared memory. In case this flag is true, then the constructor saves settings into the shared memory, otherwise the constructor tries to load them.

The creation of the logger with the default parameters can be done with another constructor:

NRPLogger(
    std::string loggerName = _defaultLoggerName.data());

Even if the loggerName is not specified at the call, it will be set with the default value _defaultLoggerName = "nrp_core". The value of the other parameters is determined in the constructor definition:

NRPLogger::NRPLogger(
    std::string loggerName)
    : NRPLogger(
        loggerName,
        NRPLogger::level_t::off,
        NRPLogger::level_t::info,
        _defaultLogDir.data(),
        false) {}

Note, that using this constructor doesn’t allow saving the settings of the logger (doSavePars = false). This constructor will always try to load them from the memory.

Currently, only the logger in the NRPCoreSim executable is initialized with explicit constructor (which is parametrized by the console parameters). And only this logger tries to save its settings to the shared memory object. The other loggers (in engine servers) try to fetch the settings from the shared memory object and apply them. In case they can’t, the default settings are applied. Thus, the child processes of the launcher inherit its logger settings by the following workflow:

  1. The launcher creates the first logger and initializes it with parameters from the console (if any, or with default ones if they are absent).

  2. The resulting settings from the launcher logger are saved into the shared memory

  3. When the forked process starts, it creates its own logger.

  4. The process tries to find the shared object with settings and get them from there

  5. If something goes wrong with the shared object, the logger is initialized with the default settings (only a message is given, that it couldn’t load settings, the process is not terminated).

Here are the optional console parameters that are used to define the logger settings:

  • --cloglevel <VAL> defines the console log level;

  • --floglevel <VAL> defines the file log level;

  • --logdir <VAL> defines the directory for the log files;

  • -l,--logconfig print the simulation configuration to DEBUG log;

The values for the level parameters can be any of trace, debug, info, warn, error, critical.

Finally, after the NRPLogger object is created, it should be at some point deleted. Note, that the destructor of the NRPLogger closes all spdlog sinks and, thus, disables the following logging. The NRPLogger object should be deleted only at the end of the operation and only one NRPLogger should be created within the process.

Static code analysis

Currently we support cppcheck as the static code analysis tool. It is integrated into our build system. Before you can use it, you will have to install it:

sudo apt install cppcheck

You can run it from the build directory with:

make cppcheck

Time Profiler

There are two macros available for time profiling: NRP_LOG_TIME and NRP_LOG_TIME_BLOCK. Both macros are only activated if TIME_PROFILE cmake variable is defined at compilation time. This allows easily hiding all time profile calls from the compiled code if wished.

NRP_LOG_TIME takes a filename parameter and records in a file with that name (and .log extension) the time difference, expressed in microseconds, between the clock time at the moment of calling and a fix time point. For example the next call:

NRP_LOG_TIME("my_time_point");

will add a record with the aforementioned time difference to a file named my_time_point.log.

NRP_LOG_TIME_BLOCK functions in a similar way than NRP_LOG_TIME but records the duration between the moment of calling and the end of the current block, ie. when a created helper object goes out of scope. For example:

{
    NRP_LOG_TIME_BLOCK("my_time_duration");
    // ... here some important code
}

will add a record with the duration, also expressed in microseconds, until the execution reaches the end of the block to a file named my_time_duration.log.

All time log files are stored in a subfolder time_logs of the experiment working directory.

Finally, both macros are defined in nrp_general_library/utils/time_utils.h, which must be included in order to use them.

To enable these macros configure nrp-core with -DENABLE_TIME_PROFILE=ON :

cmake -DENABLE_TIME_PROFILE=ON <other configuration parameters> ..

Debugging gRPC engines

gRPC troubleshooting guide