Troubleshooting

Reinstall an existing NRP

When reinstalling a whole NRP on top of an already existing one, you might run into problems that can be avoided with a thorough cleaning of old build directories. First check your .bashrc against the additional HBP commands found in this documentation. Most important is that your PYTHONPATH is set to void. Then clean up your NRP by running the following commands:

rm -rf "$HBP"/gazebo/build
rm -rf ~/.opt/platform_venv
rm -rf  "$HBP"/nrpBackendProxy/proxy_virtualenv
rm -rf ~/.opt/nrpStorage
rm -rf ~/.opt/bbp
rm ~/.spynnaker.cfg
cd "$HBP"/user_scripts && git checkout development && git pull --rebase

And then reupdate your NRP:

./update_nrp update --all

and reinstall your SpiNNaker in case you use one (Step 10).

Troubleshoot Gazebo

anaconda conflict breaks the build

If you get build errors in gazebo in the python TIFF libraries, you might have a conflict with anaconda.

undefined symbol TIFFIsTiled@LIBTIFF_4.0

You have to get anaconda out of the PATH by resetting PATH temporarily at the end of your $HOME/.bashrc, rebuild the NRP, and finally clean up your $HOME/.bashrc.

gzserver segfault on start

gzserver: /usr/include/boost/thread/pthread/recursive_mutex.hpp:101: boost::recursive_mutex::~recursive_mutex(): Assertion `!pthread_mutex_destroy(&m)' failed.

Solution: Try to launch the gzserver binary with no argument. If it runs, it probably means that you made something wrong when compiling GazeboRosPackages. Make sure that you didn’t source another catkin workspace before you built GazeboRosPackages.

gazebo cmake error (happened with AMD GPU):

CMake Error at cmake/CheckDRIDisplay.cmake:54 (STRING):
string sub-command FIND requires 3 or 4 parameters.
Call Stack (most recent call first):
CMakeLists.txt:154 (include)

--! valid dri display not found (no glxinfo or pyopengl)

Solution: install mesa-utils (glxinfo)

gazebo cmake error (missing tinyxml2): install libtinyxml2-dev.

gazebo libccd link error (/usr/bin/ld: cannot find -lccd):

You likely have a conflicting version of ros-indigo-libccd installed (from other ros packages such as MoveIt). Solution: edit cmake/SearchForStuff.cmake and replace “pkg_check_modules(CCD ccd>=1.4)” with “SET(CCD_FOUND FALSE)” delete your build directory, rerun cmake, and build as usual

Troubleshoot Nest

Come back here after NRP build if you experience issues with Nest.

Troubleshoot Gazebo Plugins (GazeboRosPackages)

Come back here if the NRP build fails on GazeboRosPackages.

Warning: If GazeboRosPackages build fails, it might be that the ROS setup file has not be sourced. If you added nrp_variables correctly in your bashrc, this should not be the case. Else, open a fresh terminal, and check that $ROS_MASTER_URI is set. If not you can manually source the setup and redo the catkin build.

source /opt/ros/noetic/setup.bash
catkin build

Troubleshoot compiling gzweb

Come back here if NRP build fails on gzweb.

To make sure that gzweb is compiled with proper Gazebo version, prepend the correct path to package configuration:

export PKG_CONFIG_PATH=$HOME/.local/lib/x86_64-linux-gnu/pkgconfig:$PKG_CONFIG_PATH

If you get node-gyp errors (“File not found”), try deleting $HOME/.npm and re-running deploy script.

You may encounter:

/usr/bin/env: node: No such file or directory
There are node-gyp build errors, exiting.

This is fixed by a symbolic link: ln -s `which nodejs` $HOME/.local/bin/node

You may also encounter core dump or segmentation faults depending on your graphics card and configuration. Install the proprietary drivers (not the X.org default driver) on your system and reboot.

You can see more possible problems and solutions here: http://gazebosim.org/tutorials?tut=gzweb_install&ver=1.9%2B&cat=gzweb#Troubleshooting

Troubleshoot Nginx

If you are having issues with nginx configuration that could mean that your nginx version is too recent. Try running

nginx -t -c $HOME/.local/etc/nginx/nginx.conf

In case the output of the command is something like ‘unknown directive “more_set_headers”’, try replacing the following lines in $HOME/.local/etc/nginx/conf.d/gzweb.conf

location / {
    #...

    #more_set_headers 'Access-Control-Allow-Origin: $cors_origin';
    #more_set_headers 'Access-Control-Allow-Methods: $cors_methods';
    #more_set_headers 'Access-Control-Allow-Headers: $cors_headers';
    #more_set_headers 'Access-Control-Expose-Headers: Content-Length';

    add_header 'Access-Control-Allow-Origin' '$cors_origin' 'always';
    add_header 'Access-Control-Allow-Methods' '$cors_methods' 'always';
    add_header 'Access-Control-Allow-Headers' '$cors_headers' 'always';
    add_header 'Access-Control-Expose-Headers' 'Content-Length' 'always';

    #...
}

and in nrp-services.conf

location / {
    #...

    #more_set_headers 'Access-Control-Allow-Origin: $cors_origin';
    #more_set_headers 'Access-Control-Allow-Methods: $cors_methods';
    #more_set_headers 'Access-Control-Allow-Headers: $cors_headers';

    add_header 'Access-Control-Allow-Origin' '$cors_origin' 'always';
    add_header 'Access-Control-Allow-Methods' '$cors_methods' 'always';
    add_header 'Access-Control-Allow-Headers' '$cors_headers' 'always';

    #...
}

location /api {
    #...

    #more_set_headers 'Access-Control-Allow-Origin: $cors_origin';
    #more_set_headers 'Access-Control-Allow-Methods: $cors_methods';
    #more_set_headers 'Access-Control-Allow-Headers: $cors_headers';

    add_header 'Access-Control-Allow-Origin' '$cors_origin' 'always';
    add_header 'Access-Control-Allow-Methods' '$cors_methods' 'always';
    add_header 'Access-Control-Allow-Headers' '$cors_headers' 'always';

    #...
}

nginx may complain that /<assets_root>/sdk does not exist. If so, just create an empty directory. No idea what is it for…

Use nginx -t -c $HOME/.local/etc/nginx/nginx.conf to get debug info if

$HOME/.local/etc/init.d/nginx restart

returns [fail].

Troubleshoot ports being used in cle-reverse-proxies

when running cle-nginx you might get an error that ports 9000 or 8080 are being used by another process. try these commands to stop default nginx and haproxy on your system:

sudo service nginx stop
sudo service haproxy stop

Troubleshoot ExDFrontend

Come back here if NRP build fails on ExDFrontend or cle-frontend fails to run.

• If you get this error in a pop up or in the web console:

  angular.js:14199 Error: UnkownConfigurationKey: <api.proxy.productionUrl>

you have to rerun a build:

cd "$HBP"/user-scripts
./configure_nrp
./update_nrp build

• Error when running npm install. If you see the following error:

  sh: node command not found
  

  you could try installing the package nodejs-legacy instead of nodejs:

  sudo apt-get install nodejs-legacy
  

• If you get this kind of JavaScript error, as reported in this post on the HBP forum, you may want to deactivate AdBlocker.

  Failed to instantiate module exdFrontendApp
  Error: [$injector:modulerr] Failed to instantiate module exdFrontendApp
  due to:
  [$injector:modulerr] Failed to instantiate module
  angulartics.google.analytics due to:
  [$injector:nomod] Module 'angulartics.google.analytics' is not
  available! You either misspelled the module name or forgot to load it.
  If registering a module ensure that you specify the dependencies as the
  second argument.
  http://errors.angularjs.org/1.4.8/$injector/nomod?p0=angulartics.google.analytics
  minErr/<@http://10.162.242.194:9000/bower_components/angular/angular.js:68:12
  module/<@h ttp://10.162.242.194:9000/bower_components/angular/angular.js:2005:
  

Other troubles

File not found error on building brainvisualizer

If you get this kind of error and the build breaks in brainvosualizer:

npm ERR! enoent ENOENT: no such file or directory, rename '/home/vonarnim/Documents/NRP/brainvisualizer/node_modules/npm/node_modules/ansistyles' -> '/home/vonarnim/Documents/NRP/brainvisualizer/node_modules/npm/node_modules/.ansistyles.DELETE'

then the solution is to manually remove the node_modules directory and rebuild

cd "$HBP"/brainvisualizer
rm -rf node_modules
cd "$HBP"/user-scripts
./update_nrp build # resumes build (without third party that were successfully built before)
./configure_nrp

git is complaning about changed package.json files that it has to stash

Just type enter to stash them and forget about the issue. It is known and we will fix it but it is not harmful to your build.

undefined symbol: ompi_mpi_logical8 OR No module named pyNN.nest

If cle-start fails with the following error:

Error starting the simulation. (service [/ros_cle_simulation/create_new_simulation] responded with an error:
error processing request: /home/florian/.opt/platform_venv/lib/python3.8/site-packages/mpi4py/MPI.so: undefined symbol: ompi_mpi_logical8) (host: localhost)

or this related error:

ImportError: No module named pyNN.nest

then you have a problem with your PYTHONPATH. Most probably your PYTHONPATH is containing /usr/lib/python3.8/dist-packages before the NRP paths. Try to erase it in your $HOME/.bashrc before loading nrp_variables.

[...]
export PYTHONPATH=
. "$HBP"/user-scripts/nrp_variables
[...]

If this does not help, the following (not very nice) solution has been reported. Before cle-start:

source $HBP_VIRTUAL_ENV/bin/activate

which “forces” the platform_venv directory to the front of the Python search path.

cannot import name config

After launching an experiment, it might fail saying cannot import name config.

pip install config

502 error because of retina

If cle-start fails with 502, possibly the pyretina is missing. Check the install commands of “retina” in the $HBP/user-scripts/nrp_functions script. Try execute them by hand.

Nginx is not starting during system boot? Add the following lines to /etc/rc.local:

if [ -f $HOME/.local/etc/init.d/nginx ]; then
    $HOME/.local/etc/init.d/nginx restart
fi

general access problems

-> file not accessible? 404? Check that the whole path to the file is readable and/or writable to you.

not loading experiment list

-> cors problems? Install CORS in your browser (google that)

-> nginx: sudo killall and service restart

Getting MoveIt package related error after launching an experiment

Error Message

RuntimeError: Unable to connect to move_group action server ‘pickup’ within allotted time (5s)

• Solution: The error caused by the input argument “wait_for_servers” of class “MoveGroupCommander” in the MoveIt package. The default value of “wait_for_servers” is set as 5 seconds. http://docs.ros.org/en/melodic/api/moveit_commander/html/classmoveit__commander_1_1move__group_1_1MoveGroupCommander.html
  If an experiment is launched and not pressed played button within 5 seconds, the aforementioned error might pop up and crash the experiment. To resolve the problem, the input parameter “wait_for_servers” should be set by a user explicitly with a high value such as 300 seconds

Example: moveit_commander.MoveGroupCommander(iiwa_group_name) should be replaced by moveit_commander.MoveGroupCommander(iiwa_group_name,wait_for_servers=300)

python DictError about some dictionary word missing

-> you probably run a local version of Flask. Check in /usr/local/lib/python3.8/dist-packages. There should be ONLY Nest related stuff. If there are pip pacakges, please remove them, especially Flask, SQLAlchemy, Werkzeug, itsdangerous and rerun $HOME/user-scripts/update_nrp build.

problems with gzweb deploy-*.sh

-> webify_models_v2.py: splits with mesh, do not put/use mesh in model names…

-> server crashes with error “Unable to find local grunt” or other nodejs package related error

cd "$HBP"/ExDFrontend
npm install

‘cle-start’ / ‘cle-rosbridge’ complains about missing rosauth.srv

sudo apt-get install ros-indigo-rosauth

‘cle-start’ / ‘cle-rosbridge’ can’t find package rosbridge_server. Add the following to .bashrc

export ROS_PACKAGE_PATH=$ROS_PACKAGE_PATH:</path/to/GazeboRosPackages>/src

‘cle-start’: backend can’t find modules (e.g. ImportError: No module named flask_restful_swagger)

There was an error during make devinstall when running $HOME/user-scripts/update_nrp build. Then run make devinstall in ExDBackend as follows:

make devinstall > devinstall-log.txt 2>&1

to see what happened. In the case of flask_restful_swagger, it might help to

• rename ExDBackend/hbp-flask-restful-swagger-master temporarily

• rerun make devinstall

Don’t forget to restore ExDBackend/hbp-flask-restful-swagger-master to its original name afterwards.

‘cle-start’: Failed to execute request http://localhost:8080/experiment. ERROR: Error: Error: connect ECONNREFUSED

Nginx is not running or not running correctly. Double check nginx is running and that there is no errors in the nginx logs.

Gazebo6 general problems or compiling GazeboRosPackages

to make sure there are no dependencies problems:

sudo apt-get purge gazebo4 # or 2 or 6
sudo apt-get remove gazebo4 # or 2 or 6
sudo apt-get install ros-indigo-desktop-full

then clean build directories and rebuild:

cd "$HBP"/GazeboRosPackages/
rm -rf build
catkin_make

it might be necessary to do

catkin_make --force-cmake

Building GazeboRosPackages failed for me with Gazebo installed from package repositories (gazebo.physics.Joint::SetMappedRotationAxis not found). If that happens, remove gazebo packages and compile from EPFL sources as above.

If building Gazebo fails with the message

Linking CXX shared library libgazebo_common.so
/usr/bin/ld: /usr/local/lib/libavcodec.a(allcodecs.o): relocation R_X86_64_32 against `ff_h263_vaapi_hwaccel' can not be used when making a shared object; recompile with -fPIC
/usr/local/lib/libavcodec.a: error adding symbols: Bad value
collect2: error: ld returned 1 exit status

You can prevent it from building with ffmpeg support by editing $HBP/gazebo/cmake/SearchForStuff.cmake. In line 391 change HAVE_FFMPEG TRUE to HAVE_FFMPEG FALSE. This will cause it to always assume that ffmpeg is not installed. Alternatively, you could compile ffmpeg from sources and create dynamic libraries instead of the static ones hat are installed from the package management.

If a similar error comes up later during the compile referencing BulletPhysics, try recompiling Bullet: First, delete the old Bullet files: Bullet does not provide a make uninstall target, so we need to do it manually. In the build folder, run

sudo xargs rm < install_manifest.txt

This will remove all files installed by make install. Then recompile and give the flag -DBUILD_SHARED_LIBS=true to cmake. This will build dynamically linked libraries. If you do this, make sure both $HOME/.local/lib and $HOME/.local/lib/x86_64-linux-gnu is on your LD_LIBRARY_PATH. The recommended way of adding it is creating the file /etc/ld.so.conf.d/gazebo.conf with the content

$HOME/.local/lib/x86_64-linux-gnu
$HOME/.local/lib

and running sudo ldconfig

If an error similar to this comes up

gazebo/physics/libgazebo_physics.so.6.0.6.hbp.1.0.0: undefined reference
to `gazebo::sensors::SensorManager::SensorsInitialized()

check your version of cmake, if it is version 3, try and downgrade it to version 2 and see if it helps

e.g.

apt list --installed | grep cmake # check the version of cmake
sudo apt-get install cmake=2.8.12.2-0ubuntu3 cmake-data=2.8.12.2-0ubuntu3 # or just downgrade to a previous version

ImportError: No module named pyretina

Follow the tutorial here: Install Retina.

Toolbar buttons icons (play/stop) show up as strange characters

This issue is due to missing fonts and outdated dependencies.

• Clear browser cache and close the browser

• Update ruby (tested with 1.9.3p484)

• run “sudo gem install compass”

Robot is not moving even though spike monitor works, no errors

In GazeboRosPackages, run:

catkin build