# Troubleshooting

## Reinstall an existing NRP

When reinstalling a whole NRP on top of an already existing one, you might run into problems that can be avoided with a thorough cleaning of old build directories.
First check your .bashrc against the additional HBP commands found in this documentation. Most important is that your `PYTHONPATH` is set to void.
Then clean up your NRP by running the following commands:

```bash
rm -rf "$HBP"/gazebo/build
rm -rf ~/.opt/platform_venv
rm -rf  "$HBP"/nrpBackendProxy/proxy_virtualenv
rm -rf ~/.opt/nrpStorage
rm -rf ~/.opt/bbp
rm ~/.spynnaker.cfg
cd "$HBP"/user_scripts && git checkout development && git pull --rebase
```

And then reupdate your NRP:

```bash
./update_nrp update --all
```

and reinstall your SpiNNaker in case you use one (Step 10).

## Troubleshoot Gazebo

### anaconda conflict breaks the build

If you get build errors in gazebo in the python TIFF libraries, you might have a conflict with anaconda.

    undefined symbol TIFFIsTiled@LIBTIFF_4.0

You have to get anaconda out of the `PATH` by resetting `PATH` temporarily at the end of your `$HOME/.bashrc`, rebuild the NRP, and finally clean up your `$HOME/.bashrc`.

### gzserver segfault on start


    gzserver: /usr/include/boost/thread/pthread/recursive_mutex.hpp:101: boost::recursive_mutex::~recursive_mutex(): Assertion `!pthread_mutex_destroy(&m)' failed.

*Solution*: Try to launch the gzserver binary with no argument.
If it runs, it probably means that you made something wrong when compiling GazeboRosPackages.
Make sure that you didn't source another catkin workspace before you built GazeboRosPackages.

### gazebo cmake error (happened with AMD GPU):

```text
CMake Error at cmake/CheckDRIDisplay.cmake:54 (STRING):
string sub-command FIND requires 3 or 4 parameters.
Call Stack (most recent call first):
CMakeLists.txt:154 (include)

--! valid dri display not found (no glxinfo or pyopengl)
```

*Solution*: install mesa-utils (glxinfo)

### gazebo cmake error (missing tinyxml2): install libtinyxml2-dev.

### gazebo libccd link error (/usr/bin/ld: cannot find -lccd):

You likely have a conflicting version of ros-indigo-libccd installed (from other ros packages such as MoveIt).
*Solution*: edit cmake/SearchForStuff.cmake and replace "pkg_check_modules(CCD ccd>=1.4)" with "SET(CCD_FOUND FALSE)"
delete your build directory, rerun cmake, and build as usual

## Troubleshoot Nest

Come back here after NRP build if you experience issues with Nest.

## Troubleshoot Gazebo Plugins (GazeboRosPackages)

Come back here if the NRP build fails on GazeboRosPackages.

**Warning**: If GazeboRosPackages build fails, it might be that the ROS setup file has not be sourced. If you added `nrp_variables` correctly in your bashrc, this should not be the case.
Else, open a fresh terminal, and check that `$ROS_MASTER_URI` is set. If not you can manually source the setup and redo the ```catkin build```.


```bash
source /opt/ros/noetic/setup.bash
catkin build
```

## Troubleshoot compiling gzweb

Come back here if NRP build fails on gzweb.

To make sure that gzweb is compiled with proper Gazebo version, prepend the correct path to package configuration:

```bash
export PKG_CONFIG_PATH=$HOME/.local/lib/x86_64-linux-gnu/pkgconfig:$PKG_CONFIG_PATH
```

If you get node-gyp errors ("File not found"), try deleting `$HOME/.npm` and re-running deploy script.

You may encounter:


    /usr/bin/env: node: No such file or directory
    There are node-gyp build errors, exiting.


This is fixed by a symbolic link: ```ln -s `which nodejs` $HOME/.local/bin/node```

You may also encounter core dump or segmentation faults depending on your graphics card and configuration. Install the proprietary drivers (not the X.org default driver) on your system and reboot.

You can see more possible problems and solutions here: <http://gazebosim.org/tutorials?tut=gzweb_install&ver=1.9%2B&cat=gzweb#Troubleshooting>

## Troubleshoot Nginx

If you are having issues with nginx configuration that could mean that your nginx version is too recent.
Try running

```bash
nginx -t -c $HOME/.local/etc/nginx/nginx.conf
```

In case the output of the command is something like 'unknown directive "more_set_headers"', try replacing the following lines in `$HOME/.local/etc/nginx/conf.d/gzweb.conf`

```nginx
location / {
    #...

    #more_set_headers 'Access-Control-Allow-Origin: $cors_origin';
    #more_set_headers 'Access-Control-Allow-Methods: $cors_methods';
    #more_set_headers 'Access-Control-Allow-Headers: $cors_headers';
    #more_set_headers 'Access-Control-Expose-Headers: Content-Length';

    add_header 'Access-Control-Allow-Origin' '$cors_origin' 'always';
    add_header 'Access-Control-Allow-Methods' '$cors_methods' 'always';
    add_header 'Access-Control-Allow-Headers' '$cors_headers' 'always';
    add_header 'Access-Control-Expose-Headers' 'Content-Length' 'always';

    #...
}
```

and in `nrp-services.conf`

```nginx
location / {
    #...

    #more_set_headers 'Access-Control-Allow-Origin: $cors_origin';
    #more_set_headers 'Access-Control-Allow-Methods: $cors_methods';
    #more_set_headers 'Access-Control-Allow-Headers: $cors_headers';

    add_header 'Access-Control-Allow-Origin' '$cors_origin' 'always';
    add_header 'Access-Control-Allow-Methods' '$cors_methods' 'always';
    add_header 'Access-Control-Allow-Headers' '$cors_headers' 'always';

    #...
}

location /api {
    #...

    #more_set_headers 'Access-Control-Allow-Origin: $cors_origin';
    #more_set_headers 'Access-Control-Allow-Methods: $cors_methods';
    #more_set_headers 'Access-Control-Allow-Headers: $cors_headers';

    add_header 'Access-Control-Allow-Origin' '$cors_origin' 'always';
    add_header 'Access-Control-Allow-Methods' '$cors_methods' 'always';
    add_header 'Access-Control-Allow-Headers' '$cors_headers' 'always';

    #...
}
```

nginx may complain that `/<assets_root>/sdk` does not exist. If so, just create an empty directory. No idea what is it for...

Use ```nginx -t -c $HOME/.local/etc/nginx/nginx.conf``` to get debug info if
```bash
$HOME/.local/etc/init.d/nginx restart
```
returns `[fail]`.

## Troubleshoot ports being used in cle-reverse-proxies
when running cle-nginx you might get an error that ports 9000 or 8080 are being used by another process. try these commands to stop default nginx and haproxy on your system: 

```bash
sudo service nginx stop
sudo service haproxy stop
```

## Troubleshoot ExDFrontend

Come back here if NRP build fails on ExDFrontend or cle-frontend fails to run.

* If you get this error in a pop up or in the web console:


    angular.js:14199 Error: UnkownConfigurationKey: <api.proxy.productionUrl>

you have to rerun a build:

```bash
cd "$HBP"/user-scripts
./configure_nrp
./update_nrp build
```

* Error when running npm install.
    If you see the following error:

    ```bash
    sh: node command not found
    ```

    you could try installing the package nodejs-legacy instead of nodejs:

    ```bash
    sudo apt-get install nodejs-legacy
    ```

* If you get this kind of JavaScript error, as reported in this post on the HBP forum, you may want to deactivate AdBlocker.

    ```bash
    Failed to instantiate module exdFrontendApp
    Error: [$injector:modulerr] Failed to instantiate module exdFrontendApp
    due to:
    [$injector:modulerr] Failed to instantiate module
    angulartics.google.analytics due to:
    [$injector:nomod] Module 'angulartics.google.analytics' is not
    available! You either misspelled the module name or forgot to load it.
    If registering a module ensure that you specify the dependencies as the
    second argument.
    http://errors.angularjs.org/1.4.8/$injector/nomod?p0=angulartics.google.analytics
    minErr/<@http://10.162.242.194:9000/bower_components/angular/angular.js:68:12
    module/<@h ttp://10.162.242.194:9000/bower_components/angular/angular.js:2005:
    ```

## Other troubles

### File not found error on building brainvisualizer

If you get this kind of error and the build breaks in brainvosualizer:

    npm ERR! enoent ENOENT: no such file or directory, rename '/home/vonarnim/Documents/NRP/brainvisualizer/node_modules/npm/node_modules/ansistyles' -> '/home/vonarnim/Documents/NRP/brainvisualizer/node_modules/npm/node_modules/.ansistyles.DELETE'

then the solution is to manually remove the node_modules directory and rebuild

```bash
cd "$HBP"/brainvisualizer
rm -rf node_modules
cd "$HBP"/user-scripts
./update_nrp build # resumes build (without third party that were successfully built before)
./configure_nrp
```

### git is complaning about changed package.json files that it has to stash

Just type enter to stash them and forget about the issue. It is known and we will fix it but it is not harmful to your build.

### undefined symbol: ompi_mpi_logical8 OR No module named pyNN.nest

If cle-start fails with the following error:

    Error starting the simulation. (service [/ros_cle_simulation/create_new_simulation] responded with an error:
    error processing request: /home/florian/.opt/platform_venv/lib/python3.8/site-packages/mpi4py/MPI.so: undefined symbol: ompi_mpi_logical8) (host: localhost)

or this related error:

    ImportError: No module named pyNN.nest

then you have a problem with your `PYTHONPATH`. Most probably your `PYTHONPATH` is containing `/usr/lib/python3.8/dist-packages` before the NRP paths. Try to erase it in your `$HOME/.bashrc` before loading `nrp_variables`.

```bash
[...]
export PYTHONPATH=
. "$HBP"/user-scripts/nrp_variables
[...]
```

If this does not help, the following (not very nice) solution has been reported. Before cle-start:

```bash
source $HBP_VIRTUAL_ENV/bin/activate
```

which "forces" the platform_venv directory to the front of the Python search path.

### cannot import name config

After launching an experiment, it might fail saying cannot import name config.

```bash
pip install config
```

### 502 error because of retina

If cle-start fails with 502, possibly the pyretina is missing. Check the install commands of "retina" in the `$HBP/user-scripts/nrp_functions` script. Try execute them by hand.

### Nginx is not starting during system boot? Add the following lines to /etc/rc.local:

```bash
if [ -f $HOME/.local/etc/init.d/nginx ]; then
    $HOME/.local/etc/init.d/nginx restart
fi
```

### general access problems

-> file not accessible? 404? Check that the whole path to the file is readable and/or writable to you.

### not loading experiment list

-> cors problems? Install CORS in your browser (google that)

-> nginx: sudo killall and service restart

### Getting MoveIt package related error after launching an experiment
Error Message 
> RuntimeError: Unable to connect to move_group action server 'pickup' within allotted time (5s)

* Solution:
The error caused by the input argument "wait_for_servers" of  class "MoveGroupCommander" in the MoveIt package.
The default value of "wait_for_servers" is set as 5 seconds.
http://docs.ros.org/en/melodic/api/moveit_commander/html/classmoveit__commander_1_1move__group_1_1MoveGroupCommander.html<br>
If an experiment is launched and not pressed played button within 5 seconds, the aforementioned error might pop up and crash the experiment.
To resolve the problem, the input parameter "wait_for_servers" should be set by a user explicitly with a high value such as 300 seconds

Example:
`moveit_commander.MoveGroupCommander(iiwa_group_name)`
should be replaced by
`moveit_commander.MoveGroupCommander(iiwa_group_name,wait_for_servers=300)`



### python DictError about some dictionary word missing

-> you probably run a local version of Flask. Check in `/usr/local/lib/python3.8/dist-packages`. There should be ONLY Nest related stuff. If there are pip pacakges, please remove them, especially *Flask*, *SQLAlchemy*, *Werkzeug*, *itsdangerous* and rerun `$HOME/user-scripts/update_nrp build`.

### problems with gzweb deploy-*.sh

-> webify_models_v2.py: splits with mesh, do not put/use mesh in model names...

-> server crashes with error "Unable to find local grunt" or other nodejs package related error

```bash
cd "$HBP"/ExDFrontend
npm install
```

### 'cle-start' / 'cle-rosbridge' complains about missing rosauth.srv

```bash
sudo apt-get install ros-indigo-rosauth
```

### 'cle-start' / 'cle-rosbridge' can't find package rosbridge_server. Add the following to .bashrc

```bash
export ROS_PACKAGE_PATH=$ROS_PACKAGE_PATH:</path/to/GazeboRosPackages>/src
```

### 'cle-start': backend can't find modules (e.g. ImportError: No module named flask_restful_swagger)

There was an error during make devinstall when running `$HOME/user-scripts/update_nrp build`. Then run `make devinstall` in `ExDBackend` as follows:

```bash
make devinstall > devinstall-log.txt 2>&1
```

to see what happened. In the case of flask_restful_swagger, it might help to

* rename ExDBackend/hbp-flask-restful-swagger-master temporarily
* rerun make devinstall

Don't forget to restore `ExDBackend/hbp-flask-restful-swagger-master` to its original name afterwards.

### 'cle-start': Failed to execute request <http://localhost:8080/experiment>. ERROR: Error: Error: connect ECONNREFUSED

Nginx is not running or not running correctly. Double check nginx is running and that there is no errors in the nginx logs.

### Gazebo6 general problems or compiling GazeboRosPackages

to make sure there are no dependencies problems:

```bash
sudo apt-get purge gazebo4 # or 2 or 6
sudo apt-get remove gazebo4 # or 2 or 6
sudo apt-get install ros-indigo-desktop-full
```

then clean build directories and rebuild:

```bash
cd "$HBP"/GazeboRosPackages/
rm -rf build
catkin_make
```

it might be necessary to do

```bash
catkin_make --force-cmake
```

Building GazeboRosPackages failed for me with Gazebo installed from package repositories (gazebo.physics.Joint::SetMappedRotationAxis not found). If that happens, remove gazebo packages and compile from EPFL sources as above.

If building Gazebo fails with the message

    Linking CXX shared library libgazebo_common.so
    /usr/bin/ld: /usr/local/lib/libavcodec.a(allcodecs.o): relocation R_X86_64_32 against `ff_h263_vaapi_hwaccel' can not be used when making a shared object; recompile with -fPIC
    /usr/local/lib/libavcodec.a: error adding symbols: Bad value
    collect2: error: ld returned 1 exit status

You can prevent it from building with ffmpeg support by editing `$HBP/gazebo/cmake/SearchForStuff.cmake`. In line 391 change `HAVE_FFMPEG TRUE` to `HAVE_FFMPEG FALSE`. This will cause it to always assume that ffmpeg is not installed. Alternatively, you could compile ffmpeg from sources and create dynamic libraries instead of the static ones hat are installed from the package management.

If a similar error comes up later during the compile referencing BulletPhysics, try recompiling Bullet:
First, delete the old Bullet files: Bullet does not provide a make uninstall target, so we need to do it manually. In the build folder, run

```bash
sudo xargs rm < install_manifest.txt
```

This will remove all files installed by make install. Then recompile and give the flag `-DBUILD_SHARED_LIBS=true` to cmake. This will build dynamically linked libraries. If you do this, make sure both `$HOME/.local/lib` and `$HOME/.local/lib/x86_64-linux-gnu` is on your `LD_LIBRARY_PATH`. The recommended way of adding it is creating the file `/etc/ld.so.conf.d/gazebo.conf` with the content

```bash
$HOME/.local/lib/x86_64-linux-gnu
$HOME/.local/lib
```

and running `sudo ldconfig`

If an error similar to this comes up

    gazebo/physics/libgazebo_physics.so.6.0.6.hbp.1.0.0: undefined reference
    to `gazebo::sensors::SensorManager::SensorsInitialized()

check your version of cmake, if it is version 3, try and downgrade it to version 2 and see if it helps

e.g.

```bash
apt list --installed | grep cmake # check the version of cmake
sudo apt-get install cmake=2.8.12.2-0ubuntu3 cmake-data=2.8.12.2-0ubuntu3 # or just downgrade to a previous version
```

### ImportError: No module named pyretina

Follow the tutorial here: Install Retina.

### Toolbar buttons icons (play/stop) show up as strange characters

This issue is due to missing fonts and outdated dependencies.

* Clear browser cache and close the browser
* Update ruby (tested with 1.9.3p484)
* run "sudo gem install compass"

### Robot is not moving even though spike monitor works, no errors

In GazeboRosPackages, run:

```bash
catkin build
```