Project not buildable without X11 Display

I get the following output when trying to build the project (via axii). I'm using an ssh session (with -XC flags), and then switch the user within the session (su ...).

Build module armarx/core/arviz_godot ...
Set env vars: CMAKE_BUILD_PARALLEL_LEVEL='16' ...
Run 'nice ionice -c 2 -n 7 cmake --build . --target all' in 
'/home/robdekon_local/repos/armarx/core/arviz_godot/build' ...
────────────────────────────────────────────────── Command output ───────────────────────────────────────────────────
Consolidate compiler generated dependencies of target arviz_godot_interfaces
[  1%] Built target arviz_godot_interfaces
Consolidate compiler generated dependencies of target godot-cpp
[ 88%] Built target godot-cpp
Consolidate compiler generated dependencies of target arviz_godot
[100%] Built target arviz_godot
[100%] Export 'arviz-godot' executable ...
Create symlink /home/robdekon_local/repos/armarx/core/arviz_godot/source/arviz_godot/project/bin/x11/libgdavi.so to /home/robdekon_local/repos/armarx/core/arviz_godot/build/source/arviz_godot/lib/avi/bin/x11/libgdavi.so.0.1.0 ...
Export Godot project ...
Godot Engine v3.4.2.stable.official.45eaa2daf - https://godotengine.org
X11 connection rejected because of wrong authentication.
ERROR: X11 Display is not available
   at: initialize (platform/x11/os_x11.cpp:127)
source/arviz_godot/project/CMakeFiles/arviz_godot_app.dir/build.make:73: recipe for target 'source/arviz_godot/project/avi.x86_64' failed
make[2]: *** [source/arviz_godot/project/avi.x86_64] Error 255
CMakeFiles/Makefile2:547: recipe for target 'source/arviz_godot/project/CMakeFiles/arviz_godot_app.dir/all' failed
make[1]: *** [source/arviz_godot/project/CMakeFiles/arviz_godot_app.dir/all] Error 2
Makefile:145: recipe for target 'all' failed
make: *** [all] Error 2

I tried disabling the arviz_godot target, but I get the same error. It would be good if the build doesn't depend on having an X11 connection.