ArmarX Setup Tool

Setup and maintenance tool for ArmarX and its dependencies.

Installation

If not done yet, install these system packages:

sudo apt install git build-essential python3-venv libpython3-dev graphviz-dev libssl-dev libbz2-dev

gcc and libpython3-dev are required by python-Levenshtein which is used to suggest alternatives on typos.

Clone the repository, e.g. into ~/armarx-setup, using either (1) or (2):

cd ~  # Or another directory of your choice.

# (1) With SSH access:
git clone git@gitlab.com:ArmarX/meta/setup.git armarx-setup
# (2) Without SSH access:
git clone https://gitlab.com/ArmarX/meta/setup.git armarx-setup

cd armarx-setup/

Run bootstraph.sh. This sets up a virtual python environment .venv and makes the armarx-setup command globally available:

./bootstrap.sh  

Pass --no-bashrc if you do not want to have your ~/.bashrc changed automatically.

Open a new console or source the ~/.bashrc by running:

source ~/.bashrc

Optional: Enable shell completion (needs to be done only once):

armarx-setup self enable-completion

To update the setup tool itself, run:

armarx-setup self update

Getting Started

To get going, initialize a new workspace:

armarx-setup init

Unless you know what you do, choose yes when you are asked about modifying the ~/.bashrc. (See Directory-Aware Environment below for how to use multiple workspaces.) Afterwards, open a new console.

To run the setup, upgrade your workspace:

armarx-setup upgrade

By default, this will install the ArmarX framework packages and their dependencies.

For more details, see Usage below.

Usage

See which commands are available:

armarx-setup --help

Create a Workspace

Initialize a new workspace:

armarx-setup init

The active workspace is specified by the ARMARX_WORKSPACE environment variable. It can be set by sourcing the file armarx-workspace.rc generated in your workspace. If you chose to source the created armarx-workspace.rc in your ~/.bashrc during the workspace initialization (recommended), open a new console or run source ~/.bashrc for this to take effect.

If you want to use multiple workspaces instead (i.e. not have ARMARX_WORKSPACE set to a specific workspace by default), have a look at the section Directory-Aware Environment below.

Manage a Workspace

Introducing Modules and the Module Graph

The ArmarX Setup Tool is based on modules.

Let's start with an example:

• There is a module armarx/ArmarXCore which clones, configures, builds and installs the ArmarX package ArmarXCore.
• In addition, the module armarx/ArmarXCore depends, or requires, another module named simox, which clones, configures, builds and installs the standalone CMake package Simox.
• The module simox, in turn, requires the module apt/eigen3, which installs the linear algebra library Eigen via a system package.

This dependency chain can be represented by a directed (acyclic) graph, where modules are nodes and dependencies build edges,

(armarx/ArmarXCore) --> (simox) --> (apt/eigen3)

or, visually:

This module graph is the heart of ArmarX Setup. In general, modules can clone repositories, build projects, install system packages, and more. Most importantly, modules have dependencies to other modules (their required modules). They build the module graph, which represents a program of how your workspace should be set up.

Apropos: Your workspace specifies one or several modules it would to have installed in its config file armarx-workspace.json. These modules are added to the module graph, as well as their dependencies, and their dependencies, and so on.

You can even visualize the active module graph by running (optional arguments in []):

armarx-setup info --graph [--reduce] [--with-system]

By the way, the module name, e.g. armarx/ArmarXCore, is also the path inside your workspace where a module representing a code base will be cloned or downloaded to (e.g. ~/code/armarx/ArmarXCore if your workspace is ~/code). At the same time, they are paths to .json files in the data/modules directory, which specifies what the module does.

Managing Workspace Modules

You can modify the workspace by adding or removing modules:

armarx-setup add module/name
armarx-setup remove module/name

# Examples:
armarx-setup add armarx/ArmarXCore
armarx-setup add armarx_integration/robots/armar6

To list the modules added to the workspace:

# List just the added modules:
armarx-setup list

# List the added modules and their direct and indirect dependencies.
armarx-setup list --deps

# List all modules in the database (independent of the workspace):
armarx-setup list --all

To search for a module, you can do:

armarx-setup list --all | grep -i "search term"

To show detailed information about the current workspace:

armarx-setup info

# To include pure system modules:
armarx-setup info --system

# To plot the module graph (requires additional packages, see setup below):
armarx-setup info --graph
armarx-setup info --graph --reduce  # See the transitively reduced module graph

Upgrade a Workspace

The most important command is upgrade. Use it to

• install system requirements such as apt packages (you are asked beforehand),
• update/clone repositories,
• prepare (e.g., run cmake),
• build (e.g., run cmake --build), and
• install

all modules and their (direct and indirect) dependencies:

armarx-setup upgrade

You can skip any of these steps, for example (see armarx-setup upgrade --help):

armarx-setup upgrade --no-system --no-update

You can also directly run one of these steps:

armarx-setup system
armarx-setup update  # add '--parallel' to speed this up
armarx-setup prepare
armarx-setup build
armarx-setup install

You can also execute a custom command in each module's directory inside the workspace (if it exists):

armarx-setup exec ls
armarx-setup exec "echo \$(pwd)"
armarx-setup exec "git status"

To get the status of each git repository in a nice table, run:

armarx-setup git status

The Local Workspace Config (armarx-workspace.json)

The local workspace config file (armarx-workspace.json in your workspace directory) contains the added modules as well as global module settings that will be used by all modules (if appropriate):

{
  "modules": {
    ...
  },

  "global": {
    ...
  }
}

At the moment, you can specify

• CMake definitions during the prepare step (prepare/cmake/definitions/)
• The CMake generator (prepare/cmake/generator/)
• Environment variables during the build step (build/cmake/env)
• Environment variables exported to the armarx-workspace.rc file (install/env)

If you need additonal settings, please create an issue.

{
  "global": {
    "prepare": {
      "cmake": {
        "definitions": {
          "KEY": "VALUE",
          ...
        },
        "generator": {
          "name": "Ninja" / "Unix Makefiles",
          "args": "..."
        }
      }
    },
    "build": {
      "cmake": {
        "env": {
          "KEY": "VALUE",
          ...
        }
      }
    },
    "install": {
      "env": {
        "KEY": "VALUE",
         ...
      }
    }
  }
}

Parallel Build

To build with multiple cores, add this to your local workspace config (replace 7 by the number of cores appropriate for your system):

{
  "global": {
    "build": {
      "cmake": {
        "env": {
          "CMAKE_BUILD_PARALLEL_LEVEL": "7"
        }
      }
    }
  }
}

Cmake Build Type

To enable debugging, set the build type to RelWithDebInfo:

{
  "global": {
    "prepare": {
      "cmake": {
        "definitions": {
          "CMAKE_BUILD_TYPE": "RelWithDebInfo"
        }
      }
    }
  }
}

Disable Tests (ArmarX Projects Only)

To disable tests, add this to your local workspace config:

{
  "global": {
    "prepare": {
      "cmake": {
        "definitions": {
          "ARMARX_BUILD_TESTS": "OFF"
        }
      }
    }
  }
}

Disable CMake's Global Registry

Some projects, such as all ArmarX packages, might register themselves in the global CMake registry found in ~/.cmake/packages. This however violates the concept of isolated workspaces. All modules should expose the CMake project they contain by providing a project_DIR environment variable. Therefore, it is advised to disable the global registration via the following:

{
  "global": {
    "prepare": {
      "cmake": {
        "definitions": {
          "CMAKE_EXPORT_NO_PACKAGE_REGISTRY": "ON"
        }
      }
    }
  }
}

Use Ninja Instead of Unix Makefiles

To use Ninja as your default CMake generator, simply adjust your global workspace config like so:

{
  "global": {
    "prepare": {
      "cmake": {
        "generator": {
          "name": "Ninja",
          "args": "-k9999"
        }
      }
    }
  }
}

Make sure that the apt package ninja-build is installed.

Ninja Only: Colored Output

In contrast to make, the compiler output won't contain any colors if ninja is used. To overcome this limitation, add the following to the workspace config:

{
  "global": {
    "prepare": {
      "cmake": {
        "definitions": {
          "FORCE_COLORED_OUTPUT": "ON",
          "ARMARX_CMAKE_ADDON_FORCE_COLORED_OUTPUT": "ON"
        }
      }
    }
  }
}

Use Ccache

To make use of Ccache, depend on the apt/ccache module to export the $CCACHE variable, and then set the corresponding CMake variables in the global workspace config as follows:

{
  "modules": {
    "apt/ccache": {},
    ...
  },

  "global": {
    "prepare": {
      "cmake": {
        "definitions": {
          "CMAKE_C_COMPILER_LAUNCHER": "$CCACHE",
          "CMAKE_CXX_COMPILER_LAUNCHER": "$CCACHE"
        }
      }
    }
  }
}

(Optional) Define the cache size (example, 100GB) (after running upgrade or system once to install Ccache):

ccache -M 100G

IDE integration

VS Code

Define the build tasks in ~/.config/Code/User/tasks.json. The following will build the project belonging to the currently opened file:

{
  "version": "2.0.0",
  "tasks": [
    {
      "label": "ax_build",
      "type": "shell",
      "command": [
        "armarx-setup build -n -f",
        "${file}"
      ],
      "problemMatcher": {
        "owner": "cpp",
        "fileLocation": [
          "relative",
          "${workspaceRoot}/build"
        ],
        "pattern": {
          "regexp": "^(.*):(\\d+):(\\d+):\\s+(warning|error):\\s+(.*)$",
          "file": 1,
          "line": 2,
          "column": 3,
          "severity": 4,
          "message": 5
        }
      },
      "group": {
        "kind": "build",
        "isDefault": true
      }
    }
  ]
}

You can also update your keybindings (~/.config/Code/User/keybindings.json)

{
      "key": "ctrl+b",
      "command": "workbench.action.tasks.build"
}

Then just press Ctrl+B to trigger the build.

Advanced Usage

Setup

Instead of running bootstrap.sh, you can also manually set up the tool's environment.

Create the virtual python environment by either (1) or (2):

# (1) Using pip and virtalenv
python3 -m venv .venv
source .venv/bin/activate
pip install wheel  # This must be done before requirements.txt due to python-levenshtein.
pip install -r requirements.txt

# (2) Using Python Poetry
poetry install

Optional: To allow plotting the module graph, install:

sudo apt-get install graphviz graphviz-dev
pip install -r plotting_requirements.txt

./bin/armarx-setup is a bash script which first activates the python environment, then calls the python script ./armarx-setup in the root directory if this repository (.).

If you have activated the python environment, you can also directly call ./armarx-setup or python -m armarx_setup.

To be able to run armarx-setup from anywhere, add this line

export PATH="$HOME/armarx-setup/bin:$PATH"

to your ~/.bashrc by running (take care to use 'single quotes', not "double quotes"):

echo 'export PATH="$HOME/armarx-setup/bin:$PATH"' >> "$HOME/.bashrc"

Then source the ~/.bashrc or open a new console.

Directory-Aware Environment (optional)

Instead of sourcing the armarx-workspace.rc file in your .bashrc, armarx-workspace.rc can also be sourced when entering the workspace in the terminal. To do so, you need to install direnv first

sudo apt install direnv

Depending on the shell you use, you need add the following hook to your ~/.bashrc (see https://direnv.net/docs/hook.html for more details).

eval "$(direnv hook bash)"

# Suppress the output when entering the directory.
export DIRENV_LOG_FORMAT=

Open a new terminal. Within the root of your workspace, create a file .envrc with the content source armarx-workspace.rc by running

echo "source armarx-workspace.rc" > .envrc

Save and close the file. There will be a warning, that you need to execute

direnv allow

If you do so, this file will always be sourced when you enter the directory.

Extend the Built-In Module Database

The built-in module definitions are stored in data/modules. That is, a module name is the path to a .json file relative to the root of the module database.

You can extend the built-in module database by setting the environment variable ARMARX_MODULES, which is a colon (:) separated list of root directories in which to look for modules (similar to how you can modify the PATH variable to find executables). This can be done by adding lines like this to the ~/.bashrc:

export ARMARX_MODULES="$HOME/armarx-modules:$ARMARX_MODULES"

Aliases

If you would like to use an alias command instead of armarx-setup and want to keep the shell completion, you can do it like this (e.g. for the alias axii):

cd .../armarx-setup/bin/
ln -s armarx-setup axii

To enable shell completion for the alias, open the file armarx-setup/armarx_setup/cli/config_files/armarx-setup-complete.bash and change

...
_armarx_setup_completion_setup() {
    complete -o nosort -F _armarx_setup_completion armarx-setup
}
...

to (replacing axii with your alias):

...
_armarx_setup_completion_setup() {
    complete -o nosort -F _armarx_setup_completion armarx-setup
    complete -o nosort -F _armarx_setup_completion axii
}
...

You can also define custom aliases for special calls by adding the following lines to your ~/.bashrc:

# build the project in this directory and its dependencies
alias bt="armarx-setup build -t"

# build the project in this directory without its dependencies
alias btn="armarx-setup build -t -n"

# build all projects and ignore errors
alias ba="armarx-setup build -i"