Documentation on toAron/fromAron

As a follow up from a discussion with @plewnia in February, I would suggest to add the following to the documentation of Aron:

---

---

toAron/fromAron

There are several versions of functions named toAron/fromAron or similar. To avoid confusion, their differences are explained in the following.

First, note that there are 3 representations being involved when doing conversions to or from ARON:

1. The business object, bo, is the C++/Python/... class that you actually use in your code. Typically, it does not only contain the data, but provide methods to act on them. It is not specific to ARON in any way.
2. The data transfer object, dto, is an autogenerated C++/Python/... class that is created from the type definition in the ARON .xml file. It is the representation of the type submitted via network in the high-level language. In C++, they typically reside in an arondto:: (sub)namespace
3. The "aron dto"/"ice" representation (the wording is less clear here), which is the low-level serialized form that can actually be transmitted of the network.

In C++

The toAron/fromAron functions defined in aron_conversions.h convert between the first and second layer. They typically take two arguments, namely toAron(dto, const bo) respectively fromAron(const dto, bo). In some cases, methods returning the result instead of writing it to one of the parameters exist as well, i.e., dto = toAron(const bo) respectively bo = fromAron(const dto), which take only one argument.

In contrast, there are toAron/fromAron functions that do not take any arguments. The convert between the second and third layer. In C++, they are autogenerated methods of the second layer's dto types.

This means, myDto.toAron() of a dto defined in arondto:: converts from the second to the third layer. If you instead want to convert a first-layer business object to a second-layer data transfer object, this is not what you need. You might think of an bo.toAron() method, but this usually does not exist - and shall not exist, as it would result in the business object having to be aware of the existence of aron. Instead, use the arondto::toAron(dto, const bo).

In Python

The first layer types again are the normal python classes defined in the application code. For the second layer, standardization in terms of where to put it in the code is currently lower than in the C++ case. However, the second layer types are always python data classes, which shall inherit from AronDataclass defined in the armarx_memory.aron.aron_dataclass module.

To convert between the first and second layer, one option is to define class methods on the second layer type. They take one argument, and have the signatures to_aron(dto) -> bo/from_aron(bo) -> dto.

To convert between the second and third layer, AronDataclass types inherit from_aron_ice(aron_ice_data) and to_aron_ice() methods. The aron_ice_data are the ones provided to the callback in the for_each_instance_data_ice() function.

---

---

Example for toAron/fromAron in python [going beyond what was discussed yet]

Defining the types

Business object, i.e., first layer:

class TheType:
    def __init__(some_attribute: int, some_optional_attribute: float or None = None):
        self.some_attribute: int = some_attribute
        self.some_optional_attribute: float or None = some_optional_attribute
        # not all variables of the bo need to be reflected in the dataclass, e.g.:
        is_ready: bool = False

    def do_something():
        pass

Data transfer object, i.e., second layer:

# if the following is impossible in your python version, leave it out and
# put type annotations referring to the own class in "quotation" marks
from __future__ import annotations

from dataclasses import dataclass
from armarx_memory.aron.aron_dataclass import AronDataclass
# "as ...Bo" is recommended for clarity
from path.of.the.bo_module import TheType as TheTypeBo


@dataclass
class TheTypeDto(AronDataclass):
    # for optionals, it is important to use the Union notation here, not `or`!
    some_attribute: int
    some_optional_attribute: Union[float, None]

    @classmethod
    def from_aron(cls, dto: TheTypeDto) -> TheTypeBo:
        bo = TheTypeBo(
             dto.some_attribute,
             dto.some_optional_attribute
             # if a parameter is a complex type itself, use something like:
             # TheOtherTypeDto.from_aron(dto.the_complex_attribute) 
        )
        return bo

    @classmethod
    def to_aron(cls, bo: TheTypeBo) -> TheTypeDto:
        dto = cls(
            bo.some_attribute,
            bo.some_optional_attribute
        )
        return dto

Conversions

In the following examples, the usage of to_aron and from_aron conversions is embedded into memory operations:

Reading from a memory

from armarx_memory.client import MemoryNameSystem, Reader
from armarx_memory.core import MemoryID
# Despite typically being imported as "dto", it refers to the
# third layer, not the second one!
from armarx.armem import data as dto
from armarx_memory.aron.aron_ice_types import AronIceTypes

core_segment_id: MemoryID = MemoryID('MemoryName', 'CoreSegmentName')
mns = MemoryNameSystem.wait_for_mns()
reader = mns.get_reader(core_segment_id)

query_result = reader.query_core_segment(
    some_memory_id.core_segment_name, latest_snapshot=True
)
bo_results: List[TheType] = []

# covering all conversions from layer three to layer one
# (the second parameter, "metadata", can be omitted, if
#  provide_metadata is set to False.)
def from_aron_ice_to_bo(
    memory_id: dto.MemoryID,
    metadata: dto.EntityInstanceMetadata,
    aron_ice_data: AronIceTypes.Dict
) -> None:
    # here, from_ice not only converts to a dto, which
    # needs to be passed to from_aron afterwards, but
    # directly yields the bo
    memory_id: MemoryID = MemoryID.from_ice(id)

    dto = TheTypeDto.from_aron_ice(aron_ice_data)
    bo = TheTypeDto.from_aron(dto)

    bo_results.append(bo)

reader.for_each_instance_data_ice(
    from_aron_ice_to_bo, query_result, provide_metadata=True
)

As a python speciality, there is not only the dictionary representing the third layer, but a pythonic representation between the second and third layer. Usually, it is used only internally, but can be useful for debugging, if conversions fail. It is much easier to read than the verbose dictionary of the third layer.

from armarx_memory.aron.conversion.pythonic_from_to_aron_ice import pythonic_from_aron_ice

...

def from_aron_ice_to_bo(
    memory_id: dto.MemoryID,
    metadata: dto.EntityInstanceMetadata,
    aron_ice_data: AronIceTypes.Dict
) -> None:
    print(
        f'Data for memory id {MemoryID.from_ice(memory_id)} are:\n'
        f'{pythonic_from_aron_ice(aron_ice_data)}'
    )
    ...

...

Writing to a memory

from armarx_memory.client import MemoryNameSystem, Writer, Commit
from armarx_memory.core import MemoryID, time_usec

core_segment_id: MemoryID = MemoryID('MemoryName', 'CoreSegmentName')
mns = MemoryNameSystem.wait_for_mns()
writer = mns.get_writer(core_segment_id)

bo: TheType = TheType(5, 5.2)

commmit: Commit = Commit()
dto = TheTypeDto.to_aron(bo)
commit.add(
    entity_id=MemoryID(
        core_segment_id.memory_name,
        core_segment_id.core_segment_name,
        "ProviderSegmentName",
        "EntityName"
    ),
    referenced_time_usec=time_usec(),
    instances_data=[dto.to_aron_ice()]
)
writer.commit(commit)

---

---

@plewnia:

• Can these explanations please get approved?
• Empirically, it seems that the core segment of a reader/writer can be changed. Is it perhaps sufficient to provide a memory ID containing only the memory name?
• Could we use a less ambiguous name for the import of "from armarx.armem import data as dto"? However, aron_ice or ice_types would also be ambiguous, due to the use in armarx_memory.aron.aron_ice_types.

assigned to @daab

Some Questions:

1. what do you mean with "For the second layer, standardization in terms of where to put it in the code is currently lower than in the C++ case." ?

2. "The "aron dto"/"ice" representation (the wording is less clear here), which is the low-level serialized form that can actually be transmitted of the network." -> where did you have the aron dto wording from? Canonically aron dto refers to level 2; level 3 does not really have a name canonically, I think (yet)

Some additional points:

A) if we have examples for python, we should also have some for C++

B) I think the python examples are a bit much? I don't really like that it intermingles so much with reading/writing from/to the memory

C) I think we have several versions of doing the from and to aron in python, I do it differently actually (see below)

class Person:
# this is the BO
    def __init__(self, given_name: str, family_name: str, roles: List):
        self.roles = roles
        self.given_name = given_name
        self.family_name = family_name

    def to_aron(self) -> "armarx.aron.data.dto.GenericData":
        from armarx_memory.aron.conversion import to_aron

        dto = to_aron(
            {
                "given_name": self.given_name,
                "family_name": self.family_name,
                "roles": self.roles,
            }
        )
        return dto

    @classmethod
    def from_aron(cls, dto: "armarx.aron.data.dto.GenericData"):
        from armarx_memory.aron.conversion import from_aron

        d = from_aron(dto)
        return cls(**d)

D) I don't think I completely agree with your definition of when we use pythonic_from_aron_ice, it is more historically, that this has not yet been brought to outer layers I think

So before I give approval:

1. we need to discuss the questions above
2. we need to at least add an empty section and a TODO for a C++ example
3. we should reduce the python examples to just IDF/aron
4. the python examples should show both methods for to/from_aron
5. for now I would just mention the pythonic_from_aron_ice method without to much explanation on when to use it and add a TODO for me, as I need to think about this

"Empirically, it seems that the core segment of a reader/writer can be changed. Is it perhaps sufficient to provide a memory ID containing only the memory name?"

I will add this to my (endless) ToDo List and see if it is actually needed or not, if not I will change it

"Could we use a less ambiguous name for the import of "from armarx.armem import data as dto"? However, aron_ice or ice_types would also be ambiguous, due to the use in armarx_memory.aron.aron_ice_types."

Why do you think this is ambiguous? Did you have cases where this was problematic?

Thank you for your feedback!

Regarding your questions:

1.) In C++, it is clear (?) that there is an aron_conversions.h and aron_conversions.cpp next to the aron folder, which contains the .xml aron type files. In python, I am not aware of a standardization. Currently, in my code I have a util folder in which all interfaces to external components go (csv export, ArmarX integration, ...) and an aron folder inside, which then contains the second-layer dataclasses. However, as the aron conversions between first and second layer are directly implemented as methods of these second-layer classes, this structure already differs from C++, where the conversions are in a separate file.

2.) For example from the use of ice interfaces: One instantiates the params as a second-layer arondto::SomeType instance, and then calls the auto-generated toAronDTO() method before submitting it to the ice interface. Similarly, the SkillExecutionRequest uses ToAronDictDTO().

Regarding the additional points:

A) I assumed it was "clear" in C++, while there seemed to be a lack of documentation in the python case. Nevertheless, I see the point of aiming for the documentation to be complete. Do you think we should add a separate C++ example here, or just refer to the memory tutorial, which covers the conversions as far as I remember?

B) Initially, I wanted to just show the part which I now entitled from_aron_ice_to_bo(), but realized that there were too many types a user might not be familiar with. Therefore, I wanted to show the relation to for_each_instance_data_ice, and have an executable example. But I see the point of mixing topics, and would be open to separating it. We do not yet have a memory example for python, do we?

C) Interesting - this is part of what I meant with being less standardized, to the best of my knowledge. While I see the advantage of having everything at one place, I do not like that this makes the business object dependent (and aware) of aron. For example, if one writes a library or uses an external library that shall be usable without ArmarX, adding the dataclasses is easy, while inheriting from the plain types to create ArmarX-compatible types looks less favorable to me.

Do I understand correct that you never define a second-layer dataclass explicitly, but make use of a python dict being converted to an AronDataclass object by the to_aron() which you probably import from somewhere?

D) Good to know that it is not intended to remain like this. Until now, I found it helpful for debugging. What do you mean by bringing it to outer layers?

E (second comment from you)) That you will change it: does that mean the MemoryName shall be sufficient in the future, or that the CoreSegment name shall be required and shall remain constant during the use of the reader? I would like to already mention how it shall be, so that people can write/prepare their (new) code accordingly.

F (third comment from you)) As you said, dto is associated to the second layer. However, in for_each_instance_data_ice, the memory ID is provided as dto.MemoryID, which one can pass to MemoryID.from_ice(...). To me, this implies that the argument of MemoryID.from_ice is actually an ice type (also called "aron dict", or not?) of the third layer, rather than as an AronDataclass of the second layer. Furthermore, in the armarx_memory.client.Reader.py, the import from armarx.armem import data as dto is annotated with # The ice type namespace., which again implied to me it is part of the third layer rather then the second layer. However, importing as ice_types would also be confusing, as there is another similarly named import of from armarx_memory.aron.aron_ice_types import AronIceTypes.

1. I see what you mean, I would formulate it differently I think. Maybe just mention that there is no set standardization of how to handle it there

A) I think we should add an example, especially because I plan on redoing the memory tutorial (again...)

B) we do: https://git.h2t.iar.kit.edu/sw/armarx/python3-armarx/-/blob/master/examples/memory_client.py

C) To be honest I just copied what Fabian did here; and yes you understand correctly

D) basically that you can (and should if it works out the way I want) use it as a person writing a new core segment in python for example

E) I plan on you only having to provide the memory name

F) I see... I have to take a look at that again before I can give 100% correct answers here

added documentation label