Wrong shape when committing arrays via python, and insufficient checks, resulting in invalid data after LTM re-import
The behavior leading to the investigations was:
- The objective is to store a movement primitive in the memory, which (among others) contains a 140-dimensional float64 vector
- Writing the learned primitive from python to the memory did not raise an error
- Reading from the memory to python did not raise an error, and yielded correct data
- After exporting the movement primitive to the LTM and re-loading it into the memory, reading to python yielded an error, even in the already pythonic_from_aron_ice() processing step
The problems occurred in dev/programming-by-demonstration-updated, while the dev/programming-by-demonstration branch was not affected. The difference of "-update" is that master had been merged into the branch. The commit 3aeceedd from the master branch, which is labeled "Uncommitted changes", is suspected to cause the problems.
It seems that wrong treatment of shapes, in combination with a lack of checks, leads to an erroneous behavior while writing, and a very late detection:
- Expected data:
- As float64 requires 8 byte per entry (64bit / 8bit_per_byte), the vector should have a size of 140 * 8 = 1120 bytes.
- As the last entry of the ARON shape is treated as the number of bytes per item, the shape should be (140, 8)
- When committing to the memory from python, the commit 3aeceedd introduced to always set the last shape entry to 1, rather than the number of bytes per item. This leads to committing all 1120 bytes, with a wrong shape of (140, 1) being written to the metadata. There is no check preventing this.
- When reading from the memory to python for the first time, the number of bytes not matching the shape is not checked for. As all bytes are actually there, the conversion works fine.
- Probably (this is an hypothesis), the memory export and import respects the declared shape, and therefore exports/imports only the first byte of each entry.
- Thus, after import, the shape is still (140, 1), but now, indeed only 140 bytes are available
- When trying to read, this fails. However, the reason is not that 140 bytes are less bytes than the required 140*8 bytes, but just due to 8 not being a divisor of 140. Otherwise, at least np.frombuffer() would work, despite providing a wrong result (array.reshape() might fail afterwards).
Suggested fix:
- The number of bytes should again be reported in the last entry of the shape, as it was before 3aeceedd.
- It should not be possible to commit a number of bytes that does not match the declared shape. Thus, an assertion is included.
- Also during reading, inconsistencies should be detected. This covers three aspects:
- There can be a mismatch between the declared shape, based on the declared number of bytes per item, and the provided number of bytes.
- There can be a mismatch between the declared number of bytes per item and the number of bytes per item required by the declared type. This and the aforementioned case indicate wrong metadata, but the underlying data might still be intact. Therefore, these checks should only raise warnings.
- Finally, there can be a mismatch between the declared shape, in combination with the actual number of bytes per item according to the detected data type, and the provided number of bytes. In such case, either np.frombuffer() or array.reshape() are expected to fail. Thus, this should be an assertion, to catch this case already outside of the numpy call.