RT log messages passing from RT to non-RT
We investigated, how log messages are currently passed from the RT thread to non-RT parts of the unit.
- Beginning with
ControlThreadOutputBuffer- This seems to be a SPSC queue, implemented as ring buffer (own implementation, non-standard
- Unclear: how are sensor data and control targets written to buffer?
- Found out, how messages are added
- This seems to be a SPSC queue, implemented as ring buffer (own implementation, non-standard
-
ControlThreadOutputBuffer::addMessageToLog(...)is used to write messages into anRtMessageLogBuffervia macros
/cc @weiner
Activity
-
detail::RtMessageLogBuffer::resetchecks for the maximal allowed buffer size, but still resizes the buffer even ifrequiredSpace>maximalBufferSize.
-
When disabeling the non-rt logging (everything that has to with writing to csv files and the backlog) only keeping things necessary for dataStreaming, the unit runs >20mins (I only have to leave and can't let it run any longer). So far I have also not found any problems with any other components. We still have to investigate this further.
I looked at it a little bit further and and could narrow down the location of the error:
When I add a try catch block around it, I was able to catch the error, but then something goes wrong at a different place:
[9909][13:55:24.886][ArmarDEUnit][void armarx::handleExceptions()]: Caught std::exception: std::bad_alloc Trace created by calling ARMARX_TRACE: >>> * -2: /home/armar-user/code/armarx/RobotAPI/source/RobotAPI/components/units/RobotUnit/RobotUnitModules/RobotUnitModulePublisher.cpp:232 : armarx::TimedVariantPtr armarx::RobotUnitModule::Publisher::publishControlUpdates(const SensorAndControl&, bool, bool, const armarx::JointAndNJointControllers&, const std::vector<armarx::JointController*>&) >>> * -1: /home/armar-user/code/armarx/RobotAPI/source/RobotAPI/components/units/RobotUnit/RobotUnitModules/RobotUnitModulePublisher.cpp:515 : virtual void armarx::RobotUnitModule::Publisher::publish(armarx::StringVariantBaseMap, armarx::StringVariantBaseMap) [9908][13:55:24.886][ArmarDEUnit][std::cerr]: doLogging 743 Caught std::exception: std::bad_alloc Trace created by calling ARMARX_TRACE: [9908][13:55:24.890][ArmarDEUnit][std::cerr]: doLogging 743 Caught std::exception: std::bad_alloc Trace created by calling ARMARX_TRACE: [9908][13:55:24.892][ArmarDEUnit][std::cerr]: doLogging 743 Caught std::exception: std::bad_alloc Trace created by calling ARMARX_TRACE: [9909][13:55:24.893][ArmarDEUnit][void armarx::handleExceptions()]: Caught std::exception: std::bad_alloc Trace created by calling ARMARX_TRACE: >>> * -4: /home/armar-user/code/armarx/RobotAPI/source/RobotAPI/components/units/RobotUnit/util/introspection/ClassMemberInfo.h:168 : static std::map<std::__cxx11::basic_string<char>, IceInternal::Handle<armarx::VariantBase> > armarx::introspection::ClassMemberInfo<CommonBaseT, ClassT>::ToVariants(const IceUtil::Time&, const CommonBaseT*) [with CommonBaseT = armarx::SensorValueBase; ClassT = devices::ethercat::hand::armarde::SensorDataValue] >>> * -3: /home/armar-user/code/armarx/RobotAPI/source/RobotAPI/components/units/RobotUnit/RobotUnitModules/RobotUnitModulePublisher.cpp:309 : armarx::TimedVariantPtr armarx::RobotUnitModule::Publisher::publishSensorUpdates(bool, const SensorAndControl&) >>> * -2: /home/armar-user/code/armarx/RobotAPI/source/RobotAPI/components/units/RobotUnit/RobotUnitModules/RobotUnitModulePublisher.cpp:299 : armarx::TimedVariantPtr armarx::RobotUnitModule::Publisher::publishSensorUpdates(bool, const SensorAndControl&) >>> * -1: /home/armar-user/code/armarx/RobotAPI/source/RobotAPI/components/units/RobotUnit/RobotUnitModules/RobotUnitModulePublisher.cpp:515 : virtual void armarx::RobotUnitModule::Publisher::publish(armarx::StringVariantBaseMap, armarx::StringVariantBaseMap) [9908][13:55:24.893][ArmarDEUnit][std::cerr]: doLogging 743 Caught std::exception: std::bad_alloc Trace created by calling ARMARX_TRACE: [9908][13:55:24.895][ArmarDEUnit][std::cerr]: doLogging 743 Caught std::exception: std::bad_alloc Trace created by calling ARMARX_TRACE: [9908][13:55:24.896][ArmarDEUnit][std::cerr]: doLogging 743 Caught std::exception: std::bad_alloc Trace created by calling ARMARX_TRACE: [9908][13:55:24.898][ArmarDEUnit][std::cerr]: doLogging 743 Caught std::exception: std::bad_alloc Trace created by calling ARMARX_TRACE: [9908][13:55:24.899][ArmarDEUnit][std::cerr]: doLogging 743 Caught std::exception: std::bad_alloc Trace created by calling ARMARX_TRACE: [9908][13:55:24.901][ArmarDEUnit][std::cerr]: doLogging 743 Caught std::exception: std::bad_alloc Trace created by calling ARMARX_TRACE: [9909][13:55:24.902][ArmarDEUnit][void armarx::handleExceptions()]: Caught std::exception: std::bad_alloc Trace created by calling ARMARX_TRACE: >>> * -4: /home/armar-user/code/armarx/RobotAPI/source/RobotAPI/components/units/RobotUnit/RobotUnitModules/RobotUnitModulePublisher.cpp:318 : armarx::TimedVariantPtr armarx::RobotUnitModule::Publisher::publishSensorUpdates(bool, const SensorAndControl&) >>> * -3: /home/armar-user/code/armarx/RobotAPI/source/RobotAPI/components/units/RobotUnit/RobotUnitModules/RobotUnitModulePublisher.cpp:309 : armarx::TimedVariantPtr armarx::RobotUnitModule::Publisher::publishSensorUpdates(bool, const SensorAndControl&) >>> * -2: /home/armar-user/code/armarx/RobotAPI/source/RobotAPI/components/units/RobotUnit/RobotUnitModules/RobotUnitModulePublisher.cpp:299 : armarx::TimedVariantPtr armarx::RobotUnitModule::Publisher::publishSensorUpdates(bool, const SensorAndControl&) >>> * -1: /home/armar-user/code/armarx/RobotAPI/source/RobotAPI/components/units/RobotUnit/RobotUnitModules/RobotUnitModulePublisher.cpp:515 : virtual void armarx::RobotUnitModule::Publisher::publish(armarx::StringVariantBaseMap, armarx::StringVariantBaseMap) [9908][13:55:24.902][ArmarDEUnit][std::cerr]: doLogging 743 Caught std::exception: std::bad_alloc Trace created by calling ARMARX_TRACE: [9908][13:55:24.904][ArmarDEUnit][std::cerr]: doLogging 743 Caught std::exception: std::bad_alloc Trace created by calling ARMARX_TRACE: [9908][13:55:24.905][ArmarDEUnit][std::cerr]: doLogging 743 Caught std::exception: std::bad_alloc Trace created by calling ARMARX_TRACE: [9908][13:55:24.906][ArmarDEUnit][std::cerr]: doLogging 743 Caught std::exception: std::bad_alloc Trace created by calling ARMARX_TRACE: [9908][13:55:24.908][ArmarDEUnit][std::cerr]: doLogging 743 Caught std::exception: std::bad_alloc Trace created by calling ARMARX_TRACE: [9908][13:55:24.909][ArmarDEUnit][std::cerr]: doLogging 743 Caught std::exception: std::bad_alloc Trace created by calling ARMARX_TRACE: [9908][13:55:24.911][ArmarDEUnit][std::cerr]: doLogging 743 Caught std::exception: std::bad_alloc Trace created by calling ARMARX_TRACE: [9908][13:55:24.912][ArmarDEUnit][std::cerr]: doLogging 743 Caught std::exception: std::bad_alloc Trace created by calling ARMARX_TRACE: [9908][13:55:24.914][ArmarDEUnit][std::cerr]: doLogging 743 Caught std::exception: std::bad_alloc Trace created by calling ARMARX_TRACE: terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc terminate called recursively Aborted (core dumped)mentioned in merge request !329 (merged)
Please register or sign in to reply