Wissenschaft & Publikationen
TWIN4TRUCKS steht für den Transfer wissenschaftlicher Forschungsergebnisse in die industrielle Praxis. Die folgenden Publikationen sind rund um das Projekt TWIN4TRUCKS entstanden.
Veröffentlichung geplant am 16.07.2025
A Context-Aware Assembly Assistance Framework in Augmented Reality Using Skill-Based Production Approach and Digital Twin Integration
Snehal Walunj, Parsha Pahlevannejad, Alexandra Ritter, Simon Bergweiler, Martin Ruskwoski
Advanced assistance systems tackle the challenges of high product complexities by providing adaptive support to workers, aligning with Industry 5.0 goals. This paper introduces a context-aware assembly-assistance system with an Augmented-Reality interface, using the Asset-Administration Shell as a Digital-Twin (DT) to represent production information, with modular services like pre-condition check, assembly assistance, and quality inspection orchestrated via a BPMN workflow for skill-based production. It provides real-time assembly guidance and integrates quality feedback mechanisms to identify issues, facilitating worker training. This system represents a leap forward in digitally transformed manufacturing, for improving productivity and user experience in assisted assembly processes.
24.06.2025
Enabling non-smart assets using peripheral digital twins
Alexandra Ritter, Manuel Reif, Pascal Rübel, Benjamin Blumhofer, Simon Jungbluth, Martin Ruskowski
Numerous operating resources are used in manufacturing, from the warehouse to the point of use in a machinery park. The resources vary in their amounts of sensor technology and intelligence, resulting in diverse volumes of data. In the case of autonomous mobile robots or the motors of machines, data is provided by multiple sensors and described in standardized information models for analyzing their status and for improving the manufacturing processes.
In contrast, there is data enrichment of the information models of operating resources that have minimal or no sensor technology. The need for a cost-efficient and robust design is often an argument against increasing the number of sensors available on such resources, as they are found in large numbers within companies, e. g. rudimentary tools, logistic carriers or pallets. In a variety of scenarios, it is advantageous to possess a more extensive set of information regarding these non-smart assets. When these assets are utilized in a production process and significantly influence the outcome, it is crucial to acquire a comprehensive knowledge of them and ensuring uninterrupted data integration.
This paper introduces a concept for generating information about non-smart assets using peripheral assets and their information models. The approach is illustrated through the example of asset tracking for non-smart work piece carriers and their transported products. An information model will be developed for the work piece carriers, which will be enriched with data from the information models of the peripheral devices and the carried products. The information model and the approach for data enrichment are based on asset administration shell.
In contrast, there is data enrichment of the information models of operating resources that have minimal or no sensor technology. The need for a cost-efficient and robust design is often an argument against increasing the number of sensors available on such resources, as they are found in large numbers within companies, e. g. rudimentary tools, logistic carriers or pallets. In a variety of scenarios, it is advantageous to possess a more extensive set of information regarding these non-smart assets. When these assets are utilized in a production process and significantly influence the outcome, it is crucial to acquire a comprehensive knowledge of them and ensuring uninterrupted data integration.
This paper introduces a concept for generating information about non-smart assets using peripheral assets and their information models. The approach is illustrated through the example of asset tracking for non-smart work piece carriers and their transported products. An information model will be developed for the work piece carriers, which will be enriched with data from the information models of the peripheral devices and the carried products. The information model and the approach for data enrichment are based on asset administration shell.
Published in: The 34th International Conference on Flexible Automation and Intelligent Manufacturing (FAIM 2025)
16.06.2025
A Multi-Camera Vision-Based Approach for Fine-Grained Assembly Quality Control
Shashank Mishra; Ali Nazeri; Achim Wagner; Martin Ruskow; Didier Stricker; Jason Raphael Rambach
Quality control is a critical aspect of manufacturing, particularly in ensuring the proper assembly of small components in production lines. Existing solutions often rely on single-view imaging or manual inspection, which are prone to errors due to occlusions, restricted perspectives, or lighting inconsistencies. These limitations require the installation of additional inspection stations, which could disrupt the assembly line and lead to increased downtime and costs. This paper introduces a novel multi-view quality control module designed to address these challenges, integrating a multi-camera imaging system with advanced object detection algorithms. By capturing images from three camera views, the system provides comprehensive visual coverage of components of an assembly process. A tailored image fusion methodology combines results from multiple views, effectively resolving ambiguities and enhancing detection reliability. To support this system, we developed a unique dataset comprising annotated images across diverse scenarios, including varied lighting conditions, occlusions, and angles, to enhance applicability in real-world manufacturing environments. Experimental results show that our approach significantly outperforms single-view methods, achieving high precision and recall rates in the identification of improperly fastened small assembly parts such as screws. This work contributes to industrial automation by overcoming single-view limitations, and providing a scalable, cost-effective, and accurate quality control mechanism that ensures the reliability and safety of the assembly line. The dataset used in this study is publicly available to facilitate further research in this domain. It can be accessed at https://cloud.dfki.de/owncloud/index.php/s/CkCHqbwPjMCsiQf.
Published in: 33rd. European Signal Processing Conference (EUSIPCO-2025), September 8-12, Palermo, Italy, IEEE, 10/2025.
11.06.2025
ToF-360 – A Panoramic Time-of-flight RGB-D Dataset for Single Capture Indoor Semantic 3D Reconstruction
Hideaki Kanayama; Mahdi Chamseddine; Suresh Guttikonda; So Okumura; Soichiro Yokota; Didier Stricker; Jason Raphael Rambach
3D scene understanding is a key research topic for various automation areas. Many RGB-D datasets today focus on reconstruction of entire scenes. However, their scanning processes are time-consuming, requiring multiple or continuous recordings using a scanner with a limited angle of view. Such datasets often contain data affected by stitching artifacts or poor quality annotation masks projected directly from 3D to image. In this paper, we present ToF-360. This is the first RGB-D dataset obtained by a unique Time-of- Flight (ToF) sensor capable of 360→ omnidirectional RGB-D scanning within seconds. In addition to the raw data in a fisheye format and equi-rectangular projection (ERP) images from the device, we provide manually labeled high-quality, pixel-level, 2D semantics and room layout annotations and introduce a benchmark for three practical tasks: 2D semantic segmentation, 3D semantic segmentation, and layout estimation. We demonstrate that our dataset helps to better represent real-world scenarios and push the limits of existing state-of-the-art methods. The dataset is publicly available at https://doi.org/10.57967/hf/5074.
12.05.2025
Comparative Analysis of Synthetic Data Generation
for Object Detection: CAD Models vs. 3D Scans of Industrial Items and Hybrid Approaches
Abdullah Farrukh, Tatjana Legler, Achim Wagner, Martin Ruskowski
Deep learning techniques, particularly in object detection, are becoming increasingly common in industrial settings. However, the challenges posed by industrial objects — such as intricate surface textures and complex geometries — often require the creation of custom training datasets. Publicly available datasets typically do not provide sufficient coverage for these unique characteristics. In many cases, producing real-world datasets for low-volume, high-variability production scenarios is both time-consuming and costly. In this paper, we evaluate the use of synthetic data generated using NVIDIA’s Isaac-Sim as an efficient alternative. We compare the use of CAD models and 3D scans of real assets, reconstructed using state-of-the-art 3D reconstructions methods, e.g. structured light scans and Neural Radiance Fields (NeRFs). For this, we utilize pre-existing hardware and software tools and set the focus on the usability in an Industry 4.0 environment. The generated synthetic datasets are used to train a YOLO-based object detection model for a worker assistance system that provides context-based assembly instructions. The model is tested with real image data of two objects with distinct surface and texture properties. Initial results demonstrate performance that exceeded expectations.
28.04.2025
Evaluating and Integrating Positioning Technologies: A Framework for Industrial Applications
Dennis Salzmann; Florian Herrmann; Christoph Fischer; Hans Dieter Schotten
This paper presents a comprehensive framework for evaluating and integrating positioning technologies tailored for industrial applications, addressing the growing demand for reliable Real-Time Locating Systems (RTLS). Despite advancements in indoor positioning technologies, the industrial sector still faces challenges such as lack of standardization, interoperability issues, and the absence of a unified evaluation method. The proposed framework includes an evaluation testbed that emulates various industrial use cases, focusing on key performance metrics like accuracy, precision, robustness, and energy efficiency. By integrating technologies such as 5G, UWB, and infrared, the framework allows the empirical testing of hybrid solutions to identify the most suitable RTLS for different industrial environments. The framework is exemplified through the Twin4Trucks project, which tests multiple positioning technologies in controlled settings, ensuring better accuracy and adaptability. This work aims to fill gaps in RTLS standardization and provide insights for future optimization of industrial positioning systems. Index Terms—Real-Time Locating Systems (RTLS), Evaluation framework, Industrial environments, Positioning technologies, Performance metrics, Design of experiments (DoE)
Published in: 2025 IEEE/ION Position, Location and Navigation Symposium (PLANS)At: Salt Lake City, UT
29.11.2024
Adapting to Changes: A Novel Framework for Continual Machine Learning in Industrial Applications
Jibinraj Antony; Dorotea Jalušić; Simon Bergweiler; Ákos Hajnal; Veronika ´labravec; Márk Emődi; Dejan Strbad; Tatjana Legler; Attila Csaba Marosi
This paper is dedicated to solving the problem of concept drift in industrial plants using artificial intelligence methods. For this purpose, methodological approaches and procedures are considered and analyzed. Based on the findings, reference architectures were developed at different abstraction levels that can be used in an industrial environment and enable continuous machine learning. Continuous machine learning offers the possibility of adapting to dynamic changes in the production environment, which are reflected in constantly changing data sets. Through a combination of machine learning techniques, a novel and practical framework for continuous learning, also known as lifelong learning, is presented. The integration of problem-focused machine learning methods is advancing in production, e.g., predictive maintenance, process optimization or fault detection. Thereby, fully or semi-automated adaptations to changing environments requiring continuous improvements are less often explored, although practical use cases often require adaptive capabilities as the physical data distribution may change over time. In this paper, the application was continuously improved based on case studies and empirical results, and finally validated with a quality assurance application. Various methods and approaches for detecting concept and data deviations, retraining, packaging and model updating had to be investigated, which led to the question of what a real industry-oriented implementation could look like. The result is a reference architecture that can run on cloud and edge computing resources. This reference architecture is validated in real-world application in the parquet production sector, proving its feasibility and efficiency.
22.11.2024
Enhancing flexibility in intralogistics 4.0 by using Services, Capabilities
Benjamin Blumhofer, Philipp Richard, Tatjana Legler, Martin Ruskowski
Manufacturing companies face a dynamic environment shaped by various factors that profoundly affect their production capacities. These factors encompass trends like shortened product life cycles, a surge in product variants, and subsequent reductions in batch sizes. Consequently, companies must adapt their operational capabilities, ensuring existing machinery remains versatile while seamlessly integrating new equipment into their production facilities, following the ”plug and produce” approach. These shifts also reverberate through intralogistics, altering flexibility requirements and methods for individualized goods handling. Despite significant progress in modeling production information using the Capability-Skill-Service model, its application in intralogistics is relatively limited to date. However, given the potential benefits, the integration of this model into both intralogistics and production promises to address one of the key challenges of Industry 4.0: the harmonization of planning and execution processes in production and intralogistics. Closing this gap, this paper proposes an architectural framework that includes core components and information models for a Capability-Skill-Service-based Intralogistics 4.0 application. This framework not only facilitates the seamless integration of intralogistics with production planning and execution, but also lays the foundation for greater flexibility and efficiency in manufacturing. With an implementation the framework and the information models are validated.
27. Juni 2024
Model Predictive Control Based Reference Generation for Optimal Proportional Integral Derivative Control
Fatos Gashi, Khalil Abuibaid, Martin Ruskowski, Achim Wagner
We introduce an alternative approach towards optimal proportional integral derivative (PID) control, consisting of model predictive control (MPC) based reference generation. To this end, we have integrated the reference as part of optimization variables of the resulting problem, where a deliberate sequence of errors is induced to obtain an optimal PID control action. In addition, the desired behavior of the PID controller is achieved without the need for internal modification of the PID gains. To better highlight the ability of coping with poor PID tuning, several test cases consisting of progressively degraded PID gains are presented. Validation of the proposed strategy is displayed by comprehensive simulations using two different plants.
17.05.2024
Resolving Symmetry Ambiguity in Correspondence-based Methods for Instance-level Object Pose Estimation
Yongliang Lin; Yongzhi Su; Sandeep Prudhvi Krishna Inuganti; Yan Di; Naeem Ajiforoushan; Hanqing Yang; Yu Zhang; Jason Raphael Rambach
Estimating the 6D pose of an object from a single RGB image is a critical task that becomes additionally challenging when dealing with symmetric objects. Recent approaches typically establish one-to-one correspondences between image pixels and 3D object surface vertices. However, the utilization of one-to-one correspondences introduces ambiguity for symmetric objects. To address this, we propose SymCode, a symmetry-aware surface encoding that encodes the object surface vertices based on one-to-many correspondences, eliminating the problem of one-to- one correspondence ambiguity. We also introduce SymNet, a fast end-to-end network that directly regresses the 6D pose parameters without solving a PnP problem. We demonstrate faster runtime and comparable accuracy achieved by our method on the T-LESS and IC-BIN benchmarks of mostly symmetric objects. The code is available at https://github.com/lyltc1/SymNet.
Published in: IEEE Transactions on Image Processing (TIP), IEEE, 3/2025.
11. August 2023
U-RED: Unsupervised 3D Shape Retrieval and Deformation for Partial Point Clouds
Yan Di, Chenyangguang Zhang, Ruida Zhang, Fabian Manhardt, Yongzhi Su, Jason Rambach, Didier Stricker, Xiangyang Ji, Federico Tombari
In this paper, we propose U-RED, an Unsupervised shape REtrieval and Deformation pipeline that takes an arbitrary object observation as input, typically captured by RGB images or scans, and jointly retrieves and deforms the geometrically similar CAD models from a pre-established database to tightly match the target. Considering existing methods typically fail to handle noisy partial observations, U-RED is designed to address this issue from two aspects. First, since one partial shape may correspond to multiple potential full shapes, the retrieval method must allow such an ambiguous one-to-many relationship. Thereby U-RED learns to project all possible full shapes of a partial target onto the surface of a unit sphere. Then during inference, each sampling on the sphere will yield a feasible retrieval. Second, since real-world partial observations usually contain noticeable noise, a reliable learned metric that measures the similarity between shapes is necessary for stable retrieval. In U-RED, we design a novel point-wise residual-guided metric that allows noise-robust comparison. Extensive experiments on the synthetic datasets PartNet, ComplementMe and the real-world dataset Scan2CAD demonstrate that U-RED surpasses existing state-of-the-art approaches by 47.3%, 16.7% and 31.6% respectively under Chamfer Distance.
01. August 2023
Skill-basierte Intralogistik: Transport von Produkten an Produktionsmodule durch mobile Roboter
Benjamin Blumhofer, Alexandra Ritter, Jesko Hermann, Martin Ruskowski
There is a trend towards developing individualized solutions in the context of product exchange between production modules and autonomous mobile robots (AMR). These solutions are typically implemented via a central controller, utilizing pre-programmed processes and a fixed physical positioning of the AMR. Unfortunately, such solutions can be expensive and difficult to transfer to other implementations. Skill-based production offers a promising alternative, enabling a transport that is vendorindependent and resilient, by utilizing horizontal communication.
Published in: atp magazin (08/2023).
Wir freuen uns auf den Austausch! Bei Fragen oder Anmerkungen zu unseren Publikationen können Sie uns jederzeit kontaktieren.