.png)
In our previous blog post in this series, we presented an overview of the skill sets needed to deploy a sound cloud infrastructure foundation. Laying a strong infrastructure foundation is critical for supporting a fully featured data platform. However, even once you have laid that foundation, you will have numerous architecture and technology choices to make.
When processing robotics data, especially data from AMRs, the processing workload can be bursty. Often, robotics organizations store a significant amount of data onboard their robots and offload it all at once at the end of a shift or when the robot has access to sufficient network bandwidth. As this already creates a delay in the development team’s ability to access the data, making relevant information available as quickly as possible is important.
Event-driven architecture (EDA) in the cloud can help with such bursty workloads.
The loose coupling of components in EDA enables better fault isolation. The failure of one component doesn’t impede the operation of others. Additionally, making changes to or replacing one component has little to no impact on the rest of the system. These characteristics add a high degree of flexibility and resilience to EDA systems, which also facilitates rapid experimentation and innovation when adapting to changing requirements.
You can improve cost efficiency using EDA through resource optimization. Serverless and autoscaling cloud components let you spin up the resources that you need when you need them. You avoid costs associated with idle resources and simplify infrastructure management. Using message queues and other asynchronous communication mechanisms, you can run non-blocking operations with improved performance.
Microservices can be combined with EDA – often referred to as an event-driven microservices architecture (EDMA) – to further decentralizing processing and allowing events to be handled using independent resources. They are lightweight, focused on a specific business capability, and can be developed and deployed independently.
Microservices benefit from being technology-agnostic. The right tools can be applied to solve the problem at hand, including the language used to write the service code, the data store(s) used to persist data, and the application interface provided to consumers.
For example, you might choose Java, which has a very mature ecosystem and support for enterprise features such as security and transactions. Golang has an established, richly featured standard library that provides concurrency capabilities, networking, and data structures. If your team is more experienced in Python, you may decide to use it instead and leverage the Python ecosystem of frameworks for building microservices, such as Flask. Your choices may be determined by available SDKs or other integrations required by your system, as well as the skills within your team.
Microservice architecture provides the flexibility to select an appropriate data store based on requirements and tradeoffs between attributes such as performance, consistency, and scalability. Structured data with strong relationships between entities may be stored in a relational database, while the need to efficiently store and retrieve key-value pairs at scale would be better served by a key-value store. Graph databases are optimized for modeling and querying complex relationships between entities, and NoSQL databases are best used for unstructured or semi-structured data.
Communication choices vary also for microservices. The need for synchronous communications may lead you to use well-designed APIs with clear contracts, while message queues provide a scalable way to provide asynchronous communications.
Development teams benefit from the loose coupling of all of these components within an EDMA, as changes to one service have minimal impact on other services. However, the propagation of component types creates additional challenges.
Implementing a capability across all components requires a well-coordinated, consistent approach using a solution that supports integration with all of the required components. For example, authentication and authorization are critical to ensuring data security. A single approach and technical solution should be implemented across the entire system.
Maintenance and operationalization of an EDMA system are also challenging. Supporting CI/CD pipelines across multiple components requires effort and coordination in order to ensure that producers and consumers are in sync. Additionally, it is imperative to build these systems in such a way that they report consistent logs and metrics in order to provide the observability needed to support and troubleshoot all aspects of the system.
The Model-Prime platform leverages an EDMA and many of the technologies mentioned above to provide a comprehensive platform for managing, enriching, and analyzing robotics data. Find more details about how you can use our platform at https://docs.model-prime.com, or contact us for a demo.
