Experience shows that the requirements for modeling processes in a web shop are never completely identical, especially when it comes to medium and large e-commerce platforms. In almost every project the underlying data structures are different and the data on the shop side changes almost every minute. In this respect, the stability of the processes behind the interfaces (catalogue import, order export, indexing, etc.) usually represents one of the greatest challenges in a e-commerce project.
With Pacemaker, we provide a powerful platform with which even the most complex scenarios can be mapped and the largest amount of data can be reliably imported from any third-party system into Magento for further processing.
Through Pacemaker solution partners and/or shop operators get the know-how how to model the processes as a real added value with the fully configured and delivered pipelines, so that even in complex scenarios it is ensured that the data displayed in the shop is valid at all times. Pacemaker sets the right pace for the processes in your shop.
Transparency of processes
Control of sub-processes
Flexibility of processes
Stability & Repeatability
In order to guarantee maximum flexibility and reusability, Pacemaker was developed using a component-based approach.
Pacemaker 1.0 consists of the following components:
- M2IF Standard Connector
- M2IF Converter Library
- M2IF Parallel Import (Beta)
- M2IF Performance Booster
- M2IF (Professional Edition)
- M2IF (Community Edition)
- Order Export
- Index Suspender
- Process Pipelines
- Pipeline definition for importing warehouse stock
- Pipeline Definition for Price Import
- Pipeline Definition for Catalog Import
- Pacemaker Statistics
- Pacemaker GUI (Backend + CLI)
The picture on the right shows the integration of Pacemaker within the Magento software architecture. Pacemaker is compatible with Magento version 2.2 or higher in the Community and Commerce Editions.
The Pacemaker core component called ProcessPipelines is responsible for the coordination and execution of the modeled processes. Processes are basically any process that runs in the shop, from generating the sitemap and data indexing to importing catalog data.
If the M2IF standard connector, called client on the right, cannot be used for the connection of third-party systems, the data provided by the third-party system (PIM, ERP, etc.) is converted into the standard format via a specific transformer (M2IF Converter Library) and then imported again via the standard connector.
For import processes, data transformation and the Converter Library play a central role, as they translate the various data formats for the respective target systems. This is necessary because, for example, an order can have a different format in ERP than in Magento - depending on the requirements, special, configurable mappings are implemented here.
With M2IF, a free and open source Magento 2 import framework, Pacemaker relies by default on a proven open source solution for data import. The integration into Pacemaker is not done explicitly as a Magento extension, but the available CLI application is used. With this kind of integration, Pacemaker strictly underlines its component-based approach and ensures a decoupling of processes and the underlying business logic.
Due to its flexibility and light weight, M2IF can easily be extended by plug-ins within projects. Since the existing functionality for importing attribute sets, attributes, categories, products, customers and customer addresses has already covered a large part of the frequently required functionality, the performance and excellent support for the development of connectors for PIM and ERP systems such as Akeneo, SAP or Dynamics, which are often necessary for connecting third-party systems, plays a central role.
Already with the Community Edition an increase in performance of up to 30 % compared to the Magento Standard Import/Export Module is possible. The M2IF Performance Booster PlugIn included in Pacemaker can increase performance by up to 30%. In addition to performance, M2IF in combination with Pacemaker is characterized by optimized memory consumption and the ability to use Redis as a cache.
The components made available by M2IF are combined via the integrated workflow engine by configuration files in such a way that an individual solution for the respective requirements is ultimately created within the scope of the project. Due to the high flexibility and configuration possibilities, it is sufficient in many projects if data is only mapped configurationally. The extension of M2IF by additional programming is usually not necessary.
As the only solution available on the market, M2IF offers the possibility to run complete imports as part of a database transaction. This approach increases performance by collecting data at the end of the transaction and ensuring that either all or no data is imported. During the import it can still be used as usual, a disturbance for the end customer is not noticeable.
The concept behind the process pipelines has existed for a long time in the form of a design pattern and is often used in software development. Gitlab with its build pipelines is often mentioned as a prime example of a successful implementation, as many developers have already had frequent contact with it.
Pipelines are processes that consist of many small steps. The relationships between the processes and the steps are defined using a defined schema. This means that processes and their steps can be executed independently of each other, but at the same time controlled by a higher-level process with regard to their interdependencies to each other and to the system environment. Pacemaker also provides a uniform logging interface for pipelines so that monitoring and condition monitoring can be carried out centrally across all steps, which can be carried out at intervals of days or weeks if required.
Pacemaker also makes use of this design pattern by relying on our Process Pipeline Extension as a central component. Like Gitlab, Pacemaker goes one step further and makes the completed and running processes visible to shop operators, editors, administrators and developers in the backend and on the console.
Since all steps of a pipeline are logically summarized via their definition, an overview of all running processes can be displayed in the backend or via the console. The screenshot on the right shows such an overview in the backend. This allows you to quickly determine which status an order has or whether the catalog import has been successfully completed. If errors have occurred, a simple click on the failed step quickly and easily clarifies what the concrete cause was.
Currently in almost all projects, processes are executed via CRON, which in e-commerce and thus in the Magento context almost always means background processes. CRON is a daemon that can be used to start programs at any time, even recurrently if desired. In the case of Pacemaker, we naturally speak Magento here. Very simply, Magento is triggered by CRON once per minute by default and thus the processes configured there are started.
CRON has always been an integral part of the Linux system and is an ideal basis for starting programs and processes just because of its simple operation and reliability. With increasing complexity, especially when we talk about larger e-commerce platforms, processes can no longer simply be started, but with increasing runtime it becomes necessary to be able to model and coordinate them. Why is this so? Long-running processes such as imports, exports and indexing often require exclusive database access and block each other. The result of a terminated process is often missing or inconsistent data: The biggest likely accident for a shop operator, if you think of wrong or missing prices, for example.
Based on our experience, we can say that this architecture has few advantages but all the more disadvantages.
- Simple and easy monitoring of processes via the operating system's on-board resources
- Possibility of central control by CRON when which process starts
- With CRON, only a 1-minute cycle is possible, processes cannot be directly and fine-granularly interlinked with each other.
- No or only very limited possibility for monitoring in the Magento backend
- Time- and/or logically separated steps cannot be combined, e.g. continuous order status
- Access to running processes is difficult or even impossible
- No or only limited reusability of functionality
- Splitting into smaller partial steps not possible due to lack of coordination
- No repetition of partial steps, only of the entire process
- Monolithic and long-running processes with very high complexity
Pacemaker takes a completely different, much more flexible approach. For the time-controlled execution of processes, the so-called heartbeat, CRON is also used, so to speak as a clock generator. In contrast to the traditional one-process architecture, the heartbeat only checks whether a process should be started or not and does not execute it independently.
The actual execution of the process is performed by a so-called runner, which can either run on the same server or on a different server. Depending on the requirements, the number of runners can be scaled arbitrarily. This decoupling alone increases the flexibility of the processes enormously, since each process can be controlled, so to speak. In addition to control, it is also possible to divide a process into any number of small steps, whereby each step can in turn be executed as a separate process.
Processes, but also individual steps, can be executed in parallel. Thus, defined by the dependencies of the steps, a significantly better throughput time of the entire process chain can be achieved.
- 1. Download the data from the source system - depending on requirements, this can also be several parallel downloads, since the data is aggregated from different target systems.
- 2. Transformation of the data - In this step the data of the source system is transformed into a suitable form for Magento. The result of the transformation in the example are CSV files for the entities, categories, attributes and products.
- 3. Blocking the Magento Cron Indexer (TechDivsion_IndexSuspendender)
- 4. Import of attributes (configuration of attributes, option values for select attributes, attribute sets)
- 5. Import of categories
- 6. Import of products (SKUs and attribute values)
- 7. Media download (product pictures)
- 8. Indexing (Magento- and Search-Index)
- 9. Activation of the Magento Cron indexer
- 10. Optimization of the images
- 1. Download data from source system
- 2. Transformation of data
- 3. Blocking the Magento Cron Indexer
- 4. Import of data
- 5. Activation of the Magento Cron indexer
- 1. Export the order to ERP
- 2. Confirm order after receipt of payment - sets step 3 to status skipped as soon as this step is completed
- 3. Cancellation of the order in ERP - special step, which will only be executed if step 2 has not already been completed. After successful execution of the step, all further steps of the pipeline are set to skipped and the pipeline is completed as successful.
- 4. Retrieving the invoices from the ERP (for the pipeline order)
- 5. Retrieving the shipments from the ERP (for the pipeline order)
- 6. Retrieving the credits from the ERP (for the pipeline order)
- 7. Retrieving the returns from the ERP (for the pipeline order)
As a Pacemaker Solution Partner you have direct access to our team of certified software developers and architects, constantly growing documentation and best practices. Together with us, you can master even the most extensive projects.
And best of all: The partnership is free of charge for you!
As a Pacemaker Solution Partner, you will benefit from the following advantages:
- Non-binding and uncomplicated technical support via a closed support chat
- Access to the Pacemaker developer documentation (still under construction)
- The right to install Pacemaker on any number of servers for testing, development and sales purposes.
- Full access to our Gitlab with the entire source code
- Support in the sales phase of your project
- Use of the name and logo for joint marketing activities
- Inclusion in our list of Solution Partners
For every Pacemaker license sold to a customer you have acquired, you will additionally receive
- 5 project-related support tickets with a processing time of up to one hour per ticket
- 30 % lifetime commission on the sales price
If the 5 included support tickets are not sufficient, you can of course book any number of tickets or, after an appropriate lead time, technical consulting by one of our software architects remotely or on-site.