The experience shows that the requirements for modeling processes in a web shop are not completely identical, especially when it comes to medium and large e-commerce platforms. In almost every project the underlying data structures are different and the data on the store side is changing nearly every minute. In this respect, the stability of the processes behind the interfaces (catalogue import, order export, indexing, etc.) usually represents one of the major hurdles in a shopping project.

With Pacemaker, we provide a powerful platform for mapping even the most complex scenarios and importing the largest amount of data from any third-party system reliably into Magento to be processed later on.

Pacemaker enables Solution Partners and/or Shop Operators to add real value to their pipelines by providing them with the know-how they need to model processes so that even in complex scenarios they can ensure that the data displayed in the shop is valid at all times. Pacemaker sets the right pace for the processes in your shop

 

powered by YouTube: While playing this video a direct connection to one of YouTube's servers will be established. More info in our Privacy Statement (only in german).

   
powered by YouTube: While playing this video a direct connection to one of YouTube's servers will be established. More info in our Privacy Statement (only in german).
powered by YouTube: While playing this video a direct connection to one of YouTube's servers will be established. More info in our Privacy Statement (only in german).
Process transparency

Process transparency

Clear logging

Clear logging

Control of subtasks

Control of subtasks

Flexibility of processes

Flexibility of processes

Performance Execution

Performance Execution

Stability & Repeatability

Stability & Repeatability

Currently, in almost all projects processes are executed via CRON, which in e-commerce and thus in the Magento context almost always means background processes. CRON is a daemon that can be used to start programs at any time, even recurrently if desired. Very simply, Magento is triggered by CRON once per minute by default and thus the processes configured there are started.

CRON has always been an integral part of the Linux system and its ease of use and reliability make it an ideal basis for starting programs and processes. With increasing complexity, especially when it comes to larger e-commerce platforms, processes can no longer simply be started, but with increasing runtime it becomes necessary to be able to model and coordinate them. Why is this so? Long-running processes such as import, export and indexing often require exclusive database access and block each other. The result of a terminated process is often missing or inconsistent data. The biggest disaster for a retailer, for example, would be wrong or missing prices.

Through the central control of the entire pipeline, the entire process is visualized both in the backend GUI (admin) and in the CLI (developer).

 

  • Fast & easy error detection
  • Fast evaluation of log files
  • Status of each subtransaction is easier to enter

Unlike a system whose processes are triggered by CRON jobs, Pacemaker offers the ability to gain complete control over the processes through the polling principle. If this principle is consistently implemented, all processes triggered by events or the connection of third-party systems, e.g. via API, are intercepted and placed under the control of Pacemaker.

 

  • High planning reliability and reliability
  • Events are not lost
  • Time and status controlled process execution

The pipeline pattern used by Pacemaker automatically distributes server resources based on their utilization.

 

  • No planning required for scaling
  • Optimal utilization of server resources
  • Dynamic adaptation to project requirements

In order to guarantee maximum flexibility and reusability, Pacemaker was developed using a component-based approach.

 

Pacemaker 1.0 consists of the following components:

 

  • Pacemaker GUI (Backend + CLI)
  • Pipeline Definition for Catalog Import
  • Pipeline Definition for Price Import
  • Pipeline definition for importing warehouse stock
  • Process Pipelines
  • Index Suspender
  • M2IF (Professional Edition)
  • M2IF Performance Booster
  • M2IF Parallel Import (Beta)
  • M2IF Converter Library
  • M2IF Standard Connector

 

The picture on the right shows the integration of Pacemaker within the Magento software architecture. Pacemaker is compatible with Magento version 2.2 or higher in the Community and Commerce Editions.

 

The Pacemaker core component called ProcessPipelines is responsible for the coordination and execution of the modeled processes. Processes are basically any process that runs in the shop, from generating the sitemap and data indexing to importing catalog data.

 

If the M2IF standard connector, called client on the right, cannot be used for the connection of third-party systems, the data provided by the third-party system (PIM, ERP, etc.) is converted into the standard format via a specific transformer (M2IF Converter Library) and then imported again via the standard connector.

For import processes, data transformation and the Converter Library play a central role, as they translate the various data formats for the respective target systems. This is necessary because, for example, an order can have a different format in ERP than in Magento - depending on the requirements, special, configurable mappings are implemented here.

 

 

With M2IF, a free Magento 2 import framework, Pacemaker relies by default on a proven open source solution for data import. The integration into Pacemaker is not done explicitly as a Magento extension, but instead the available CLI application is used. With this kind of integration, Pacemaker strictly underlines its component-based approach and ensures a decoupling of processes and the underlying business logic.

Due to its flexibility and light weight, M2IF can easily be extended by plug-ins within projects. Since the existing functionality for importing attribute sets, attributes, categories, products, customers and customer addresses has already covered a large part of the frequently required functionality, the performance and excellent support for the development of connectors for PIM and ERP systems such as Akeneo, SAP or Dynamics, which are often necessary for connecting third-party systems, plays a central role.

Already with the Community Edition an increase in performance of up to 30 % compared to the Magento Standard Import/Export Module is possible. The M2IF Performance Booster PlugIn included in Pacemaker can increase performance by up to another 30%. In addition to performance, M2IF in combination with Pacemaker is characterized by optimized memory consumption and the ability to use Redis as a cache.

The components made available by M2IF are combined via the integrated workflow engine via configuration files in such a way that an individual solution for the respective requirements is ultimately created within the scope of the project. Due to the high flexibility and configuration possibilities, it is sufficient in many projects if data is only mapped configurationally. The extension of M2IF by additional programming is usually not necessary.

As the only solution available on the market, M2IF offers the possibility to run complete imports as part of a database transaction. This approach increases performance by collecting data at the end of the transaction and ensuring that either all or no data is imported. During the import it can still be used as usual, a disturbance for the end customer is not noticeable.

The concept behind the process pipelines has existed for a long time in the form of a design pattern and is often used in software development. Gitlab with its build pipelines is often mentioned as a prime example of a successful implementation, as many developers have already had frequent contact with it.

 

Pipelines are processes that consist of many small steps. The relationships between the processes and the steps are defined using a defined schema. This means that processes and their steps can be executed independently of each other, but at the same time controlled by a higher-level process with regard to their interdependencies to each other and to the system environment. Pacemaker also provides a uniform logging interface for pipelines so that monitoring and condition monitoring can be carried out centrally across all steps, which can be performed at intervals of days or weeks if required.

 

Pacemaker also makes use of this design pattern by relying on our Process Pipeline Extension as a central component. Like Gitlab, Pacemaker goes one step further and makes the completed and running processes visible to shop operators, editors, administrators and developers in the backend and on the console.

 

Since all steps of a pipeline are logically combined via their definition, an overview of all running processes can be displayed in the backend or via the console. The screenshot on the right shows such an overview in the backend. This allows you to quickly determine which status an order has or whether the catalog import has been successfully completed. If errors have occurred, a simple click on the failed step quickly and easily clarifies what the concrete cause was.

Pacemaker takes a completely different, much more flexible approach. For the time-controlled execution of processes, the so-called heartbeat, CRON is also used, so to speak as a clock generator. In contrast to the traditional one-process architecture, the heartbeat only checks whether a process should be started or not and does not execute it independently.

The actual execution of the process is performed by a so-called runner, which can either run on the same server or on a different server. Depending on the requirements, the number of runners can be scaled arbitrarily. This decoupling alone increases the flexibility of the processes enormously, since each process can be controlled. In addition to control, it is also possible to divide a process into any number of small steps, whereby each step can in turn be executed as a separate process. Processes, but also individual steps, can be executed in parallel. Thus, defined by the dependencies of the steps, a significantly better throughput time of the entire process chain can be achieved.

 

  1. Downloading the data from the source system - depending on requirements, this can also be several parallel downloads if the data has to be aggregated from different systems
  2. Transformation of the data: In this step the data of the source system is transformed into a suitable form for Magento. The result of the transformation CSV files for the entities, categories, attributes and products
  3. Blocking the Magento Cron indexer (TechDivsion_IndexSuspender).
  4. Import of attributes (configuration of attributes, option values for select attributes, attribute sets
  5. Import of categories
  6. Import of products (SKUs and attribute values)
  7. Media download (product pictures)
  8. Indexing (Magento- and Search-Index)
  9. Release of the Magento Cron indexer
  10. Optimization of the images

 

  1. Download the data from the source system
  2. Transformation of the data
  3. Blocking the Magento Cron indexer
  4. Import of the data
  5. Release of the Magento Cron indexer

 

 

  1. Export the order to ERP
  2. Confirm order after receipt of payment - sets step 3 to status skipped as soon as this step is completed
  3. Cancellation of the order in ERP - special step, which will only be executed if step 2 has not already been completed. After successful execution of the step, all further steps of the pipeline are set to skipped and the pipeline is completed as successful.
  4. Retrieving the invoices from the ERP (for the pipeline order)
  5. Retrieving the shipments from the ERP (for the pipeline order)
  6. Retrieving the credits from the ERP (for the pipeline order)

As a Pacemaker Solution Partner you have direct access to our team of certified software developers and architects, constantly growing documentation and best practices. Together with us, you can master even the most extensive projects. 

And best of all: The partnership is free of charge for you!

 

  • As a Pacemaker Solution Partner you benefit from the following advantages:
  • Inclusion in our list of Solution Partners 
  • Use of the name and logo for joint marketing activities
  • Support in the sales phase of your project
  • Full access to our Gitlab with the entire source code
  • The right to install Pacemaker on any number of servers for testing, development and sales purposes.
  • Access to the Pacemaker developer documentation (still under construction)
  • Non-binding and uncomplicated technical support via a closed support chat

 

For every Pacemaker license sold to a customer you have acquired, you will additionally receive

 

  • 30 % lifetime commission on the sales price
  • 5 project-related support tickets with a processing time of up to one hour per ticket

 

If the 5 support tickets included in the license price are not sufficient, you can of course book any number of tickets or, after an appropriate lead time, technical consulting by one of our software architects remotely or on-site.

By requesting the download, you give TechDivision GmbH your consent, which can be revoked at any time, that we may inform you by e-mail about our products and events on the topics of e-commerce, web, agile, etc.. You will immediately receive a confirmation email to verify your email address and then the requested document (double opt-in). We will never pass on your data to third parties. Detailed information on the contents, the logging of your registration, the sending of emails via MailChimp, as well as unsubscribe options can be found in our privacy policy.

Data import or - consistency problems?

We would be pleased to show you in a personal meeting how efficient our new product "Pacemaker for Magento" is. 

Tim Wagner CTO