Skip to main content

· 8 min read
Doug Sillars

In our previous post on image processing workflows, we built a Netflix Conductor workflow that took an image input and then ran 2 tasks: The first task resizes and reformats the image, and the second task uploads the image to an AWS S3 bucket.

With today's varied screen sizes and varied browser support, it is a common requirement that the image processing pipeline must create multiple images with different sizes and formats of each image.

To do this with a Conductor workflow, we'll utilize the FORK operation to create parallel processes to generate multiple versions of the same image. The FORK task creates multiple parallel processes, so that each image will be created asynchronously - ensuring a fast and efficient process.

In this post, our workflow will create 2 versions of the same image - a jpg and a webp.

· 2 min read
orkes

What is Conductor​

Conductor is a Microservices orchestration platform from Netflix, released under Apache 2.0 Open Source License.

Design for failures​

Failures and service degradation are the facts of any system; this is especially true with large interconnected systems running in the cloud. Conductor is designed with principles that systems can and will go down, degrade in performance and any dependencies should be able to handle such failures.

Tasks in Conductor​

Conductor workflows are the orchestration of many activities known as tasks. Each task represents ideally a stateless worker who does the work and produces output given a specific input. The tasks are typically running outside the Conductor server and there are many factors that could affect their availability.

Designing for failures​

Conductor allows you to define your stateful applications that can handle failures and temporary degradation of services and without having to write code.

Configuring tasks to handle failures​

Each task in Conductor can be configured in a way how it responds to availability events such as:

  1. Failures
  2. Timeouts
  3. Rate limits.

Here is a sample task definition:

{
  "createdBy": "user",
  "name": "sample_task_name_1",
  "description": "This is a sample task for demo",
  "responseTimeoutSeconds": 10,
  "timeoutSeconds": 30,
  "timeoutPolicy": "TIME_OUT_WF",
  "retryCount": 3,
  "retryLogic": "FIXED",
  "retryDelaySeconds": 5,
  "rateLimitPerFrequency": 0,
  "rateLimitFrequencyInSeconds": 1
}

retry* parameters specify how to handle cases where the task execution fails and retries can be configured to be with fixed delay or exponential backoff. Similarly timeout* parameters specify how much time to give for a task to complete execution and if the task should be marked as 'Timed Out' if it runs longer than that.

More Details​

https://orkes.io/content/docs/how-tos/Tasks/task-configurations

Follow us at https://github.com/Netflix/conductor/ for the source code and updates.

· 9 min read
Doug Sillars

There are many tools available to work with images - resizing, changing the format, cropping, changing colors, etc. Tools like Photoshop require a lot of manual work to create an image. Online tools for image processing are also extremely popular. But, rather than doing the work manually, or paying for a service to modify your images, wouldn't it be cool to have a workflow that does image resizing for you automatically? In this post, we'll build just this using Conductor to orchestrate the microservices involved, and to create an API-like surface for image processing.

In this post, we'll run Conductor locally on your computer. The Conductor workflow consists of two tasks. The first task reads an image and resizes it according to the parameters provided (labeled "image_convert_resize_ref" in the image below). The second task ("image_toS3_ref" below) takes the resized image and saves it to an Amazon S3 bucket.

Image processing workflow diagram in Conductor

Using a microservice architecture for this process allows for easy swapping of components, and allows for easy extension of the workflow - easily adding additional image processing steps (or even swapping in and out different processes for different workflows). We could also easily change the location of the saved file based on different parameters.

· 6 min read
Boney Sekh

Microservices have emerged as the dominant application development paradigm in the software world today. It has tremendous benefits both from a business and technical perspective due to its fundamental characteristics of agility, scalability, and resiliency.

However, implementing microservices are hard! The inherently distributed nature of this architectural pattern introduces complexity across multiple areas especially around Transaction Management, Data Consistency, and Process Automation. In a distributed system, Business Transactions can span across multiple services. Since we no longer have the ability to run a single ACID transaction, it requires careful coordination across these services to ensure that you have a consistent and reliable system at the end of a business process.

Solutions to solve this “coordination” problem have led to the rise of a new set of application patterns that can be broadly classified into two main groups - Choreography and Orchestration.