Skip to main content

· 3 min read

We're excited to share that after months of beta testing with Conductor users, Orkes Cloud is live today and ready to be battle tested with your Conductor deployment. Orkes Cloud delivers Conductor as a hosted service and abstracts away set up, tuning, patching, and managing high performance Conductor clusters. Enterprise features like role-based access control and single sign-on give you peace of mind. You have complete control over where you host your data and compute and can scale seamlessly based on your needs. We’re offering a free plan to get you started, or if you need a more all-encompassing enterprise plan you’ll only pay for what you use.

We hope you'll sign up for Orkes Cloud today and give us your feedback and feature requests. We're committed to supporting the Conductor community by giving back to the open-source project and collaborating with Netflix on the Conductor roadmap. Our ultimate goal in delivering Conductor as a hosted service is to help you delegate the operational complexities to Orkes so you can get back to building the next big thing.

Orkes has also closed $9.3 million in funding, an important milestone for our company as we continue to invest in building out our team, creating an enterprise ecosystem, and investing further in the open-source project and community.

We're honored to be working with some of the most pedigreed investors in enterprise cloud infrastructure. Dharmesh Thakker from Battery Ventures, Sandeep Bhadra of Vertex Ventures US, notable cloud angel investors Mahendra Ramsinghani and Gokul Rajaram, and seasoned executives at Fortune 100 companies have not just invested in Orkes, but were our early champions as we took on this endeavor. They shared our belief that the Conductor community deserves more cloud-native support and enterprise-grade features.

We always knew that what we built with Conductor at Netflix was special and important, but not just how much so until we dispersed to other large, fast-growth companies. Everywhere we looked, companies were moving toward a microservices-based, cloud-hosted architecture, offering unprecedented flexibility but with unique challenges to orchestrate all of those disparate pieces in a resilient, scalable fashion.

Conductor gets developers 90% of the way there. Because Conductor was built to scale with Netflix's massive growth, it is infinitely scalable to any level of complexity or bandwidth. However, the developers using Conductor OSS also have to spend a significant amount of time diverting their attention away from building applications to managing hosting, availability, cluster management and security patching. It's not time well spent, it's work that is a chore, and it's not the fun or strategic part of building applications.

Orkes Founders

Last year, I connected with my former colleagues at Netflix, Viren Baraiya, who originally came up with the idea for Conductor, and Boney Sekh, who was instrumental as we built it out, to talk about what we were seeing. We felt like we had a responsibility to help solve these operational problems for teams adopting Conductor and a cloud-based microservices architecture. When I reached out to my former colleague at Microsoft, Dilip Lukose, who also saw the growth of microservices himself as an early product leader at AWS, he was also excited about the potential to fill a critical hole in the market.

That's why we created Orkes. We can't wait to hear what you think and look forward to growing with you as we evolve Orkes Cloud to better serve your needs.

· 4 min read
Doug Sillars

In early Feb 2022, we had our first meetup on Netflix Conductor: Using Conductor in Production. It was also co-hosted by the Netflix Conductor team.

We had two excellent talks from Maros Marasalek at FRINX and Nick Tomlin at Netflix.

After the two talks, we had 2 roadmap sessions. The first session was by the Netflix Conductor team - where they discussed recent releases, and walked through the Open Source roadmap for the coming months. The second session, by our own Viren Baraiya, introduced Orkes, and our plans for extending Netflix Conductor.

Conductor in FRINX

Maros' presentation showed how the FRINX team has integrated Conductor into their product, and how the FRINX tooling helps to build custom workflows.

Bridging human and system workflows with Conductor

Nick Tomlin works on the Finance team at Netflix, and his team has built a set of Conductor workflows that enable other Netflix teams to quickly build and share workflows.

Netflix Conductor roadmap

Our third talk was from the Netflix Conductor team - where they presented the roadmap for Netflix Conductor for the coming months.

Introducing Orkes

Finally, one of our founders (and the committer of the first line of Conductor code) Viren Baraiya presented Orkes and our roadmap:

Q&A

Throughout the meetup, the attendees asked a number of great questions. They are reproduced here for visibility outside the meetup.

Hello, thanks for organizing such an event. I would like to know if there are any performance metrics for Conductor? We are planning to use it in a system with heavy traffic(multimillion requests each of them would trigger a workflow) and I would like to know if it’ll be reliable enough. Thank you :)

Conductor is horizontally scalable and we have known users scaling to handle workloads at the scale you mentioned. Here is a recent discussion on scale on Github, and a post from Netflix talking about the scale.

Conductor was built ground up for high reliability and performance. There are several companies that are running multi-million workflows on their core business flows.

I hear Netflix also is using Temporal.io which is cadence based. https://www.youtube.com/watch?v=JQ6FRTnQWFI How do the two overlap at Netflix and the reason behind the use of Temporal?

Conductor is the default workflow orchestration tool of choice at Netflix. That said, engineers are free to choose the tool that’s best for their needs. The Spinnaker team at Netflix felt that Temporal is best aligned with their needs.

You mention CLI. Any thoughts about SDK as well?

Yes, we are working on that - stay tuned :) (Note: Check out the Orkes video above on things we are planning for next couple quarters)

Can AWS Lambda be hooked up as a task in workflow

AWS Lambda can be hooked up, but it's not available as a default task. There is an extension available that will allow integrating with AWS lambdas. We can send you details about this.

How often are conductor versions released to the community?

Frequent releases - upto twice per month. You can see the recent releases here: https://github.com/Netflix/conductor/releases

From the previous presenter, what is the expected timeline for all those features to be released?

Most of the features we mentioned today will be released over the next 2-3 quarters in phases.

Any pointers about production level setup instructions of Dynomite (with the consideration of DR) on K8s cluster?

Please connect with us on our community slack channel or via Github discussions. Dynomite is not actively maintained anymore and is on its way towards being deprecated. There are alternatives that will offer similar features of Dynomite.

Thank you

Thank you to all the speakers for their incredible presentations, and also to our community for such great questions. This is our first of many meetups, and we hope to see all of you (and everyone else) at our next meetup.

In the meantime - don't forget to Star the Conductor repository.

For the latest updates on Conductor and Orkes, please subscribe to our YouTube channel, and follow us on Twitter.

· 5 min read
Doug Sillars

In our initial image processing workflow using Netflix Conductor, we initially built a workflow that takes one image, resizes it and uploads it to S3.

In our 2nd post, we utilized a fork to create two images in parallel. When building this workflow, we reused all of the tasks from the first workflow, connecting them in a way that allowing for parallel processing of two images at once.

In both of these workflows, two tasks are reused: image_convert_resize and upload_toS3. This is one great advantage of using microservices - we create the service once, and reuse it many times in different ways.

In this post, we'll take that abstraction a step further, and replace the tasks in the two forks with a SUB_WORKFLOW. This allows us to simplify the full workflow by abstracting a frequently used set of tasks into a single task.

· 5 min read
Doug Sillars

In recent posts, we have built several image processing workflows with Conductor. In our first post, we created an image processing workflow for one image - where we provide an image along with the desired output dimensions and format. The workflow output a link on Amazon S3 to the desired file.

In the 2nd example, we used the FORK System task to create multiple images in parallel. The number of images was hardcoded in the workflow - as FORK generates exactly as many paths as are coded into the workflow.

As number of images is hardcoded in the workflow - only 2 images are created. When it comes to image generation, there is often a need for more images (as new formats become popular) or sizes - as more screens are supported.

Luckily, Conductor supports this flexibility, and has a feature to specify the number of tasks to be created at runtime. In this post, we'll demonstrate the use of dynamic forks, where the workflow splitting is done at runtime.

Learn how to create a dynamic fork workflow in this post!

· 8 min read
Doug Sillars

In our previous post on image processing workflows, we built a Netflix Conductor workflow that took an image input, and then ran 2 tasks: The first task resizes and reformats the image, and the second task uploads the image to an AWS S3 bucket.

With today's varied screen sizes, and varied browser support, it is a common requirement that the image processing pipeline must create multiple images with different sizes and formats of each image.

To do this with a Conductor workflow, we'll utilize the FORK operation to create parallel processes to generate multiple versions of the same image. The FORK task creates multiple parallel processes, so each image will be created asynchronously - ensuring a fast and efficient process.

In this post, our workflow will create 2 versions of the same image - a jpg and webp.

· 2 min read
orkes

What is Conductor

Conductor is a Microservices orchestration platform from Netflix, released under Apache 2.0 Open Source License.

Design for failures

Failures and service degradation are the fact of any system, this is especially true with large interconnected systems running in cloud. Conductor is designed with principles that systems can and will go down, degrade in performance and any dependencies should be able to handle such failures.

Tasks in Conductor

Conductor workflows are orchestration of many activities known as task. Each task represents a (ideally) stateless worker that given a specific input does the work and produces output. The tasks are typically running outside of Conductor server and there are many factors that could effect their availability.

Designing for failures

Conductor allows you to define your stateful applications that can handle failures and temporary degradation of services and without having to write code for that.

Configuring tasks to handle failures

Each task in Conductor can be configured how it responds to availability events such as: 1) Failures 2) Timeouts and 3) Rate limits.

Here is a sample task definition:

{
"createdBy": "user",
"name": "sample_task_name_1",
"description": "This is a sample task for demo",
"responseTimeoutSeconds": 10,
"timeoutSeconds": 30,
"timeoutPolicy": "TIME_OUT_WF",
"retryCount": 3,
"retryLogic": "FIXED",
"retryDelaySeconds": 5,
"rateLimitPerFrequency": 0,
"rateLimitFrequencyInSeconds": 1
}

retry* parameters specify how to handle cases where the task execution fails and retries can be configured to be with fixed delay or exponential backoff. Similarly timeout* parameters specify how much time to give for task to complete execution and if the task should be marked as 'Timed Out' if it runs longer than that.

More Details

https://orkes.io/content/docs/how-tos/Tasks/task-configurations

Follow us on https://github.com/Netflix/conductor/ for the source code and updates.

· 9 min read
Doug Sillars

There are many tools available to work with images - resizing, changing the format, cropping, changing colors, etc. Tools like Photoshop require a lot of manual work to create image. Online tools for image processing are also extremely popular. But, rather than doing the work manually, or paying for a service to modify your images, wouldn't it be cool to have a workflow that does image resizing for you automatically? In this post, we'll build just this using Conductor to orchestrate the microservices involved, and to create an API-like surface for image processing.

In this post, we'll run Conductor locally on your computer. The Conductor workflow consists of two tasks. The first task reads in an image and resizes it according to the parameters provided (labeled "image_convert_resize_ref" in the image below). The second task ("image_toS3_ref" below) takes the resized image and saves the it to an Amazon S3 bucket.

Diagram of our image processing workflow

Using a microservice architecture for this process allows for easy swapping of components, and allows for easy extension of the workflow - easily adding additional image processing steps (or even swapping in and out different processes for different workflows). We could also easily change the location of the saved file based on different parameters.

· 6 min read

Microservices have emerged as the dominant application development paradigm in the software world today. It has tremendous benefits both from a business and technical perspective due to its fundamental characteristics of agility, scalability, and resiliency.

However, implementing microservices are hard! The inherently distributed nature of this architectural pattern introduces complexity across multiple areas especially around Transaction Management, Data Consistency, and Process Automation. In a distributed system, Business Transactions can span across multiple services. Since we no longer have the ability to run a single ACID transaction, it requires careful coordination across these services to ensure that you have a consistent and reliable system at the end of a business process.

Solutions to solve this “coordination” problem have led to the rise of a new set of application patterns that can be broadly classified into two main groups - Choreography and Orchestration.