Skip to main content

· 10 min read
Doug Sillars

In our previous post on Using Conductor to Parse Data, we discussed a Netflix Conductor workflow that extracts data from GitHub, transforms it, and then uploads the results to Orbit. This basically describes an ETL (Extract, Transform, Load) process - automated as a Conductor workflow. In this post, we'll go in-depth as to how the workflow is constructed - examining what each task does. This workflow will run daily at midnight GMT, ensuring that the data in our Orbit instance is always up to date with the data on GitHub.

· 5 min read
Doug Sillars

Data processing and data workflows are amongst the most critical processes for many companies today. Many hours are spent collecting, parsing and analyzing data sets. There is no need for these to repetitive processes to be manual - we can automate them. In this post, we'll build a Conductor workflow that handles ETL (Extraction, Transformation and Loading) data for a mission critical process here at Orkes.

As a member of the Developer Relations team here at Orkes, we use a tool called Orbit to better understand our users and our community. Orbit has a number of great integrations that allow for easy connections into platforms like Slack, Discord, Twitter and GitHub. By adding API keys from the Orkes implementations of these social media platforms, the integrations automatically update the community data from these platforms into Orbit.

This is great, but it does not solve all of our needs. Orkes Cloud is based on top of Netflix Conductor, and we'd like to also understand who is interacting and using that GitHub repository. However, since Conductor is owned by Netflix, our team is unable to leverage the automated Orbit integration.

However, our API keys do allow us to extract the data from GitHub, and our Orbit API key can allow us to upload the extracted data into our data collection. We could do this manually, but why not build a COnductor workflow to do this process for us automatically?

In this post, I'll give a high level view of the automation required for Extraction of the data from Github, Transformation the data to a form that Orbit can accept, and then to Load the data into our Orbit collection.

· 3 min read
orkes

Tl;dr We are announcing a new community contribution repository for Conductor - https://github.com/Netflix/conductor-community, and Netflix and Orkes are partnering to co-manage it.

Conductor is a workflow orchestration engine developed and open-sourced by Netflix. At Netflix, Conductor is the de facto orchestration engine, used by a number of teams to orchestrate workflows at scale. If you are new to Conductor, we highly recommend taking a look at our GitHub repository and documentation.

Conductor was designed by Netflix to be extensible, making it easy to add or change components - even major components like queues or storage. This extendability makes adding new features and tasks easy to incorporate, without affecting the Conductor core. These implementations are based on well-defined interfaces and contracts that are defined in the core conductor repository.

Since open sourcing Conductor, we have seen huge community adoption and active interest from the community providing patches, features and extensions to Conductor.
Some of the key features that have been developed and contributed by the community into Conductor open source repository today are:

  1. Support for Postgres and MySQL backends
  2. Elasticsearch 7 support
  3. Integration with AMQP and NATS queues
  4. GRPC support
  5. Support for Azure blob stores
  6. Postgres based external payload storage
  7. Do While loops
  8. External Metrics collectors such as Prometheus and Datadog
  9. Support for Kafka and JQ tasks
  10. Various bug fixes, patches including most recent the fix for log4j vulnerabilities and many other more features and fixes

The number of community contributions, especially newer implementations of the core contracts in Conductor has increased over the past few years. We love that Conductor is finding use in many other organizations (link to the list), and that these organizations are submitting their changes back to the community version of Conductor,

This increase in engagement and growth of the community, while incredible, is a double edged sword. By no means does the Conductor team want to slow or limit these contributions, but the integration of third party implementations has been slower than we would like due to the team’s bandwidth.

In order to encourage (and to speed up the integration of) community-contributions to Conductor, we are announcing a new repository dedicated to supporting community contributions. The repository will be hosted at https://github.com/Netflix/conductor-community and will be seeded with the existing community contributed modules. Further, we are partnering with Orkes (https://orkes.io/) to co-manage the community repository along with Netflix, helping us with code reviews and creating releases for the community contributions.

We think this new structure will enable us to review the PRs and community contributions in a timely manner and allow the community to be more autonomous longer term.

We will continue to publish artifacts from the community repository at the same maven coordinates under com.netflix.conductor group and the artifact names will remain the same with full binary compatibility. This means that there is no change to users of Conductor: install, updates and usage remain the same.

Please see https://github.com/Netflix/conductor-community#readme for the details on the modules and release details. You can also find FAQs that address the most common questions.

We look forward to continued engagement with the community making Conductor the best open source orchestration engine.

· 9 min read
Yulia Gavrilova

Microservice architecture is becoming more and more common in the realization of business ideas. Developers can easily add or remove features; update specific parts of applications without interfering with general workflow; and concurrently use diverse technologies and programming languages based on their business needs.

The microservice orchestrator controls the execution of processes in this distributed architecture and its role is increasingly important, as it makes it easier to uncover and fix problems, as well as manage the development of the entire system, even when we’re talking about systems that have more than one programming language in their tech stack.

In this article, you’ll learn about workflow orchestration and its benefits and limitations when building a microservice platform that features several programming languages.

· 5 min read
James Stuart

What is Conductor

Conductor is a workflow orchestration engine that connects all of your microservices together to create fully functional workflows that can run at scale. Each workflow is comprised of tasks. Tasks can be System Tasks, these are provided by Conductor, or Custom Tasks which are called Workers. These workers can be written in any language - from Conductor's point of view - data goes in, and results come out - the language that processes the data is irrelevant.

Workflows are defined in JSON and Tasks are defined in JSON. Workflows can be composed of other workflows. The way I see it everything is just data.

Clojure is a functional programming language that runs on the JVM, with really interesting features, that match really well with the way I understand Conductor. Workflows and tasks are "just data" and Clojure programs are "just data"

In the POST I want to show you how you can create tasks, workflows and run Clojure workers, with just data.

· 5 min read
Doug Sillars

What is a workflow? Wikipedia says "A workflow consists of an orchestrated and repeatable pattern of activity, enabled by the systematic organization of resources into processes that transform materials, provide services, or process information."

None of that is incorrect. But it is certainly a mouthful. If I am asked in an elevator what a workflow orchestration is, I like to use analogies to make the point as approachable as possible:

"It's like a recipe for code. A recipe has a series of steps, that must be run in a specific order. Iyt can also have different options for food allergies, or different ingredients you might have on hand? A workflow is a recipe for your code, and can be built to handle many of the changes that might reasonably occur when the code runs."

Workflows as recipes

You may have seen one of the name videos out there of parents teaching kids code by writing out the steps to create a Peanut butter and Jelly sandwich.

We're not going to be quite as silly as that Dad, but we'll run through some instructions on how to create a PB&J from Instructables..

If you look at the URL of that recipe - it appears it took 4 tries to get it right :D https://www.instructables.com/How-to-Make-a-Peanut-Butter-and-Jelly-Sandwich-4/

The steps are (skipping some substeps for clarity):

  1. Gather Your Ingredients for the Sandwich
  2. Pull Out Two Slices of Bread
  3. Open Peanut Butter and Jelly
  4. Spread the Peanut Butter Onto One Slice of Bread
  5. Spread the Jelly Onto the Other Slice of Bread
  6. Combine the Two Slices
  7. Clean Up Your Workspace
  8. Enjoy Your Sandwich

(if you read closely, I skipped 3 steps - wearing gloves, removing the crust and cutting the sandwich in half... because, well - that's all just ridiculous)

The eight steps above are a workflow. They must be performed in that order to create a sandwich - you cannot spread the PB before you lay out the bread.

Building a Conductor Workflow

We can turn these 8 steps into a Conductor workflow.

PB&J example workflow

If you'd like to see the definition of this workflow, you can check out the code on the Orkes Playground. It's free to sign up.

This workflow shows clear steps with arrows pointing to the next step - so it is visually possible to see how the workflow will progress.

Improving the workflow

Each task is run by a microservice, so making changes and adding tasks is easy to do. In this example - you see that the recipe calls for PB to be spread first, and then the jelly. This works for a single human - these steps each take 2 hands. But there is no reason that the jelly cannot go first, and THEN the PB. Or - if you had more hands - the PB and the Jelly could be spread simultaneously.

Independent tasks

Let's remove the PB dependency from the jelly spreading - as the order of PB vs. jelly does not matter.

We can do this in Conductor with a FORK. A fork splits your workflow into 2 asynchronous tasks, and then a JOIN reconnects the the workflow into a single path.

We can now apply a fork to apply PB, and a second fork to apply the jelly:

PB&J example workflow

This is version 2 of the workflow, and you can see it in the playground

Now, these operations are independent, and if there is space in the jelly task queue - that can be completed ahead of the PB.

Recipe variations

Often, recipes have variations to preparation, and they can be read like an IF statement in programming (If (fresh tomatoes) {do x}, else if (tinned tomatoes) {do y}.

Cooking a burrito

Let's look at a common example - from a burrito in my freezer. The preparation directions vary depending on the cooking method.

burrito instructions

We can emulate this in a workflow using a switch task. The Switch task takes in the workflow ovenType input ${workflow.input.ovenType} and based on this value will make a decision. The default case is for the microwave, and then second decisionCase is set to "oven". From that input, the different tasks can be run.

burrito workflow

Conclusion

Workflows are a series of tasks that must be followed in a certain order. In this post, we used cooking recipes as an analogy to a workflow - they too are a series of tasks that must be followed in a certain order.

We created sample workflows for making a peanut and jelly sandwich (version 1 and version2) and another workflow to cook a frozen burrito with microwave or oven instructions.

<<<<<<< Local Changes We are able to reuse a number of tasks:

  • The zap task is used twice in the microwave branch.
  • The bake task in the oven branch.
  • The flip task is used in both branches.

When reusing tasks, the taskReferenceName (shown at the top of the box) must be unique.======= If you're curious about how to build a workflow orchestration - it might be a fun exercise to try your favorite recipe as a workflow. You can build on the workflows from this post in our free playground. Feel free to share what you came up with in our Discord. We love seeing creative uses of workflows!>>>>>>> External Changes

· 9 min read
Paul Ibeabuchi

Microservice architecture is an architecture where an application is split into separate services, and each service is run and managed independently. In a microservice architecture, every service is focused on handling one major function and is solely responsible for its own data management.

Microservice architecture is often recommended for larger applications because it allows services to be managed by dedicated teams. Testing and deployment also become easier, as they can be carried out independently for each service without affecting the overall application.

In this article, you'll learn what a microservice architecture is and when to use it, and about workflow orchestration, its benefits, and how it can be utilized in a microservice architecture. You'll also look at an example use case of microservice architecture so you can better understand the benefits, strengths, and weaknesses of this style of architecture.

· 14 min read
Nikhila Jain

A 2020 survey by the research and advisory firm Gartner has highlighted the rapid pace of innovation in cloud computing. According to the research, forty percent of enterprise solutions will host their applications on cloud infrastructure by 2023. This shifting trend will cause an increased demand for cloud services, as well as for hybrid cloud architecture.

The hybrid cloud is gaining popularity as enterprise IT leaders seek flexible, scalable options that increase cost efficiency while maintaining control over enterprise data and information. Many organizations combine on-premise infrastructure with private/public cloud resources to meet these needs.

But without the right strategy, hybrid clouds can pose a number of challenges. Through a hypothetical case study, this article will help you learn about the strengths and limitations of hybrid cloud architecture.

· 3 min read
Doug Sillars

Conductor is a workflow orchestration engine that connects all of your microservices together to create fully functional workflows that can run at scale. Each workflow is comprised of tasks - and many of these tasks are powered by external workers - or microservices. These workers can be written in any language - from Conductor's point of view - data goes in, and results come out - the language that processes the data is irrelevant.

Each worker has to connect to your Conductor instance, and regularly poll for work in the queue. There has long been Java, Go and Python SDKs to easily connect your apps to Conductor, but for building in other languages, this code had to be created by each development team.

Today, we announce that major improvements to the Golang and Python SDKs, and announce C# and Clojure SDKs.

Further, all of the (non-Java) SDKs have a new GitHub home: the Conductor SDK repository is your new source for Conductor SDKs:

Coming soon:

· 9 min read
Azeez Lukman

Microservices are a common and popular approach to building modular, scalable software with autonomous services. Large complex products are broken down into individual services responsible for a specific business function, such as user authentication or store checkout.

A microservice-based application might require several services to interact with each other to complete a business scope. The coordination of these interactions is known as a workflow or a saga. There are two models for implementing a workflow: choreography and orchestration. With choreography, you let each part of the system inform the other of its job and let it work out the details, while with orchestration, you rely on a central brain to guide and drive the execution processes.

As orchestrated systems have grown more expansive, the problem of efficiently orchestrating related business logics has become more pronounced. In this article, you will learn about the microservice orchestration workflow and its importance in relation to modern software architecture practices.

What is Microservice Orchestration?

A microservice orchestration pattern involves a central orchestration service (the orchestrator) that typically contains the entire business workflow logic and issues commands to and awaits responses from worker microservices. Think of this as an orchestra where a central conductor is responsible for keeping the orchestra in sync and coordinating the members to produce a cohesive musical piece. Using orchestrators for your application is essential for efficiently managing applications based on microservices.

Before going into the specifics of microservice orchestration, it is helpful to familiarize yourself with the components of microservice-based architecture. For example, in a microservice-based e-commerce application, the following could come into play during the process of purchasing a product:

  • a service for listing all products;

  • a service for adding products to the cart and reserving that product from the inventory;

  • a service for handling the payment; and

  • a service that manages the shipment of the item.

Each of these microservices is autonomous. In other words, microservices can be individually scaled up or down without having to worry about the entire application. However, they are all required to interact with each other to fulfill the purchase. It might be tempting to have the services talk to each other directly as needed. However, as your architecture and the number of services grow, this can quickly get messy and difficult to maintain. This is where orchestration comes into play.

A microservice orchestration workflow is an architectural method of coordinating microservices for software systems and applications, in which loosely coupled services receive commands from a central controller, referred to as the orchestrator. The orchestrator acts as a brain, driving the execution processes; it sends a call to each service and awaits a reply before proceeding. The concept of a microservice orchestration workflow can be best described through a hypothetical use case.

microservice orchestration workflow architecture diagram

The architectural diagram of this hypothetical use case shows the interactions between the various services involved in the process when following an orchestration workflow. Looking at the diagram above:

  1. The orchestrator receives a trigger that initializes the workflow, starting with “Products Service.”

  2. When this service has created an order with the products in the customer's cart, it returns some response to the orchestrator.

  3. The orchestrator then calls the “Inventory Service” to reserve the products in the cart.

  4. Next, the orchestrator calls the “Payment Service” to handle the payment.

  5. After successful payment, the orchestrator moves on to the “Shipping Service,” which clears the products for shipment.

In a choreography workflow, on the other hand, the microservices are not managed by a central service; however, they are all aware of the business goals and rely on certain events from other services that determine how they function. Each service publishes the actions it has taken to a message stream such as SQS or Kafka. Other services subscribe and listen for events they are interested in from these streams and take the appropriate actions.

choreography orchestration workflow architecture diagram

In the choreography architecture above:

  1. The “Products Service” creates an order with the items in the customer's cart and publishes an "Order Created" event to a stream on the messaging platform.

  2. The “Inventory Service” and “Payment Service” consume from this message stream. The “Inventory Service” handles reserving the products in the cart, and the “Payment Service” handles the payment and publishes the “Payment Success” event.

  3. On receipt of an inventory “Payment Success” event, the “Shipping Service” goes ahead and clears the products that were reserved for shipment to the customer.

Why is Microservice Orchestration Important?

Microservice architecture involves decomposing your application into a set of services to improve agility and allow teams to scale. One of the main purposes of this architectural pattern is to have each service as an independently deployable component with well-defined interfaces; in this way, the scope of implemented changes can be limited to a single service.

However, you must coordinate the execution of multiple microservices to deliver the outcomes that users want, and this is why microservice orchestration is important. Orchestration allows you to put a service in charge of the other services. The service in charge is aware of the entire flow that is required and is responsible for putting the other services to work to achieve those aims.

Microservice orchestration enables you to process flows ranging from simple linear workflows to very complex dynamic workflows that run for multiple days with minimal effort and high visibility into the processes. To properly illustrate the benefits you can obtain with an orchestration workflow when managing your microservices, let’s take a look at a case study from Netflix.

Netflix is an enterprise company that has shifted toward orchestration workflows. The streaming service had traditionally used the choreography method, which involves peer-to-peer tasks that are tightly coupled; this became harder to scale with growing business needs and associated increasing complexities like determining what remains for a movie setup to be complete and updating their SLAs.

Later, Netflix switched to an orchestration workflow and eventually built their own container orchestration engine—Conductor—which has helped orchestrate over 2.6 million process flows, from simple linear workflows to complex dynamic workflows over multiple days.

Why Are So Many Developers Adopting This Architectural Paradigm?

As mentioned, there are two major techniques you can use if you need to execute many services to get your desired result. Orchestration, in which a central orchestrator component serves as the coordinator and is in charge of activating each service, and choreography, in which the services perform independently and are only loosely connected.

Developers are increasingly adopting orchestration because it has significant benefits that can make development and management easier for individual microservices without compromising the big picture. However, it should be noted that microservice orchestration is not without its limitations.

Benefits and Limitations of Microservice Orchestration

There are several benefits and challenges associated with the implementation of an orchestration workflow, many of which are related to how microservices interact with one another to achieve a business outcome.

Benefits of Microservice Orchestration

Central observability of process definition, status, and metrics: The orchestration framework can capture detailed information about each executed process instance, which it can make available for analytics. This allows you to answer questions about specific instances (such as, “Where is my order?”), as well as analytical queries (such as, how many products were ordered).

Synchronous processes: These provide a good way to control the process flow. For example, when a product’s service needs to successfully complete before the inventory service is processed.

Scalable orchestration on cloud-native platforms: When you scale up these services, you scale with errors in mind. Microservice orchestration provides you with insights into your processes, helping you coordinate various transactions that involve a large number of independent services.

Single fail point: Orchestration workflow allows you to easily trace out any error that occurs during the process flow, figure out why it failed, and debug. Writing tests for your microservices is important to help prevent errors from making it to the live service.

Limitations of Microservice Orchestration

When orchestrating microservices in an enterprise environment, you’ll find that some business functions can require hundreds or even thousands of microservices. Since the orchestration workflow is synchronous, it's possible that such processes will take a long time to finish.

Furthermore, as the orchestrator needs to communicate with each service and get a response before moving to the next, this makes services highly dependent upon each other. Failure at any point could cause the entire process to fail. While for some business processes this is required behavior, others might require the process to complete regardless; for instance, running analytics on an order that’s being processed shouldn’t prevent the checkout flow from being completed.

Who Can Benefit from Microservice Orchestration?

With microservices gradually becoming the default pattern for managing business logics, a strong architecture is needed for their coordination. Adopting an orchestration workflow could improve the seamless interaction between these services.

Many businesses still implement service-oriented architectures (SOAs) orchestrated by an enterprise service bus (ESB). However, as business needs grow, adding more business logics and microservices to the system can be challenging; the entire flow is not immediately visible, making it harder to alter a service without the risk of disrupting another.

Microservice orchestration offers a solution here, as it helps you visualize the end-to-end processes across your microservices, so you know what services would be affected by your updates, allowing you to easily address your increasing business needs.

More concretely, an orchestration workflow might be ideal for you if one or more of the following are critical for your business:

  • The ability to track and manage workflows from a single point.

  • A user interface to visualize process flows.

  • The ability to synchronously process all tasks.

  • The ability to efficiently scale to a high number of concurrently running process flows.

  • A queuing service abstracted from clients.

  • The requirement to operate services over HTTP or other transport layers such as gRPC.

Conclusion

In this article, you learned about the importance of keeping your microservices autonomous and flexible. You also learned about the use of microservice orchestration to effectively communicate, visualize, identify, and resolve the challenges of managing microservices.

The downside of the process of building an orchestration system that implements all of the features your business requires is that it’s rather complex and time consuming. A purpose-built framework offering scalable and low-overhead orchestration— like Netflix’s Conductor, is an open source tool that fits this purpose.

Orkes is a platform that offers a fully managed, cloud-hosted version of Conductor with tiered support. Orkes builds on top of Netflix Conductor to abstract out installation, tuning, patching, and managing high-performing Conductor clusters. Learn more about Orkes and get started for free within minutes here.