Best Apache Airflow Alternatives in 2024

Find the top alternatives to Apache Airflow currently available. Compare ratings, reviews, pricing, and features of Apache Airflow alternatives in 2024. Slashdot lists the best Apache Airflow alternatives on the market that offer competing products that are similar to Apache Airflow. Sort through Apache Airflow alternatives below to make the best choice for your needs

  • 1
    JS7 JobScheduler Reviews
    Top Pick See Software
    Learn More
    Compare Both
    JS7 JobScheduler, an Open Source Workload Automation System, is designed for performance and resilience. JS7 implements state-of-the-art security standards. It offers unlimited performance for parallel executions of jobs and workflows. JS7 provides cross-platform job execution and managed file transfer. It supports complex dependencies without the need for coding. The JS7 REST-API allows automation of inventory management and job control. JS7 can operate thousands of Agents across any platform in parallel. Platforms - Cloud scheduling for Docker®, OpenShift®, Kubernetes® etc. - True multi-platform scheduling on premises, for Windows®, Linux®, AIX®, Solaris®, macOS® etc. - Hybrid cloud and on-premises use User Interface - Modern GUI with no-code approach for inventory management, monitoring, and control using web browsers - Near-real-time information provides immediate visibility to status changes, log outputs of jobs and workflows. - Multi-client functionality, role-based access management - OIDC authentication and LDAP integration High Availability - Redundancy & Resilience based on asynchronous design and autonomous Agents - Clustering of all JS7 Products, automatic fail-over and manual switch-over
  • 2
    Stonebranch Reviews
    See Software
    Learn More
    Compare Both
    Stonebranch’s Universal Automation Center (UAC) is a Hybrid IT automation platform, offering real-time management of tasks and processes within hybrid IT settings, encompassing both on-premises and cloud environments. As a versatile software platform, UAC streamlines and coordinates your IT and business operations, while ensuring the secure administration of file transfers and centralizing IT job scheduling and automation solutions. Powered by event-driven automation technology, UAC empowers you to achieve instantaneous automation throughout your entire hybrid IT landscape. Enjoy real-time hybrid IT automation for diverse environments, including cloud, mainframe, distributed, and hybrid setups. Experience the convenience of Managed File Transfers (MFT) automation, effortlessly managing and orchestrating file transfers between mainframes and systems, seamlessly connecting with AWS or Azure cloud services.
  • 3
    ActiveBatch Workload Automation Reviews
    See Software
    Learn More
    Compare Both
    ActiveBatch by Redwood is a centralized workload automation platform, that seamlessly connects and automates processes across critical systems like Informatica, SAP, Oracle, Microsoft and more. Use ActiveBatch's low-code Super REST API adapter, intuitive drag-and-drop workflow designer, over 100 pre-built job steps and connectors, available for on-premises, cloud or hybrid environments. Effortlessly manage your processes and maintain visibility with real-time monitoring and customizable alerts via emails or SMS to ensure SLAs are achieved. Experience unparalleled scalability with Managed Smart Queues, optimizing resources for high-volume workloads and reducing end-to-end process times. ActiveBatch holds ISO 27001 and SOC 2, Type II certifications, encrypted connections, and undergoes regular third-party tests. Benefit from continuous updates and unwavering support from our dedicated Customer Success team, providing 24x7 assistance and on-demand training to ensure your success.
  • 4
    Union Cloud Reviews

    Union Cloud

    Union.ai

    See Software
    Learn More
    Compare Both
    Union.ai Benefits: - Accelerated Data Processing & ML: Union.ai significantly speeds up data processing and machine learning. - Built on Trusted Open-Source: Leverages the robust open-source project Flyte™, ensuring a reliable and tested foundation for your ML projects. - Kubernetes Efficiency: Harnesses the power and efficiency of Kubernetes along with enhanced observability and enterprise features. - Optimized Infrastructure: Facilitates easier collaboration among Data and ML teams on optimized infrastructures, boosting project velocity. - Breaks Down Silos: Tackles the challenges of distributed tooling and infrastructure by simplifying work-sharing across teams and environments with reusable tasks, versioned workflows, and an extensible plugin system. - Seamless Multi-Cloud Operations: Navigate the complexities of on-prem, hybrid, or multi-cloud setups with ease, ensuring consistent data handling, secure networking, and smooth service integrations. - Cost Optimization: Keeps a tight rein on your compute costs, tracks usage, and optimizes resource allocation even across distributed providers and instances, ensuring cost-effectiveness.
  • 5
    Rivery Reviews

    Rivery

    Rivery

    $0.75 Per Credit
    Rivery’s ETL platform consolidates, transforms, and manages all of a company’s internal and external data sources in the cloud. Key Features: Pre-built Data Models: Rivery comes with an extensive library of pre-built data models that enable data teams to instantly create powerful data pipelines. Fully managed: A no-code, auto-scalable, and hassle-free platform. Rivery takes care of the back end, allowing teams to spend time on mission-critical priorities rather than maintenance. Multiple Environments: Rivery enables teams to construct and clone custom environments for specific teams or projects. Reverse ETL: Allows companies to automatically send data from cloud warehouses to business applications, marketing clouds, CPD’s, and more.
  • 6
    Minitab Connect Reviews
    The most accurate, complete, and timely data provides the best insight. Minitab Connect empowers data users across the enterprise with self service tools to transform diverse data into a network of data pipelines that feed analytics initiatives, foster collaboration and foster organizational-wide collaboration. Users can seamlessly combine and explore data from various sources, including databases, on-premise and cloud apps, unstructured data and spreadsheets. Automated workflows make data integration faster and provide powerful data preparation tools that allow for transformative insights. Data integration tools that are intuitive and flexible allow users to connect and blend data from multiple sources such as data warehouses, IoT devices and cloud storage.
  • 7
    KNIME Analytics Platform Reviews
    Two complementary tools, one enterprise-grade platform. Open source KNIME Analytics Platform to create data science. Commercial KNIME Server to produce data science. KNIME Analytics Platform is an open-source software that creates data science. KNIME is intuitive, open, and constantly integrating new developments. It makes data science and designing data science workflows as easy as possible. KNIME Server Enterprise Software is used to facilitate team-based collaboration, automation, and management of data science workflows, as well as the deployment and management of analytical applications and services. Non-experts have access to KNIME WebPortal and REST APIs. Extensions for KNIME Analytics Platform allow you to do more with your data. Some are created and maintained by KNIME, while others are contributed by the community or our trusted partners. Integrations are also available with many open-source projects.
  • 8
    Hevo Reviews
    Hevo Data is a no-code, bi-directional data pipeline platform specially built for modern ETL, ELT, and Reverse ETL Needs. It helps data teams streamline and automate org-wide data flows that result in a saving of ~10 hours of engineering time/week and 10x faster reporting, analytics, and decision making. The platform supports 100+ ready-to-use integrations across Databases, SaaS Applications, Cloud Storage, SDKs, and Streaming Services. Over 500 data-driven companies spread across 35+ countries trust Hevo for their data integration needs.
  • 9
    Kestra Reviews
    Kestra is a free, open-source orchestrator based on events that simplifies data operations while improving collaboration between engineers and users. Kestra brings Infrastructure as Code to data pipelines. This allows you to build reliable workflows with confidence. The declarative YAML interface allows anyone who wants to benefit from analytics to participate in the creation of the data pipeline. The UI automatically updates the YAML definition whenever you make changes to a work flow via the UI or an API call. The orchestration logic can be defined in code declaratively, even if certain workflow components are modified.
  • 10
    Kedro Reviews
    Kedro provides the foundation for clean, data-driven code. It applies concepts from software engineering to machine-learning projects. Kedro projects provide scaffolding for complex machine-learning and data pipelines. Spend less time on "plumbing", and instead focus on solving new problems. Kedro standardizes the way data science code is written and ensures that teams can collaborate easily to solve problems. You can make a seamless transition between development and production by using exploratory code. This code can be converted into reproducible, maintainable and modular experiments. A series of lightweight connectors are used to save and upload data across a variety of file formats and file systems.
  • 11
    Lyftrondata Reviews
    Lyftrondata can help you build a governed lake, data warehouse or migrate from your old database to a modern cloud-based data warehouse. Lyftrondata makes it easy to create and manage all your data workloads from one platform. This includes automatically building your warehouse and pipeline. It's easy to share the data with ANSI SQL, BI/ML and analyze it instantly. You can increase the productivity of your data professionals while reducing your time to value. All data sets can be defined, categorized, and found in one place. These data sets can be shared with experts without coding and used to drive data-driven insights. This data sharing capability is ideal for companies who want to store their data once and share it with others. You can define a dataset, apply SQL transformations, or simply migrate your SQL data processing logic into any cloud data warehouse.
  • 12
    Mage Reviews
    Mage transforms data into predictions. In minutes, you can build, train, then deploy predictive models. No AI experience necessary. You can increase user engagement by ranking content in your user's homefeed. Conversion can be increased by showing users the most relevant products to purchase. You can predict which users will quit using your app to increase retention. Matching users in a marketplace can increase conversion. Data is the most crucial part of building AI. Mage will help you navigate this process and offer suggestions on how to improve data. You will become an AI expert. AI and its predictions can be confusing. Mage will explain every metric in detail, showing you how your AI model thinks. With just a few lines code, you can get real-time predictions. Mage makes it easy to integrate your AI model into any application.
  • 13
    Apache Flink Reviews

    Apache Flink

    Apache Software Foundation

    Apache Flink is a distributed processing engine and framework for stateful computations using unbounded and bounded data streams. Flink can be used in all cluster environments and perform computations at any scale and in-memory speed. A stream of events can be used to produce any type of data. All data, including credit card transactions, machine logs, sensor measurements, and user interactions on a website, mobile app, are generated as streams. Apache Flink excels in processing both unbounded and bound data sets. Flink's runtime can run any type of application on unbounded stream streams thanks to its precise control of state and time. Bounded streams are internal processed by algorithms and data structure that are specifically designed to process fixed-sized data sets. This results in excellent performance. Flink can be used with all of the resource managers previously mentioned.
  • 14
    Flowable Reviews
    Outstanding customer service and operational excellence can help you grow your business and attract new customers. Leading organizations worldwide are turning to Flowable's Intelligent Business Automation solutions to transform their business processes in today's competitive market. Delivering exceptional customer service is key to increasing customer retention and acquisition. Operational Excellence is achieved by improving business efficiency and reducing costs. Increasing Business Agility to adapt and respond to changing market conditions. To ensure business continuity, enforce Business Compliance. Flowable's conversational engagement capabilities allow you to deliver a compelling combination of automated and personal service via popular chat platforms like WhatsApp - even in highly-regulated sectors. Flowable is lightning fast and has many years of experience in real-world applications. It supports decision, case, and process modeling and can handle complex case management situations.
  • 15
    StackStorm Reviews
    StackStorm connects your apps, services, workflows, and more. StackStorm allows you to automate DevOps in your own way. StackStorm connects to what you already have, so there's no need to modify your existing processes. A great product is built on community. StackStorm has been used by many people around the globe. You can always count on receiving answers to your questions. Stackstorm can automate and streamline any aspect of your business. Here are some of the most popular applications. StackStorm is able to act as Tier 1 support when there are failures. It can troubleshoot, fix known problems and escalate to humans when necessary. Continuous deployment can be more complex than Jenkins or other specialized opinionated tools. Automate complex CI/CD pipelines. ChatOps combines automation and collaboration; it transforms devops teams to get more done faster and with more style.
  • 16
    Flyte Reviews
    The workflow automation platform that automates complex, mission-critical data processing and ML processes at large scale. Flyte makes it simple to create machine learning and data processing workflows that are concurrent, scalable, and manageable. Flyte is used for production at Lyft and Spotify, as well as Freenome. Flyte is used at Lyft for production model training and data processing. It has become the de facto platform for pricing, locations, ETA and mapping, as well as autonomous teams. Flyte manages more than 10,000 workflows at Lyft. This includes over 1,000,000 executions per month, 20,000,000 tasks, and 40,000,000 containers. Flyte has been battle-tested by Lyft and Spotify, as well as Freenome. It is completely open-source and has an Apache 2.0 license under Linux Foundation. There is also a cross-industry oversight committee. YAML is a useful tool for configuring machine learning and data workflows. However, it can be complicated and potentially error-prone.
  • 17
    Stitch Reviews
    Stitch is a cloud-based platform that allows you to extract, transform, load data. Stitch is used by more than 1000 companies to move billions records daily from SaaS databases and applications into data warehouses or data lakes.
  • 18
    SnapLogic Reviews
    SnapLogic is easy to use, quickly ramp up and learn. SnapLogic allows you to quickly create enterprise-wide apps and data integrations. You can easily expose and manage APIs that expand your world. Reduce the manual, slow, and error-prone processes and get faster results for business processes like customer onboarding, employee off-boarding, quote and cash, ERP SKU forecasting and support ticket creation. You can monitor, manage, secure and govern all your data pipelines, API calls, and application integrations from one single window. Automated workflows can be created for any department in your enterprise within minutes, not days. SnapLogic platform can connect employee data from all enterprise HR apps and data sources to deliver exceptional employee experiences. Discover how SnapLogic can help create seamless experiences powered with automated processes.
  • 19
    Prefect Reviews

    Prefect

    Prefect

    $0.0025 per successful task
    Prefect Cloud is a command centre for your workflows. You can instantly deploy from Prefect core to gain full control and oversight. Cloud's beautiful UI allows you to keep an eye on your infrastructure's health. You can stream real-time state updates and logs, launch new runs, and get critical information right when you need it. Prefect Cloud's managed orchestration ensures that your code and data are safe while Prefect Cloud's Hybrid Model keeps everything running smoothly. Cloud scheduler runs asynchronously to ensure that your runs start on the right time every time. Advanced scheduling options allow you to schedule parameter values changes and the execution environment for each run. You can set up custom actions and notifications when your workflows change. You can monitor the health of all agents connected through your cloud instance and receive custom notifications when an agent goes offline.
  • 20
    StreamSets Reviews

    StreamSets

    StreamSets

    $1000 per month
    StreamSets DataOps Platform. An end-to-end data integration platform to build, run, monitor and manage smart data pipelines that deliver continuous data for DataOps.
  • 21
    Oracle Data Integrator Reviews
    Oracle Data Integrator is a comprehensive platform for data integration that can handle all data requirements. This includes high-volume, high performance batch loads, event-driven, trickle feed integration processes, and SOA-enabled services. Oracle Data Integrator 12c (ODI12c), the latest version of Oracle’s strategic Data Integration offering, offers superior developer productivity and an improved user experience. It features a redesigned flow-based declarative user user interface and deeper integrations with Oracle GoldenGate. ODI12c builds on its flexible, high-performance architecture that provides comprehensive big data support and enhanced parallelism for data integration processes. It also includes interoperability to Oracle Warehouse Builder (OWB), allowing customers of OWB to migrate quickly and easily to ODI12c. ODI 12c can also be integrated with Oracle Enterprise Manager 12c to allow monitoring of all Oracle technologies and applications.
  • 22
    Meltano Reviews
    Meltano offers the most flexibility in deployment options. You control your data stack from beginning to end. Since years, a growing number of connectors has been in production. You can run workflows in isolated environments and execute end-to-end testing. You can also version control everything. Open source gives you the power and flexibility to create your ideal data stack. You can easily define your entire project in code and work confidently with your team. The Meltano CLI allows you to quickly create your project and make it easy to replicate data. Meltano was designed to be the most efficient way to run dbt and manage your transformations. Your entire data stack can be defined in your project. This makes it easy to deploy it to production.
  • 23
    n8n Reviews

    n8n

    n8n

    $20 per month
    Build complex automations up to 10x faster without battling APIs. The days of slogging through a mess of scripts is over. JavaScript is the best choice when you want flexibility, and UI for all other tasks. n8n lets you build flexible workflows that are focused on deep data integration. With a user-friendly interface and templates that can be shared, even the less technically minded members of your team can work together on them. Contrary to other tools, complexity does not limit you. You can build anything you want, without worrying about budget. Connect APIs without code to automate basic functions. You can also use vanilla Javascript to manipulate complex data. You can implement more than one trigger. You can branch and merge your workflows. You can even pause flows while you wait for external events. Custom HTTP requests allow you to easily interface with any API or service. Separate dev and production environments with different sets of authentication data to avoid breaking live workflows.
  • 24
    Beamer Reviews
    Easily update and engage users. You can easily announce your latest updates and receive powerful feedback using an in-app notification centre, widgets, and changelog. Install on your website or in-app to allow users to see announcements in context. Public page with your domain, custom appearance, and SEO optimization. Send out important updates and news to your site visitors and users. Schedule and schedule posts. To get more engagement, use visual content such as images, videos and animated gifs. Segmentation allows you to send targeted notifications. Create custom segments by industry and product. Get better results by sending more relevant notifications. Push notifications can be used to bring back users. Send web push notifications directly to users and website visitors to ensure they receive your announcements, even if they're not on your site. Get feedback about your latest updates and news.
  • 25
    Alooma Reviews
    Alooma allows data teams visibility and control. It connects data from all your data silos into BigQuery in real-time. You can set up and flow data in minutes. Or, you can customize, enrich, or transform data before it hits the data warehouse. Never lose an event. Alooma's safety nets make it easy to handle errors without affecting your pipeline. Alooma infrastructure can handle any number of data sources, low or high volume.
  • 26
    dbt Reviews

    dbt

    dbt Labs

    $50 per user per month
    Data teams can collaborate as software engineering teams by using version control, quality assurance, documentation, and modularity. Analytics errors should be treated as serious as production product bugs. Analytic workflows are often manual. We believe that workflows should be designed to be executed with one command. Data teams use dbt for codifying business logic and making it available to the entire organization. This is useful for reporting, ML modeling and operational workflows. Built-in CI/CD ensures data model changes are made in the correct order through development, staging, production, and production environments. dbt Cloud offers guaranteed uptime and custom SLAs.
  • 27
    Airbyte Reviews

    Airbyte

    Airbyte

    $2.50 per credit
    All your ELT data pipelines, including custom ones, will be up and running in minutes. Your team can focus on innovation and insights. Unify all your data integration pipelines with one open-source ELT platform. Airbyte can meet all the connector needs of your data team, no matter how complex or large they may be. Airbyte is a data integration platform that scales to meet your high-volume or custom needs. From large databases to the long tail API sources. Airbyte offers a long list of connectors with high quality that can adapt to API and schema changes. It is possible to unify all native and custom ELT. Our connector development kit allows you to quickly edit and create new connectors from pre-built open-source ones. Transparent and scalable pricing. Finally, transparent and predictable pricing that scales with data needs. No need to worry about volume. No need to create custom systems for your internal scripts or database replication.
  • 28
    Activiti Reviews
    Businesses solve automation problems in distributed, highly-scalable, and cost-effective infrastructures. Activiti is a lightweight, java-centric, open-source BPMN engine that supports real-world process automation requirements. Activiti Cloud, the next generation of business automation platform, offers a set cloud native building blocks that can be used on distributed infrastructures. Inmutable, scalable and pain-free Process & Decision Runtimes that can be integrated with your cloud native infrastructure. Scalable, storage-independent and extensible audit service. Scalable, storage-independent and extensible query service. A simplified system to system interaction that scales in distributed environments. Distributed & Scalable application aggregation layer. Cloud ready secure WebSocket, subscription handling as part GraphQL integration.
  • 29
    Apache Spark Reviews

    Apache Spark

    Apache Software Foundation

    Apache Spark™, a unified analytics engine that can handle large-scale data processing, is available. Apache Spark delivers high performance for streaming and batch data. It uses a state of the art DAG scheduler, query optimizer, as well as a physical execution engine. Spark has over 80 high-level operators, making it easy to create parallel apps. You can also use it interactively via the Scala, Python and R SQL shells. Spark powers a number of libraries, including SQL and DataFrames and MLlib for machine-learning, GraphX and Spark Streaming. These libraries can be combined seamlessly in one application. Spark can run on Hadoop, Apache Mesos and Kubernetes. It can also be used standalone or in the cloud. It can access a variety of data sources. Spark can be run in standalone cluster mode on EC2, Hadoop YARN and Mesos. Access data in HDFS and Alluxio.
  • 30
    Apache Gobblin Reviews

    Apache Gobblin

    Apache Software Foundation

    A distributed data integration framework which simplifies common Big Data integration tasks such as data ingestion and replication, organization, and lifecycle management. It can be used for both streaming and batch data ecosystems. It can be run as a standalone program on a single computer. Also supports embedded mode. It can be used as a mapreduce application on multiple Hadoop versions. Azkaban is also available for the launch of mapreduce jobs. It can run as a standalone cluster, with primary and worker nodes. This mode supports high availability, and can also run on bare metals. This mode can be used as an elastic cluster in the public cloud. This mode supports high availability. Gobblin, as it exists today, is a framework that can build various data integration applications such as replication, ingest, and so on. Each of these applications are typically set up as a job and executed by Azkaban, a scheduler.
  • 31
    AWS Step Functions Reviews
    AWS Step Functions, a serverless function orchestrator, makes it easy to sequence AWS Lambda and multiple AWS services into business critical applications. It allows you to create and manage a series event-driven and checkpointed workflows that maintain the application's state. The output of each step acts as an input for the next. Your business logic dictates that each step of your application runs in the right order. It can be difficult to manage a series serverless applications, manage retries, or debugging errors. The complexity of managing distributed applications increases as they become more complex. Step Functions, which has built-in operational controls manages state, sequencing, error handling and retry logic. This removes a significant operational burden from your staff. AWS Step Functions allows you to create visual workflows that allow for fast translation of business requirements into technical specifications.
  • 32
    AWS Glue Reviews
    AWS Glue, a fully managed extract-transform-and-load (ETL) service, makes it easy for customers prepare and load their data for analysis. With just a few clicks, you can create and run ETL jobs. AWS Glue simply points to the AWS Data Catalog and AWS Glue finds your data and stores metadata (e.g. AWS Glue Data Catalog contains the table definition and schema. Once your data has been cataloged, it is immediately searchable and queryable. It is also available for ETL.
  • 33
    Dagster Cloud Reviews
    Dagster is the cloud-native open-source orchestrator for the whole development lifecycle, with integrated lineage and observability, a declarative programming model, and best-in-class testability. It is the platform of choice data teams responsible for the development, production, and observation of data assets. With Dagster, you can focus on running tasks, or you can identify the key assets you need to create using a declarative approach. Embrace CI/CD best practices from the get-go: build reusable components, spot data quality issues, and flag bugs early.
  • 34
    Argo Reviews
    Open-source tools for Kubernetes that allow you to manage clusters, run workflows, and do GitOps right. Kubernetes native workflow engine that supports DAG and step-based workflows. Continuous delivery with fully-loaded UI. Advanced Kubernetes deployment strategies like Blue-Green and Canary made easy. Argo Workflows, an open-source container native workflow engine, is used to orchestrate parallel Kubernetes jobs. Argo Workflows can be used as a Kubernetes CDD. Multi-step workflows can be modeled as a sequence of tasks, or you can capture the dependencies between tasks with a graph (DAG). Argo Workflows for Kubernetes make it easy to run complex jobs such as data processing or machine learning in a fraction the time. Kubernetes can run CI/CD pipelines directly without the need to configure complex software development products. Designed from the ground-up for containers without the overhead or limitations of legacy VMs and server-based environments.
  • 35
    Dataplane Reviews
    Dataplane's goal is to make it faster and easier to create a data mesh. It has robust data pipelines and automated workflows that can be used by businesses and teams of any size. Dataplane is more user-friendly and places a greater emphasis on performance, security, resilience, and scaling.
  • 36
    Databand Reviews
    Monitor your data health, and monitor your pipeline performance. Get unified visibility for all pipelines that use cloud-native tools such as Apache Spark, Snowflake and BigQuery. A platform for Data Engineers that provides observability. Data engineering is becoming more complex as business stakeholders demand it. Databand can help you catch-up. More pipelines, more complexity. Data engineers are working with more complex infrastructure and pushing for faster release speeds. It is more difficult to understand why a process failed, why it is running late, and how changes impact the quality of data outputs. Data consumers are frustrated by inconsistent results, model performance, delays in data delivery, and other issues. A lack of transparency and trust in data delivery can lead to confusion about the exact source of the data. Pipeline logs, data quality metrics, and errors are all captured and stored in separate, isolated systems.
  • 37
    Astera Centerprise Reviews
    Astera Centerprise, a complete on-premise data management solution, helps to extract, transform profile, cleanse, clean, and integrate data from different sources in a code-free, drag and drop environment. This software is specifically designed for enterprise-level data integration and is used by Fortune 500 companies like Wells Fargo and Xerox, HP, as well as other large corporations such as Xerox, HP, HP, and many others. Enterprises can quickly access accurate, consolidated data to support their day-today decision-making at lightning speed through process orchestration, workflow automation and job scheduling.
  • 38
    Amazon MWAA Reviews

    Amazon MWAA

    Amazon

    $0.49 per hour
    Amazon Managed Workflows (MWAA), a managed orchestration service that allows Apache Airflow to create and manage data pipelines in the cloud at scale, is called Amazon Managed Workflows. Apache Airflow is an open source tool that allows you to programmatically create, schedule, and monitor a series of processes and tasks, also known as "workflows". Managed Workflows lets you use Airflow and Python to create workflows and not have to manage the infrastructure for scalability availability and security. Managed Workflows automatically scales the workflow execution to meet your requirements. It is also integrated with AWS security services, which allows you to have fast and secure access.
  • 39
    Chalk Reviews
    Data engineering workflows that are powerful, but without the headaches of infrastructure. Simple, reusable Python is used to define complex streaming, scheduling and data backfill pipelines. Fetch all your data in real time, no matter how complicated. Deep learning and LLMs can be used to make decisions along with structured business data. Don't pay vendors for data that you won't use. Instead, query data right before online predictions. Experiment with Jupyter and then deploy into production. Create new data workflows and prevent train-serve skew in milliseconds. Instantly monitor your data workflows and track usage and data quality. You can see everything you have computed, and the data will replay any information. Integrate with your existing tools and deploy it to your own infrastructure. Custom hold times and withdrawal limits can be set.
  • 40
    Google Cloud Composer Reviews

    Google Cloud Composer

    Google

    $0.074 per vCPU hour
    Cloud Composer's managed nature with Apache Airflow compatibility allow you to focus on authoring and scheduling your workflows, rather than provisioning resources. Google Cloud products include BigQuery, Dataflow and Dataproc. They also offer integration with Cloud Storage, Cloud Storage, Pub/Sub and AI Platform. This allows users to fully orchestrate their pipeline. You can schedule, author, and monitor all aspects of your workflows using one orchestration tool. This is true regardless of whether your pipeline lives on-premises or in multiple clouds. You can make it easier to move to the cloud, or maintain a hybrid environment with workflows that cross over between the public cloud and on-premises. To create a unified environment, you can create workflows that connect data processing and services across cloud platforms.
  • 41
    Yandex Data Proc Reviews
    Yandex Data Proc creates and configures Spark clusters, Hadoop clusters, and other components based on the size, node capacity and services you select. Zeppelin Notebooks and other web applications can be used to collaborate via a UI Proxy. You have full control over your cluster, with root permissions on each VM. Install your own libraries and applications on clusters running without having to restart. Yandex Data Proc automatically increases or decreases computing resources for compute subclusters according to CPU usage indicators. Data Proc enables you to create managed clusters of Hive, which can reduce failures and losses due to metadata not being available. Save time when building ETL pipelines, pipelines for developing and training models, and describing other iterative processes. Apache Airflow already includes the Data Proc operator.
  • 42
    Activeeon ProActive Reviews
    ProActive Parallel Suite, a member of the OW2 Open Source Community for acceleration and orchestration, seamlessly integrated with the management and operation of high-performance Clouds (Private, Public with bursting capabilities). ProActive Parallel Suite platforms offer high-performance workflows and application parallelization, enterprise Scheduling & Orchestration, and dynamic management of private Heterogeneous Grids & Clouds. Our users can now simultaneously manage their Enterprise Cloud and accelerate and orchestrate all of their enterprise applications with the ProActive platform.
  • 43
    Nextflow Reviews
    Data-driven computational pipelines. Nextflow allows for reproducible and scalable scientific workflows by using software containers. It allows adaptation of scripts written in most common scripting languages. Fluent DSL makes it easy to implement and deploy complex reactive and parallel workflows on clusters and clouds. Nextflow was built on the belief that Linux is the lingua Franca of data science. Nextflow makes it easier to create a computational pipeline that can be used to combine many tasks. You can reuse existing scripts and tools. Additionally, you don't have to learn a new language to use Nextflow. Nextflow supports Docker, Singularity and other containers technology. This, together with integration of the GitHub Code-sharing Platform, allows you write self-contained pipes, manage versions, reproduce any configuration quickly, and allow you to integrate the GitHub code-sharing portal. Nextflow acts as an abstraction layer between the logic of your pipeline and its execution layer.
  • 44
    Automate Schedule Reviews
    Automated workload automation for centralized Linux job scheduling. Your IT team will be able to automate all of your workflows on Windows, UNIX and Linux systems using a job scheduler. This will allow them to focus more on strategic projects that have a direct impact on the bottom line. You can use Windows Task Scheduler or cron to create isolated job schedules across your entire organization. Integrating your job scheduler with other key software applications makes it easier to see the whole picture, use data across the organization, unify job schedules, and leverage data. You can achieve your automation goals by being more efficient. Automated job scheduling simplifies your life and transforms how you do business. You can create dynamic, event-driven job plans across servers and take dependencies into consideration. This will help you achieve your business goals through better workflows. Automate Schedule provides high availability for both a master and standby server, so important tasks can continue even if there is an outage.
  • 45
    JAMS Reviews
    JAMS is a central workload automation and job scheduling system that runs, monitors and manages jobs and workflows that support business processes. JAMS is an enterprise job scheduling software that automates IT processes. It can handle simple batch processes, scripts, and complex cross-platform workflows. JAMS integrates seamlessly with other technologies in your company to provide seamless, unattended job execution. JAMS allocates resources to run jobs in a specific sequence, at a certain time, or based upon a trigger. The JAMS job scheduler allows you to manage and monitor critical batch processes from one central console. JAMS manages your entire business's schedule, from simple command lines to multi-step tasks that use ERPs, databases and BI tools. It's easy to transfer tasks from Windows Task Scheduler or SQL Agent or Cron using built-in conversion utilities. This allows you to keep jobs running without having to do any heavy lifting.
  • 46
    Argent Reviews
    Argent Guardian®, Ultra is the most scalable monitoring solution available for all Windows, Linux and UNIX (AIX HP-UX SCO Solaris, SCO, Solaris), and iSeries Servers. Argent Guardian®, Ultra monitors servers without the need to install agents. This patented agent-optional architecture gives customers the flexibility to customize their monitoring architecture. It is no longer necessary to manually schedule and manage batch processes. Automation of business processes lowers IT costs, improves IT service, and helps with compliance requirements. Argent Queue Engine and Argent Job Scheduler automate business processes. They alert customers via Argent Console when problems occur and provide Service Level Agreements to ensure that management has a complete view of IT. Argent Job Scheduler is a single point for control over all operating systems, applications, and databases for Windows and Linux, Solaris. It also supports SCO, SCO, iSeries Servers, HP-UX and AIX.
  • 47
    Astro Reviews
    Astronomer is the driving force behind Apache Airflow, the de facto standard for expressing data flows as code. Airflow is downloaded more than 4 million times each month and is used by hundreds of thousands of teams around the world. For data teams looking to increase the availability of trusted data, Astronomer provides Astro, the modern data orchestration platform, powered by Airflow. Astro enables data engineers, data scientists, and data analysts to build, run, and observe pipelines-as-code. Founded in 2018, Astronomer is a global remote-first company with hubs in Cincinnati, New York, San Francisco, and San Jose. Customers in more than 35 countries trust Astronomer as their partner for data orchestration.
  • 48
    Ctfreak Reviews

    Ctfreak

    JYP Software

    $109/year/instance
    Are you tired of managing multiple crontabs Would you like to receive a Slack message when one of your backups is lost? Ctfreak lets you quickly schedule and edit multiple types of tasks. - Execution of Powershell or Bash scripts via SSH on thousands of servers - Execution of SQL scripts on multiple databases - Generating Chart reports from SQL queries - Webhook call - Workflow to execute concurrent or sequential tasks Not to be missed: - A mobile-friendly interface OpenID Connect - Single Sign-On Notifications via Discord/Slack/Mattermost/Email REST API - Incoming Webhooks (Github/ Gitlab/ ...) - Log retrieval and consultation - Project management of user rights
  • 49
    Azure Event Hubs Reviews

    Azure Event Hubs

    Microsoft

    $0.03 per hour
    Event Hubs is a fully managed, real time data ingestion service that is simple, reliable, and scalable. Stream millions of events per minute from any source to create dynamic data pipelines that can be used to respond to business problems. Use the geo-disaster recovery or geo-replication features to continue processing data in emergencies. Integrate seamlessly with Azure services to unlock valuable insights. You can allow existing Apache Kafka clients to talk to Event Hubs with no code changes. This allows you to have a managed Kafka experience, without the need to manage your own clusters. You can experience real-time data input and microbatching in the same stream. Instead of worrying about infrastructure management, focus on gaining insights from your data. Real-time big data pipelines are built to address business challenges immediately.
  • 50
    Automic Automation Reviews
    Enterprises must automate a diverse and complex landscape of platforms, applications, and technologies in order to provide services in a digitally competitive business environment. Service Orchestration and Automation platforms are vital to scale your IT operations, and gain greater value from automation. Complex workflows must be managed across platforms, ERP systems, business applications from mainframe to microservices, multi-cloud, and other platforms. You must streamline your big data pipelines to enable self-services by data scientists and strong governance over data flows. You are required to provide compute, network, and storage resources on-prem as well as in the cloud for development or business users. Automic Automation provides the speed, reliability, and agility required to automate your digital business. Automic centrally offers the automation and orchestration capabilities required to accelerate your digital transformation from a single platform.