Automating big data workloads

A new workload automation solution provides built-in adapters to Hadoop, Sqoop, Hive, and business applications.

To keep up with growing demand for the exploration of large data sets, organizations need an easy-to-use workload automation solution. Depending on the company’s infrastructure and business needs, the tool needs to orchestrate processes involving a variety of technologies and workloads.

The Cisco® Tidal Enterprise Scheduler (TES) fits the bill. An end-to-end workload automation solution, TES offers built-in adapters to:

  • Hadoop, Sqoop, and Hive
  • Enterprise resource planning (ERP), database, data warehouse, data integration, and business intelligence applications

“TES is used to schedule processes that move data in and out of big data file systems, run data feeds from inside and outside the firewall, and execute big data workloads,” says Andrew Blaisdell, a Cisco product marketing manager specializing in workload automation for big data services.

  • Along with complex, time-based batch scheduling, Cisco TES automates workloads by initiating event-based actions.
  • An event such as the creation of a new customer record might trigger an action such as moving a set of records into a data warehouse.
  • Events can also include running self-service ad hoc reports for end users or watching FTP folders for changes.

“Cisco IT specialists have validated and run proof-of-concept testing on the TES API integration points for loading data and running Hive queries, Sqoop ETL processes, and MapReduce workloads,” Blaisdell says. “They love the deep integration and ease-of-use of TES, and see a faster time to market for their big data services.”

  • TES runs jobs and events from a single, centralized server and can manage many thousands of jobs per day.
  • All big data jobs are managed from a single instance, lowering the burden of having to mange a distributed environment.
  • The end-to-end coverage of TES also allows administrators, both inside and outside the data center, to connect and manage workloads from any data source in the enterprise.

 

More information

To learn more about end-to-end workload automation, download the white paper and solution briefs here.