Apache Oozie: The Workflow Scheduler for Hadoop

Mohammad Kamrul Islam, Aravind Srinivasan

"O'Reilly Media, Inc.", 2015年5月12日 - 272 頁

Get a solid grounding in Apache Oozie, the workflow scheduler system for managing Hadoop jobs. With this hands-on guide, two experienced Hadoop practitioners walk you through the intricacies of this powerful and flexible platform, with numerous examples and real-world use cases.

Once you set up your Oozie server, you’ll dive into techniques for writing and coordinating workflows, and learn how to write complex data pipelines. Advanced topics show you how to handle shared libraries in Oozie, as well as how to implement and manage Oozie’s security capabilities.

• Install and configure an Oozie server, and get an overview of basic concepts
• Journey through the world of writing and configuring workflows
• Learn how the Oozie coordinator schedules and executes workflows based on triggers
• Understand how Oozie manages data dependencies
• Use Oozie bundles to package several coordinator apps into a data pipeline
• Learn about security features and shared library management
• Implement custom extensions and write your own EL functions and actions
• Debug workflows and manage Oozie’s operational details

內容

• Introduction to Oozie: 1
• Oozie Concepts: 13
• Parameters Variables and Functions: 19
• Hadoop Cluster: 28
• 4: 39
• Synchronous Versus Asynchronous Actions: 73
• Parameterization: 86
• Oozie Coordinator: 99
• Data Trigger Coordinator: 117
• Oozie Bundles: 137
• Advanced Topics: 147
• Supporting New API in MapReduce Action: 165
• 23: 170
• Developer Topics: 177
• Oozie Operations: 203
• 39: 212
• 40: 243
• 24: 244

關於作者

Mohammad Kamrul Islam is currently working at Uber in data engineering team as a Staff Software Engineer. Previously, he worked at Linkedin for more than two years as Staff Software Engineer in the Hadoop development team. Before that, he worked at Yahoo for nearly five years as an Oozie architect/technical lead. His fingerprints can befound all over Oozie and is a respected voice in the Oozie community. He has been intimately involved with the Apache Hadoop ecosystem since 2009. Mohammad has a Ph.D. in Computer Science with a specialization in parallel job scheduling from Ohio State University. He received his MSCS degree from Wright State University, Ohio andBSCS from Bangladesh University of Engineering and Technology (BUET). He is a Project Management Committee (PMC) member of both Apache Oozie and Apache TEZ and frequently contributes to Apache YARN/MapReduce and Apache Hive. He was elected as the PMC chair and Vice-President of Oozie as part of the Apache Software Foundation from 2013 through 2015. Aravind Srinivasan has been involved with Hadoop in general and Oozie in particular since 2008. He is currently a Lead Application Architect at Altiscale, a Hadoop-as-a-service company, where he helps customers with Hadoop application design and architecture. His association with Big Data and Hadoop started during his time at Yahoo, where he spent almost six years working on various data pipelines for advertising systems. He has extensive experience building complicated, low latency data pipelines and also in porting legacy pipelines to Oozie. He drove a lot of Oozie’s requirements as a customer in its early days of adoption inside Yahoo and later spent some time as a Product Manager in Yahoo’s Hadoop team where he contributed further to Oozie’s roadmap. He also spent a year after Yahoo at Think Big Analytics, a Hadoop consulting firm, where he got to consult on some interesting and challenging Big Data integration projects at Facebook. He has a Masters in Computer Science from Arizona State and lives in Silicon Valley.

書目資訊

• 書名: Apache Oozie: The Workflow Scheduler for Hadoop
• 作者: Mohammad Kamrul Islam, Aravind Srinivasan
• 出版者: "O'Reilly Media, Inc.", 2015
• ISBN: 1449369774, 9781449369774
• 頁數: 272 頁