Skip to content

An implementation of a real-world map-reduce workflow in each major framework.

License

Notifications You must be signed in to change notification settings

yhwen/hadoop-framework-examples

 
 

Repository files navigation

Realistic Hadoop Data Processing Examples

This code is to accompany my blog post on map reduce frameworks

The point of the code in this repository is to provide an implementation for a business question (listed below) in each of the major Map Reduce frameworks.

Each implementation will get it's own subdirectory with it's own build and running instructions. Each framework will also get an accompanying test, and an in-depth walkthrough about implementation details.

The following implementations are complete:

The problem

The Data

We have two datasets: customers, and transactions.

Customer Fields:

Transaction Fields:

  • transaction-id (1)
  • product-id (1)
  • user-id (1)
  • purchase-amount (19.99)
  • product-description (a rubber chicken)

These two datasets are stored in tab-delimited files somewhere on HDFS.

The Question

For each product, we want to know the number of locations in which that product was purchased.

That's it!

In the real world, we might have other questions, like the number of purchases per location for each product.

About

An implementation of a real-world map-reduce workflow in each major framework.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Java 65.8%
  • Scala 24.3%
  • Python 6.6%
  • Shell 2.3%
  • PigLatin 1.0%