Approach to define a suitable Big Data architecture
This is a draft cheat sheet. It is a work in progress and is not finished yet.
Patterns in Big Data Architecture
Patterns are everywhere and indicate best practices. Therefore patterns are not invented but found. The start of software design patterns began in the early nineties of the 20th century baseed on building architectures and were first published as the famous gang of four design patterns
. Ever since the idea of patterns grew and grew to a huge collection of thounsands of patterns in nearly every discipline and part - not only reduced to software development anymore. Just to mention it - patterns can be brought together in a compound to create new more high level patterns.
There are currently pattern collections (or pattern languages) on different modern fields of software development like patterns for cloud computing
and big data solutions
. You can design a compelete solution and therefore consider all lessons learned, pitfalls, and best practices from the start.
Integration of Big Data solutions
Big Data solutions are always integrated with existing enterprise solutions and solve the problem of processing a huge volume of data, in as short time as possible, and coping with different structures.
Big Data Pipeline compound pattern
Big Data solutions are therefore always a data pipeline consisting of several steps wheras each steps identifies its input, several operations and an output.
Existing Big Data Compound Patterns
Logical Big Data Architecture