Follow the source: usage of CQRS pattern in data lake
There is a pattern in microservices architecture: Command and Query Responsibility Segregation (CQRS). This pattern helps to design multi-purpose data lake.
Read moreTag: data lake
Meta data service and schema registry in data lake
Maintaining data description is useful feature. There are some ideas, how to implement this.
Read moreORM (object-relational mapping) analog for data in data lake
We start saving data in HDFS using avro format. In previous post we have discussed about forward and backward compatibility of avro schemas. How to use this concept?
Read moreWhy contracts are important in data intensive applications with microservices
Main purpose of using microservices architecture is to increase velocity of development and reduce system complexity.
Read more
Hybrid cloud architecture for data lake applications
Big data technologies nowadays are very mature. Typically you use HDFS, or another distributed file systems, like S3, for storing data, Spark as a processor engine, and YARN as a resource manager. Next steps, wich you probably would like to achieve, are implement CI/CD (continuous integration and delivery) and move workload on demand in cloud.
Read more
Schema evolution and backward and forward compatibility for data in data lakes
We have discussed before the format for clean and derived data in data lakes. One of the popular formats for this goal is an avro format. We will talk here why it is needed and how to achieve backward and forward compatibility by designing avro schemas.
Read more
Raw, clean, and derived data in data lakes based on HDFS
You may think, that there is no need to structure data in HDFS. You can systemize it in the future. But I think this is a wrong way. We should always keep in mind: there is no free lunch. Therefore it is better to make desicions at the beginning.