There is a pattern in microservices architecture: Command and Query Responsibility Segregation (CQRS). This pattern helps to design multi-purpose data lake.Read more
Maintaining data description is useful feature. There are some ideas, how to implement this.Read more
We start saving data in HDFS using avro format. In previous post we have discussed about forward and backward compatibility of avro schemas. How to use this concept?Read more
It looks like, that separation between two infrastructure layers is increasing.
Main purpose of using microservices architecture is to increase velocity of development and reduce system complexity.
Big data technologies nowadays are very mature. Typically you use HDFS, or another distributed file systems, like S3, for storing data, Spark as a processor engine, and YARN as a resource manager. Next steps, wich you probably would like to achieve, are implement CI/CD (continuous integration and delivery) and move workload on demand in cloud.