Category: DATA PROCESSING
We will discuss here the solutions for Data Processing, like HDFS and Spark.
Hybrid cloud architecture for data lake applications
Big data technologies nowadays are very mature. Typically you use HDFS, or another distributed file systems, like S3, for storing data, Spark as a processor engine, and YARN as a resource manager. Next steps, wich you probably would like to achieve, are implement CI/CD (continuous integration and delivery) and move workload on demand in cloud.
Read more
Remote submit of spark jobs
Remote submit is a powerful feature of Apache Spark. Why it is needed? For example, you can experiment with different versions of Spark, independent of what you have in the cluster. Or if you have no direct access to cluster you can start your spark jobs remotely.
Read more
Tuning spark parameters
Tuning spark parameters is not a trivial task. In this short post I will explain how to tune some of the important parameters.
Read more
Book notes – Building Microservices: Designing Fine-Grained Systems
Book notes – Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems
Why it is a bad idea to stream data back from HDFS into Kafka
Microservices vs service oriented architecture (SOA) and how containers change the rules of the game
Microservices approach gains recently popularity. Some time ago service oriented architecture (SOA) approach was very popular. But what is the difference?
Schema evolution and backward and forward compatibility for data in data lakes
We have discussed before the format for clean and derived data in data lakes. One of the popular formats for this goal is an avro format. We will talk here why it is needed and how to achieve backward and forward compatibility by designing avro schemas.
Read more