Tag: design
Why it is a bad idea to stream data back from HDFS into Kafka
Microservices vs service oriented architecture (SOA) and how containers change the rules of the game
Microservices approach gains recently popularity. Some time ago service oriented architecture (SOA) approach was very popular. But what is the difference?
Schema evolution and backward and forward compatibility for data in data lakes
We have discussed before the format for clean and derived data in data lakes. One of the popular formats for this goal is an avro format. We will talk here why it is needed and how to achieve backward and forward compatibility by designing avro schemas.
Read more
HBase is next step in your big data technology stack
Raw, clean, and derived data in data lakes based on HDFS
You may think, that there is no need to structure data in HDFS. You can systemize it in the future. But I think this is a wrong way. We should always keep in mind: there is no free lunch. Therefore it is better to make desicions at the beginning.
Read more
Thoughts about schema-on-write and schema-on-read
There are two approcahes, which we can select for designing storage of the data. They are schema-on-read and schema-on-write.
Short note about HDFS or why you need distributed file system
Why do you need HDFS (Hadoop Distributed Files System)? If the amount of data is small and place on your computer is enough for this, then you do not need distributed file system. But if you like to process a large amount of data, which is not possible to save on one computer, then you need to think about distributed file system.