In our previous article we have discussed the root structure for HDFS. In this article we will discuss next level of the file structure, which will help to improve the speed of reading data.
Category: DATA PROCESSING
We will discuss here solutions for data processing and distributed computing, like HDFS, spark, and kubernetes.
Raw, clean, and derived data in data lakes based on HDFS
You may think, that there is no need to structure data in HDFS. You can systemize it in the future. But I think this is a wrong way. We should always keep in mind: there is no free lunch. Therefore it is better to make desicions at the beginning.
Read more
Thoughts about schema-on-write and schema-on-read
There are two approcahes, which we can select for designing storage of the data. They are schema-on-read and schema-on-write.
How to integrate Apache Kylin OLAP In Excel (pivot) [XMLA Connect and Mondrian]
Apache Kylin is very powerfull OLAP engine. It supports ODBC driver to move the data in excel, however this driver is not user friendly. Users should wright sql queries for this.
Short note about HDFS or why you need distributed file system
Why do you need HDFS (Hadoop Distributed Files System)? If the amount of data is small and place on your computer is enough for this, then you do not need distributed file system. But if you like to process a large amount of data, which is not possible to save on one computer, then you need to think about distributed file system.
Easy Master Data Management with XMDM
You can download here the new software for master data management.
Word Tree in Java and JavaScript
There is a technique to visualize the text, which is called Word Tree.
Here is a simple implementation in Java and JavaScript using SWT (Java Library), D3 (JavaScript Library), and XulRunner.
The version can be used with 64bit Windows.
You can download the application and source code here:
AllText v. 15.06.21
License: GPL v3
Downloads for Pentaho/R Intgeration Plugin
NOTE: This plugin is an old and open source version and is not the same as R script executor, which is available from Pentaho.
Source Code can be found here: GitHub Repository
License: LGPL
If you have found any issues or you have the ideas how to extend the functionality, then you can open ticket here: Issues on GitHub
Example Transformation for Plugin:
pentahor_test.zip
Version 0.0.4
RScriptPlugin-0.0.4.zip
– NullPointerException is fixed
Read more
Two use cases for Pentaho/R Integration plugin
NOTE: This plugin is an old and open source version and is not the same as R script executor, which is available from Pentaho.
R Script Plugin supports two use cases:
- Execute R code for every row in the table
- Execute R code for whole table and aggregate the result
Pentaho/R Integration Plugin can be downloaded here: Downloads
Pentaho/R Integration Plugin
Pentaho R Integration Plugin is used to execute R code direct from Pentaho Data Integration (PDI).
R Script Plugin can be downloaded here: Downloads
NOTE: This plugin is an old and open source version and is not the same as R script executor, which is available from Pentaho.
Read more