Subscribe:
RSS feed
Articles on Big Data, HPC, and Highly Scalable Software Engineering
Browsing Archives of Author »Ilya Katsov«
Distributed Algorithms in NoSQL Databases
Scalability is one of the main drivers of the NoSQL movement. As such, it encompasses distributed system coordination, failover, resource management and many other capabilities. It sounds like a big umbrella, and it is. Although it can hardly be said that NoSQL movement brought fundamentally new techniques into distributed data processing, it triggered an avalanche […]
Speeding Up Hadoop Builds Using Distributed Unit Tests
August 14, 2012
We recently worked with one of the Hadoop vendors on the continuous integration system for Hadoop core and other Hadoop-related projects like Pig, Hive, HBase. One of the challenges we faced was very slow automatic tests — full unit/integration test suite takes more than 2 hours for Hadoop core and more than 9 hours for […]
Fast Intersection of Sorted Lists Using SSE Instructions
June 5, 2012
Intersection of sorted lists is a cornerstone operation in many applications including search engines and databases because indexes are often implemented using different types of sorted structures. At GridDynamics, we recently worked on a custom database for realtime web analytics where fast intersection of very large lists of IDs was a must for good performance. From a functional […]
Probabilistic Data Structures for Web Analytics and Data Mining
May 1, 2012
Statistical analysis and mining of huge multi-terabyte data sets is a common task nowadays, especially in the areas like web analytics and Internet advertising. Analysis of such large data sets often requires powerful distributed data stores like Hadoop and heavy data processing with techniques like MapReduce. This approach often leads to heavyweight high-latency analytical processes and […]
NoSQL Data Modeling Techniques
March 1, 2012
NoSQL databases are often compared by various non-functional criteria, such as scalability, performance, and consistency. This aspect of NoSQL is well-studied both in practice and theory because specific non-functional properties are often the main justification for NoSQL usage and fundamental results on distributed systems like the CAP theorem apply well to NoSQL systems. At the same time, NoSQL […]
Tricks with Direct Memory Access in Java
February 2, 2012
Java was initially designed as a safe, managed environment. Nevertheless, Java HotSpot VM contains a “backdoor” that provides a number of low-level operations to manipulate memory and threads directly. This backdoor – sun.misc.Unsafe – is widely used by JDK itself in the packages like java.nio or java.util.concurrent. It is hard to imagine a Java developer who uses this backdoor in […]
MapReduce Patterns, Algorithms, and Use Cases
February 1, 2012
In this article I digested a number of MapReduce patterns and algorithms to give a systematic view of the different techniques that can be found on the web or scientific articles. Several practical case studies are also provided. All descriptions and code snippets use the standard Hadoop’s MapReduce model with Mappers, Reduces, Combiners, Partitioners, and sorting. This […]
Implementation of MVCC Transactions for Key-Value Stores
January 7, 2012
ACID transactions are one of the most widely used software engineering techniques, a cornerstone of the relational databases, and an integral part of the enterprise middleware where transactions are often offered as the black-box primitives. Notwithstanding all these and many other cases, the old-fashion approach to transactions cannot be maintained in a variety of modern large […]
Performance of Priority Queue Sorting with Pagination
January 2, 2012
In web applications, it is a very common task to sort some set of items according to the user-selected criteria and return only the first or N-th page of the sorted result. The page size can be much less than the total number of items, hence it is typically not reasonable to sort the entire set and […]
-
Ilya Katsov
- Distributed Algorithms in NoSQL Databases
- Speeding Up Hadoop Builds Using Distributed Unit Tests
- Fast Intersection of Sorted Lists Using SSE Instructions
- Probabilistic Data Structures for Web Analytics and Data Mining
- Hierarchical Navigation and Faceted Search on Top of Oracle Coherence
- NoSQL Data Modeling Techniques
- Tricks with Direct Memory Access in Java
- MapReduce Patterns, Algorithms, and Use Cases
- Implementation of MVCC Transactions for Key-Value Stores
- Performance of Priority Queue Sorting with Pagination
September 18, 2012
17