{"componentChunkName":"component---src-templates-post-template-js","path":"/posts/column-based","result":{"data":{"markdownRemark":{"id":"38286cf3-da67-54a5-a985-02704faed06b","html":"

Preface

\n

This is the third article of a series to summarize the key concepts of Chapter 3. Storage and Retrieval in the Designing Data Intensive Application book. The series consists of 3 articles, including log-structure storage engine (SSTables and LSM Tree), page-oriented storage engine (B Tree) and column based storage engine.

\n

Landscape of database storages

\n
\n\t\n \n \n \n \n \n \"Landscape\n \n \n \n
\n

In short, database storages are divided into two categories, row-based and column based. In this article we will focus on the core concept of column-based storage engine, and compare the pros and cons between row-based storage engine.

\n

Column-based storage engine

\n

Before introducing column-based storage engine, we need to know 2 terms first:

\n

OLTP v.s. OLAP

\n
\n\t\n \n \n \n \n \n \"OLTP\n \n \n \n
\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
OLTP (OnLine Transactional Processing)OLAP (OnLine Analytical Processing)
Primarily used byEnd user/customer, via web applicationInternal analyst, for decision support performance reporting
Main read patternSmall number of records per query, fetched by keyAggregate over large number of records
Main write patternRandom-access, low-latency writes from user inputBulk import (ETL) or event stream
What data representsLatest state of data (current point in time)History of events that happened over time
Dataset sizeGigabytes to terabytesTerabytes to petabytes
\n

Since the analytical requirements in large companies arise, and this two different scenarios actually have quite different access pattern to database, new database was evolved to enhance its performance. And one of the most important one is column-based storage engine.

\n

Star Schema

\n

While business analysts or data analysts are analyzing business data, it’s very common that there is a fact table (e.g. orders, transactions, events), and several metadata tables that fact tables will reference to them (e.g. product, customer table). The shape of schema and their foreign keys will be like a “star”. So we call it start schema.

\n

The fact table is usually very very large (> trillion data for large company), this makes analytical operation difficult for traditional row-based database.

\n

Column-based storage engine

\n
\n\t\n \n \n \n \n \n \"column\n \n \n \n
\n

The core concept of column-based storage engine is quite intuitive. We store data by column instead of by row. Imaging that we have millions of customer data, if we store data by row, then while doing aggregation (e.g. summation, average), it will be very time consuming because we may need to load each rows (Unless we have index).

\n

On the other hand, if we store data by column, then calculating all customers’ average age will only require to load the file for age column. It will tremendously enhance the performance.

\n

The benefits

\n\n

The drawbacks

\n\n

Real world DB

\n\n

Comparison matrix of Row-based & Column based storage engine

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Pros & ConsReal world DB
Row-based- Better write performance
- Easier to handle transactional operation		Cassandra / LevelDB / HBase / Lucence MySQL / PostgreSQL / MongoDB
Column-based- Largely better aggregation performance of few columns but on large portion of data Worse
- write performance, usually we will write data in batch		Clickhoise / Vertica
\n

Summary

\n

If also read the previous 2 articles about storage and retrieval, then congratulation! Now we should be able to have some thoughts in our mind to determine what databases to use for our application.

\n

Let’s recall the landscape of database storage engines. Now, how will you determine the selection of databases in your application ?

\n
\n\t\n \n \n \n \n \n \"Landscape\n \n \n \n
\n

Mine is pretty simple, for analytical requirements like internal analysis or performance reporting, we should firstly consider column-based storage engine. Then, for write intensive application, we can consider using log-structured storage engine, and on the other hands we can consider page-oriented storage engine if number of reads are usually larger than writes.

\n

Yes, this is just the first step of how to design data intensive application. But everything comes from the first step. Let’s proceed to the future chapters of this book.

","fields":{"slug":"/posts/column-based","tagSlugs":["/tag/database-storage/","/tag/column-based-based-storage-engine/","/tag/row-based-storage-engine/","/tag/designing-data-intensive-application/"]},"frontmatter":{"date":"2021-12-19T23:12:04.772Z","description":"while the analytical requirement arise in large companies, the performance for aggregating large scale data gradually cannot be accepted by using row-based storage engine. Column-based storage engines were developed for optimizing analytical requirement. They have huge improvement while query large portion of data on few columns. In this article we will talk more about its core concept.","tags":["Database Storage","Column-based based storage engine","Row-based storage engine","Designing Data Intensive Application"],"title":"Storage and Retrieval (3) - Column-based storage engine","socialImage":{"publicURL":"/static/e1c2867b251a4f300016b135407f731f/social.jpg"}}}},"pageContext":{"slug":"/posts/column-based"}},"staticQueryHashes":["251939775","401334301","825871152"]}