[Reading List] 데이터 엔지니어링 관련 논문 (Articles on tools for the Data Engineering)

도구

 

Albis

• Albis: High-Performance File Format for Big Data Systems

 

Apache Calcite

• Apache Calcite: A Foundational Framework for Optimized Query Processing Over Heterogeneous Data Sources

 

Apache Hadoop Distributed File System

• The Hadoop Distributed File System

 

Apache Hadoop Mapreduce

• MapReduce: Simplified Data Processing on Large Clusters
• Parallel MapReduce: Maximizing Cloud ResourceUtilization and Performance Improvement Using ParallelExecution Strategies 
• Parallel Data Processing with MapReduce: A Survey
• MRTuner: A Toolkit to Enable Holistic Optimization forMapReduce Jobs
• What are Distributed Execution Engines? Characteristics and types of Distributed Execution Engines

 

Apache Hadoop Yarn

• Apache Hadoop YARN: Yet Another Resource Negotiator
• Mitigating YARN Container Overhead with Input Splits 

 

Apache Hive

• Hive - A Warehousing Solution Over a Map-Reduce Framework   
• Improving the performance of Hadoop Hive by sharing scan and computation tasks

 

Apache Spark

• Resilient Distributed Datasets: A Fault-Tolerant Abstraction forIn-Memory Cluster Computing
• Matrix Computations and Optimization in Apache Spark
• Pipelined execution of stages in Apache Spark 
• Architectural Impact on Performance of In-memory Data Analytics: Apache Spark Case Study
• A Benchmarking Study to Evaluate Apache Spark on Large-Scale Supercomputers
• Performance Comparison of Spark Clusters ConfiguredConventionally and a Cloud Service
• Fast Data Processing with Spark

 

Apache Tez

• Apache Tez: A Unifying Framework for Modeling andBuilding Data Processing Applications
• Analyzing performance of Apache Tez and MapReduce with hadoop multinode cluster on Amazon cloud

Borg

• Large-scale cluster management at Google with Borg
• Borg: the Next Generation
• Borg, Omega, and Kubernetes

 

CIEL

• CIEL: a universal execution engine for distributed data-flow computing

 

 

Crail

• Crail: A High-Performance I/O Architecture for Distributed Data Processing

 

Delta Lake

• Delta Lake: High-Performance ACID Table Storage overCloud Object Stores 

 

Diff-Index

• Diff-Index: Differentiated Index in Distributed Log-Structured Data Stores

 

Dremel

• Dremel: A Decade of Interactive SQL Analysis at Web Scale

 

Druid

• Druid: A real-time analytics data store

 

Espresso

• On Brewing Fresh Espresso:LinkedIn’s Distributed Data Serving Platform

 

F1 

• F1: A Distributed SQL Database That Scales
• F1 Query: Declarative Querying at Scale

 

FusionInsight LibrA

•  FusionInsight LibrA: Huawei’s Enterprise Cloud DataAnalytics Platform 

 

Kafka

• Kafka: a Distributed Messaging System for Log Processing
• Building LinkedIn’s Real-time Activity Data Pipeline

 

Kudu

• Kudu: Storage for Fast Analytics on Fast Data

 

MemoDyn

• MemoDyn: Exploiting Weakly Consistent Data Structures for Dynamic Parallel Memoization

 

Mesa

• Mesa: Geo-Replicated, Near Real-Time, Scalable DataWarehousing

 

Millwheel

• MillWheel: Fault-Tolerant Stream Processing atInternet Scale

Monarch

• Monarch: Google's Planet-Scale In-Memory Time Series Database

 

Naiad

• Incremental, iterative data processing with timely dataflow

 

NAPA

• Napa: Powering Scalable Data Warehousing with Robust Query Performance at Google

 

PacificA

• PacificA: Replication in Log-Based Distributed Storage Systems 

 

Poster Paper

• Poster Paper: Automatic Reconfiguration of Distributed Storage

 

Pregel

• Pregel: a system for large-scale graph processing

 

Presto

• Presto: SQL on Everything

 

Ray

• Ray: A Distributed Framework for Emerging AI Applications  
• Ray: A Distributed Execution Engine for the Machine Learning Ecosystem

 

Scuba

• Scuba: Diving into Data at Facebook 

 

Spanner

• Spanner: Google's Globally-Distributed Database
• Spanner, TrueTime and the CAP Theorem

 

StreamBox

• Streambox: Modern Stream Processing on a Multicore Machine

 

SWIFT

• SWIFT: Using task-based parallelism, fully asynchronous communication, and graph partition-based domain decomposition for strong scaling on more than 100000 cores

 

Ubiq

• Ubiq: A Scalable and Fault-tolerant Log Processing Infrastructure

 

Wrangler

• Wrangler: Interactive Visual Specificationof Data Transformation Scripts

 

 

Zanzibar

• Zanzibar: Google’s Consistent, Global Authorization System

 

 

일반적 사항들

 

Data center

• High-Availability at Massive Scale: Building Google’s Data Infrastructure for Ads

 

Data Model

• The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-Scale, Unbounded, Out-of-Order Data Processing