Chendi Xue's blog

Using VIM as Cpp IDE

Mon, 11 Nov 2019 00:00:00 +0000

Using VIM as IDE has many benefits, the reason I use VIM is because it’s free and I can compile and test on the same machine I wrote. While it also has some drawbacks, I always leave lots of typo errors inside codes, forget to add headers and didn’t quote namespaces, etc. And another drawbacks is coding style, after I added one more parameter or change function name, I need to shifts rest of codes in same line to make it align.

To fix those hassle, I chose to use google-codefmt and YouCompleteMe to make VIM a little bit smart, for now I don’t have some features like tagtree, since I feel it is quite convenient for me to use “grep” to find the codes definition. But I do added auto fold in my vimrc, so I can fold function when I don’t want to see them.

Here is all the steps to make My VIM CPP IDE done.

vundle

Vundle is a project used to install plugins and manage plugins in VIM, if you don’t have vundle installed, please [click here.

google/vim-codefmt

vim-codefmt is a tool which can help you to do codefmt by vim, two ways can be used to trigger codefmt

use “:FormatCode”, vim will format your codes by detected code type.
add auto format inside vimrc, and every time when you do “:w” for saving, codes will be auto formatted.

Installation of vim-codefmt is listed as below.

Change your .vimrc and add below lines in the beginning.

set nocompatible
filetype off
set rtp+=~/.vim/bundle/Vundle.vim
call vundle#begin()
Plugin 'google/vim-maktaba'
Plugin 'google/vim-codefmt'
Plugin 'google/vim-glaive'
call vundle#end()
call glaive#Install()
" Optional: Enable codefmt's default mappings on the <Leader>= prefix.
Glaive codefmt plugin[mappings]

Trigger the installation

option 1. Launch vim and run :PluginInstall

option 2. To install from command line: vim +PluginInstall +qall

now you should installed codefmt, to make it autoformatting, add below lines in your vimrc.

augroup autoformat_settings
  autocmd FileType bzl AutoFormatBuffer buildifier
  autocmd FileType c,cpp,proto,javascript AutoFormatBuffer clang-format
  autocmd FileType dart AutoFormatBuffer dartfmt
  autocmd FileType go AutoFormatBuffer gofmt
  autocmd FileType gn AutoFormatBuffer gn
  autocmd FileType html,css,sass,scss,less,json AutoFormatBuffer js-beautify
  autocmd FileType java AutoFormatBuffer google-java-format
  autocmd FileType python AutoFormatBuffer yapf
  " Alternative: autocmd FileType python AutoFormatBuffer autopep8
  autocmd FileType vue AutoFormatBuffer prettier
augroup END

now open any unformatted codes of yours, and use “:w”, you should see codes are shifted.

YouCompleteMe

YouCompelteMe is a tool to use your Makefile or CMakefile to understand your codes and then provides auto filling advices and diagnose codes error while you writing codes.

Linux Must Install Full installation Guide

CheckList:

install python-devel and python3-devel package

install YouCompeleteMe through Vundle

vim ~/.vimrc
set nocompatible
filetype off
set rtp+=~/.vim/bundle/Vundle.vim
call vundle#begin()
Plugin 'Valloric/YouCompleteMe'
call vundle#end()
:wq
Launch vim and run :PluginInstall

Install Clangd compeleter to YouCompleteMe

cd ~/.vim/bundle/YouCompleteMe
python3 install.py --clangd-completer

Compile ycm_core

cd ~
mkdir ycm_build
cd ycm_build
cmake -G "Unix Makefiles" . ~/.vim/bundle/YouCompleteMe/third_party/ycmd/cpp
cmake --build . --target ycm_core --config Release

Enable C-family Semantic Completion I used Makefile for now, and use compiledb to generate compilation database

pip install compiledb
cd ${CPP project}
compiledb -n make
# now you should see a file called "compile_commands.json" in your root dir.

Now, YouCompleteMe should be working for your CPP codes, let’s have a try open one of your cpp code with vim

:YcmDiags

some infomation will be shown at bottom, if there is no error, it would appear to be “No warinings or errors detected.”

Other vimrc settings

filetype plugin indent on
" show existing tab with 4 spaces width
set tabstop=2
" when indenting with '>', use 4 spaces width
set shiftwidth=2
" On pressing tab, insert 4 spaces
set expandtab
set foldmethod=indent
set foldnestmax=10
set nofoldenable
set foldlevel=2
" set textwidth=90
set colorcolumn=90
set number
set foldmethod=syntax
hi Folded ctermbg=black
map <C-]> :YcmCompleter GoToImprecise<CR>
let g:ycm_enable_diagnostic_signs = 1
highlight YcmErrorLine ctermfg=White cterm=bold

How to build IKEv2 VPN server on Amazon AWS EC2

Mon, 28 Oct 2019 00:00:00 +0000

I recently setup a VPN server on AWS EC2(free tier), so I can log onto my gmail account in Mainland China. Here is a very clear and easy to follow step by step guide.

How to set up an ik2=ev2 vpn server with strongswan on ubuntu 18.04.02

Debug:

If you met some issue, please check ipsec log using below cmd:

systemctl status ipsec.service

Remember to also modify you security group on EC2, since it will only allow SSH protocal(tcp 22) as default

Apache Arrow enabling HDFS Parquet support

Tue, 20 Aug 2019 00:00:00 +0000

First Arrow building from source blog refer to here: link

install double-conversion

git clone https://github.com/google/double-conversion.git
cd double-conversion
mkdir build
cd build
cmake -DBUILD_SHARED_LIBS=ON ..
make
make install

apache arrow and gandiva

cd arrow/cpp/build
rm -rf *
cmake -DARROW_GANDIVA_JAVA=ON -DARROW_GANDIVA=ON -DARROW_PARQUET=ON -DARROW_HDFS=ON -DARROW_BOOST_USE_SHARED=ON ..
# I noticed that even when we saw succeed of above cmake procedure, there will be some error inside "CMakeFiles/CMakeError.log", those errors are not fatal. So if you failed in building, you should keep looking the issue on foreground terminal log instead of the "CMakeFiles/CMakeError.log"
make
make install

Prepare TPCDS data for spark

Fri, 12 Jul 2019 00:00:00 +0000

I am currently working on spark sql vectorization implementation, and I felt there are so many options for preparing TPCDS data on hdfs, and I chose below method becaused the simplicity and hackability.

Install DataBricks SparkSqlPerf kit

git clone https://github.com/databricks/spark-sql-perf
cd spark-sql-perf
sbt +package

Install TPCDS kit

sudo yum install gcc make flex bison byacc git
git clone https://github.com/databricks/tpcds-kit.git
cd tpcds-kit/tools
make OS=LINUX
cd ..

TPCDS Preparation script and submit

vim TPCDSPreparation.scala

import com.databricks.spark.sql.perf.tpcds.TPCDSTables
import org.apache.spark.sql._

// Set:
val rootDir: String = "hdfs://sr602:9000/tpcds_1T" // root directory of location to create data in.
val databaseName: String = "tpcds_1T" // name of database to create.
val scaleFactor: String = "1000" // scaleFactor defines the size of the dataset to generate (in GB).
val format: String = "parquet" // valid spark format like parquet "parquet".
val sqlContext = new SQLContext(sc)
// Run:
val tables = new TPCDSTables(sqlContext,
    dsdgenDir = "/mnt/nvme2/chendi/spark-sql-perf/tpcds-kit/tools", // location of dsdgen
    scaleFactor = scaleFactor,
    useDoubleForDecimal = false, // true to replace DecimalType with DoubleType
    useStringForDate = false) // true to replace DateType with StringType

tables.genData(
    location = rootDir,
    format = format,
    overwrite = true, // overwrite the data that is already there
    partitionTables = true, // create the partitioned fact tables
    clusterByPartitionColumns = true, // shuffle to get partitions coalesced into single files.
    filterOutNullPartitionValues = false, // true to filter out the partition with NULL key value
    tableFilter = "", // "" means generate all tables
    numPartitions = 400) // how many dsdgen partitions to run - number of input tasks.

// Create the specified database
sql(s"create database $databaseName")
// Create metastore tables in a specified database for your data.
// Once tables are created, the current database will be switched to the specified database.
tables.createExternalTables(rootDir, "parquet", databaseName, overwrite = true, discoverPartitions = true)

# configure spark-defaults.conf
spark.driver.extraClassPath /mnt/nvme2/chendi/spark-sql-perf/target/scala-2.11/spark-sql-perf_2.11-0.5.1-SNAPSHOT.jar
spark.executor.extraClassPath /mnt/nvme2/chendi/spark-sql-perf/target/scala-2.11/spark-sql-perf_2.11-0.5.1-SNAPSHOT.jar

# using spark-shell to run TPCDS preparation scala
# that is one in above scala, we can assume sc(sparkcontext) exists.
spark-shell --master yarn --deploy-mode client -i TPCDSPreparation.scala

# expects output as below
[root@sr602 spark-gpu]# spark-shell --master yarn --deploy-mode client -i TPCDSPreparation.scala
19/08/12 10:35:00 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
19/08/12 10:35:06 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
Spark context Web UI available at http://vsr602:4040
Spark context available as 'sc' (master = yarn, app id = application_1565311463008_0088).
Spark session available as 'spark'.
Loading TPCDSPreparation.scala...
import com.databricks.spark.sql.perf.tpcds.TPCDSTables
import org.apache.spark.sql._
rootDir: String = hdfs://sr602:9000/tpcds_1T
databaseName: String = tpcds_1T
scaleFactor: String = 1000
format: String = parquet
warning: there was one deprecation warning; re-run with -deprecation for details
sqlContext: org.apache.spark.sql.SQLContext = org.apache.spark.sql.SQLContext@541897c6
tables: com.databricks.spark.sql.perf.tpcds.TPCDSTables = com.databricks.spark.sql.perf.tpcds.TPCDSTables@1d1fd2aa
19/08/12 10:35:35 WARN Utils: Truncated the string representation of a plan since it was too large. This behavior can be adjusted by setting 'spark.debug.maxToStringFields' in SparkEnv.conf.
Pre-clustering with partitioning columns with query
SELECT
  cs_sold_date_sk,cs_sold_time_sk,cs_ship_date_sk,cs_bill_customer_sk,cs_bill_cdemo_sk,cs_bill_hdemo_sk,cs_bill_addr_sk,cs_ship_customer_sk,cs_ship_cdemo_sk,cs_ship_hdemo_sk,cs_ship_addr_sk,cs_call_center_sk,cs_catal
og_page_sk,cs_ship_mode_sk,cs_warehouse_sk,cs_item_sk,cs_promo_sk,cs_order_number,cs_quantity,cs_wholesale_cost,cs_list_price,cs_sales_price,cs_ext_discount_amt,cs_ext_sales_price,cs_ext_wholesale_cost,cs_ext_list_pr
ice,cs_ext_tax,cs_coupon_amt,cs_ext_ship_cost,cs_net_paid,cs_net_paid_inc_tax,cs_net_paid_inc_ship,cs_net_paid_inc_ship_tax,cs_net_profit
FROM
  catalog_sales_text

DISTRIBUTE BY
  cs_sold_date_sk
            .
Generating table catalog_sales in database to hdfs://sr602:9000/tpcds_1T/catalog_sales with save mode Overwrite.
[Stage 0:========>                                              (62 + 80) / 400]

Apache Arrow Gandiva on LLVM(Installation and evaluation)

Fri, 12 Jul 2019 00:00:00 +0000

It took me some time to install gandiva, paste here for future reference.

llvm-7.0: Arrow Gandiva depends on LLVM, and I noticed current version strictly depends on llvm7.0 if you installed any other version rather than 7.0, it will fail.

wget http://releases.llvm.org/7.0.1/llvm-7.0.1.src.tar.xz
tar xf llvm-7.0.1.src.tar.xz
cd llvm-7.0.1.src/
cd tools
wget http://releases.llvm.org/7.0.1/cfe-7.0.1.src.tar.xz
tar xf cfe-7.0.1.src.tar.xz
mv cfe-7.0.1.src clang
cd ..
mkdir build
cd build
cmake ..
cmake --build . -j
cmake --build . --target install
# check if clang has also been compiled, if no
cd tools/clang
mkdir build
cd build
cmake ..
make -j
make install

re2

make
make test
make install

cmake: Arrow will download package during compiling, in order to support SSL in cmake, build cmake is optional.

wget https://github.com/Kitware/CMake/releases/download/v3.15.0-rc4/cmake-3.15.0-rc4.tar.gz
tar xf cmake-3.15.0-rc4.tar.gz
cd cmake-3.15.0-rc4/
./bootstrap --system-curl --parallel=64 #parallel num depends on your server core number
make -j
make install
cmake --version
cmake version 3.15.0-rc4

Protobuf version should be above 2.6.0, I used 3.7.1 here, building from source

apache arrow and gandiva

cd arrow/cpp/build
cmake -DARROW_GANDIVA_JAVA=ON -DARROW_GANDIVA=ON ..
make -j
make install

# build java
cd ../java
# change arrow.cpp.build.dir to the relative path of cpp build dir, 
# noted: since gandiva is subdir under java, so the relative path should add one more ../
mvn clean install -P arrow-jni -am -DskipTests -Darrow.cpp.build.dir=../../cpp/build/release

run test

mvn test -pl gandiva -P arrow-jni

benchmark

Changed current gandiva MicroBenchmarkTest to a 10 fields * 200 Million records test

 @Test
  public void testAdd10() throws Exception {
    Field x = Field.nullable("x", int32);
    Field n2x = Field.nullable("n2x", int32);
    Field n3x = Field.nullable("n3x", int32);
    Field n4x = Field.nullable("n4x", int32);
    Field n5x = Field.nullable("n5x", int32);
    Field n6x = Field.nullable("n6x", int32);
    Field n7x = Field.nullable("n7x", int32);
    Field n8x = Field.nullable("n8x", int32);
    Field n9x = Field.nullable("n9x", int32);
    Field n10x = Field.nullable("n10x", int32);

    // x + n2x + n3x
    TreeNode add =
        TreeBuilder.makeFunction(
            "add", Lists.newArrayList(TreeBuilder.makeField(x), TreeBuilder.makeField(n2x)), int32);
    TreeNode add1 =
        TreeBuilder.makeFunction(
            "add", Lists.newArrayList(add, TreeBuilder.makeField(n3x)), int32);
    TreeNode add2 =
        TreeBuilder.makeFunction(
            "add", Lists.newArrayList(add1, TreeBuilder.makeField(n4x)), int32);
    TreeNode add3 =
        TreeBuilder.makeFunction(
            "add", Lists.newArrayList(add2, TreeBuilder.makeField(n5x)), int32);
    TreeNode add4 =
        TreeBuilder.makeFunction(
            "add", Lists.newArrayList(add3, TreeBuilder.makeField(n6x)), int32);
    TreeNode add5 =
        TreeBuilder.makeFunction(
            "add", Lists.newArrayList(add4, TreeBuilder.makeField(n7x)), int32);
    TreeNode add6 =
        TreeBuilder.makeFunction(
            "add", Lists.newArrayList(add5, TreeBuilder.makeField(n8x)), int32);
    TreeNode add7 =
        TreeBuilder.makeFunction(
            "add", Lists.newArrayList(add6, TreeBuilder.makeField(n9x)), int32);
    TreeNode add8 =
        TreeBuilder.makeFunction(
            "add", Lists.newArrayList(add7, TreeBuilder.makeField(n10x)), int32);
    ExpressionTree expr = TreeBuilder.makeExpression(add8, x);

    List<Field> cols = Lists.newArrayList(x, n2x, n3x, n4x, n5x, n6x, n7x, n8x, n9x, n10x);
    Schema schema = new Schema(cols);

    long timeTaken = timedProject(new Int32DataAndVectorGenerator(allocator),
        schema,
        Lists.newArrayList(expr),
        200 * MILLION, 16 * THOUSAND,
        4);
    System.out.println("Time taken for projecting 200m records of add10 is " + timeTaken + "ms");
  }

mvn test -pl gandiva -P arrow-jni -Dtest=MicroBenchmarkTest

—

Spark WholeStageCodeGen

Thu, 11 Jul 2019 00:00:00 +0000

The idea of WholeStageCodeGen is an optimization to spark, as we know, spark exec flow is based on an Iterator chains of sparkplans. Before WholeStageCodeGen, when there is two spark plan in the same stage, we should see the process as something like RDD.map{sparkplan1_exec_lambda}.map{sparkplan2_exec_lambda}

And when we have WholeStageCodeGen, two sparkPlan can be processed inside one map. Above process will be optimized as RDD.map{wholestagecodegen_exec_lambda}, this wholestagecodegen_exec_lambda is a generated glue code from sparkplan1 and sparkplan2.

So, to implement this idea, spark designed a new sparkplan – “WholeStageCodeGenExec”, then use this sparkplan to trigger a tranverse among real sparkplan to create the glue code.

Before wholestagecodegen, doExecute() of each sparlplans will be called to provide the lambda for iterator. And when wholestagecodegen is enabled, for those sparkplan who support wholestagecodegen, their doProduce and doConsume functions will be called to generate a single piece of lambda.

doProduce is the function to produce input data, so only sparkplan as a leaf execnode should override and implement its own doProduce, and other sparkplans who are not leaf actually call their child’s doProduce.

doConsume is the function to consume the input data, for example, filter sparkplan provides filter code in its doConsume function.

And by calling the leaf’s doProduce and all doConsumes function of its child, wholestagecodegen then generates the glued lambda code block.

Take below graphs for a visualize example.

Spark Sql DataFrame processing Deep Dive

Tue, 16 Apr 2019 00:00:00 +0000

Reference: Deep Dive into Spark Storage formats

How spark handles sql request

A very clear introduction of spark-sql implementation from DataBricks

From above article, we can see that a spark sql will go though Analysis, Optimizer, Physical Planning then using Code Generation to turn into RDD java codes. There is a greate amount of rules and ops inside above steps for different type of input data and different transition sql asks.

Below, I used a very simple example of plusing one to each line of a two column table to see what codes would be generated.

columnar data layout: Orc or Parquet
csv data
csv with Arrow

Columnar Data Layout

Pyspark codes

from pyspark.sql import SparkSession
from pyspark.sql.functions import udf
from pyspark.sql.types import IntegerType

def plus_one_func(v):
    return v + 1

spark = SparkSession.builder.appName("pyspark_plus_one_orc").getOrCreate()
df = spark.read.format("orc").load("/HiBench/DataFrame/Input")
df = df.withColumn('count', plus_one_func(df["count"]))
df.write.format("parquet").save("/HiBench/DataFrame/Output")

After Spark CodeGen gist url, the task will be generated as below.

public Object generate(Object[] references) {
  return new GeneratedIteratorForCodegenStage1(references);
}

final class GeneratedIteratorForCodegenStage1 extends org.apache.spark.sql.execution.BufferedRowIterator {
   // define variables ...
  public GeneratedIteratorForCodegenStage1(Object[] references){...}
  public void init(int index, scala.collection.Iterator[] inputs){
    result = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(2, 32);
  }

  protected void processNext() throws java.io.IOException {
    // CodeGenStage1 is a BufferedRowIterator
	// So when stage1 tasked being executed, task thread will call Iterator.processNext to process next Row.
	int batchIdx = 0;
	while ( columnarBatch != null ) {
	  // Notice: 
	  // columnarBatch points to the next value of the inputsIterator, which is a scala.collection.Iterator
	  // One Iterator Row in this case is one column Batch, because we used Orc. Case is same if using Parquet.
	  int numRows = columnarBatch.numRows(); // numRows should be end row number of this columnarBatch.
	  int localEnd = numRows - batchIdx;
	  for (int rowIdx = 0; rowIdx < localEnd; rowIdx ++) {
	     rowIdx += batchIdx;
		 org.apache.spark.sql.vectorized.ColumnVector columeVector_0 = columnarBatch.column(0);
		 org.apache.spark.sql.vectorized.ColumnVector columeVector_1 = columnarBatch.column(1);
		 UTF8String  first_column = columeVector_0.getUTF8String(rowIdx_0);
		 int second_column = columeVector_1.getInt(rowIdx_0);
		 project_value_1 = second_column + 1;
		 // PLUS ONE is Processed by looping in per value in column!!!
	  }
      // write to result
	  result.write(0, first_column);
	  result.write(1, project_value_1);
	  
	  batchIdx = numRows;
	  batchscan_nextBatch_0(); // get next item in Iterator as columeBatch
	}
  }
  
  private void batchscan_nextBatch_0() throws java.io.IOException {
    org.apache.spark.sql.vectorized.ColumnarBatch columnarBatch = scala.collection.Iterator.next();
  }
}

There is only one code gen stage, Although this is pyspark script, since plus_one is simple, it handled as expression inside JVM.

Findings:

When using orc or parquet as input data, org.apache.spark.sql.vectorized.ColumnarBatch and ColumnVector will be used.
For plus_one operation, data was processed batch by batch then row by row inside one batch (CPU optimizable?).

orc with arrow

ORC with Arrow Generated codes

When processing orc data with Arrow(pandas_udf), there will be two codegen stages. One is before ArrowEval, and one is after. So before ArrowEval stage codegen will treat data as columnarBatch, and each row inside columnVector batch will be wrote into sql.catalyst.expressions.codegen.UnsafeRowWriter one by one. After ArrowEval, input seems to be row based, (TODO: check inputadapter_input_0.next() for exact data layout)

CSV

CSV Generated codes

When using csv(row-based) data, input collection will be processed one row by one row(no batch).
plus one is processed inside JVM as expression, so only one codegen stage created here.

CSV with Arrow

CSV with Arrow Generated codes

Also saw two code gen stages as expected, and first codegen treats data as row based, as well as second code gen stage.

Performance with above four method

from the performance, we can tell both python udf eval and row based processing brought overhead, and row based process has higer overhead than python udf eval with Arrow.

Data return and after shuffling

Questions and Todos:

Seems spark supported vectorized well if the data can be input as columnar layout. 1) how about using pandas_udf(Arrow), will there is using vectorized.columnarBatch? 2) how csv(row based) data processing looks like?
Seems one call of processNext will process all columnarBatch of this map, verify?
what is the return(UnsafeRowWriter) layout?
Adding shuffle to parquet data to check a more complex case.

Spark and Hadoop build from Source

Tue, 16 Apr 2019 00:00:00 +0000

Hadoop building from source

git clone https://github.com/apache/hadoop.git
cd hadoop
git checkout rel/release-3.2.0
# only build binary for hadoop
mvn clean install -Pdist -DskipTests -Dtar
# build binary and native library such as libhdfs.so for hadoop
# mvn clean install -Pdist,native -DskipTests -Dtar

Using mvn install, compiled hadoop jar will be put into /root/.m2/repository/ for other projects depends on hadoop to use. Check below pic, now, we have hadoop-3.2.0 installed and can be used by continuelly spark compiling.

To run hadoop, binary path is /hadoop/hadoop-dist/target/hadoop-3.2.0/

export HADOOP_HOME=/mnt/nvme2/chendi/hadoop/hadoop-dist/target/hadoop-3.2.0/

Spark building from source

git clone https://github.com/apache/spark.git
cd spark
# check spark supported hadoop version
grep \<hadoop\.version\> -r pom.xml
    <hadoop.version>2.7.4</hadoop.version>
    <hadoop.version>3.2.0</hadoop.version>
# so we should build spark specifying hadoop version as 3.2
./build/mvn -Pyarn -Phadoop-3.2 -Dhadoop.version=3.2.0 -DskipTests clean package
# we can use package here since there is no following package depends on spark, other wise, also use install here.

Specify SPARK_HOME to spark path

Tips

use -o option when executing mvn to avoid online package update or installation.

./build/mvn -Pyarn -Phadoop-3.2 -Dhadoop.version=3.2.0 -DskipTests clean package -o

pre install requirements to mvn repo – /root/.m2/repository to avoid slow mvn downloading.

TensorFlowOnSpark: Install Tutorial Step by Step (spark on Yarn)

Tue, 16 Apr 2019 00:00:00 +0000

references:

Yahoo TensorFlowOnSpark Repo guide
Tensorflow Spark Connector

Create a directory for later on scripts and package

mkdir tensorflowonspark
cd tensorflowonspark

Install Tensorflow and Tensorflow on spark

I tried to use python2.7 running tensorflowonspark, and it failed in importing tensorflow package when spark-submit pyspark scipt. While if I changed to python3, it works. So, if you don’t have a strong reason of using python2.7, just use python3 here.

pip3 install tensorflow tensorflowonspark

Prepare spark and Hadoop

I already installed spark and hadoop(Yarn, HDFS), and if you need some guidance, please check Spark and Hadoop build from source

Install Tensorflow Spark Connector

If you want to use format “TFRecords” to input and output on HDFS, below preparation is necessary.

git clone https://github.com/tensorflow/ecosystem.git
cd ecosystem/hadoop
mvn clean install -DskipTests
cd ../spark/spark-tensorflow-connector
mvn clean install -DskipTests

Then you’ll have tensorflow-hadoop and tensorflow-spark-connector in mvn repo

ll /root/.m2/repository/org/tensorflow/
total 0
drwxr-xr-x 3 root root 20 Apr 17 09:07 parentpom
drwxr-xr-x 3 root root 20 Apr 17 09:07 proto
drwxr-xr-x 3 root root 52 Apr 17 09:19 spark-tensorflow-connector_2.11
drwxr-xr-x 3 root root 52 Apr 17 09:09 tensorflow-hadoop

Upload the jar to HDFS

hadoop fs -put target/tensorflow-hadoop-1.10.0.jar

Prepare TensorFlow on Spark zip, so training pyspark script could call.

git clone https://github.com/yahoo/TensorFlowOnSpark.git
cd TensorFlowOnSpark
zip -r tfspark.zip tensorflowonspark
cd ..

Prepare HDFS native lib (libhdfs.so)

You probably should generated it when building Hadoop from source code, and if you didn’t, check below

cd ${HADOOP_SOURCEC_CODE}/hadoop-hdfs-project
mvn install -Pdist,native -DskipTests -Dtar
find ./ -name libhdfs.so
./hadoop-hdfs-native-client/target/target/usr/local/lib/libhdfs.so 

Prepare MNIST example package

mkdir mnist
cd mnist
curl -O "http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz"
curl -O "http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz"
curl -O "http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz"
curl -O "http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz"
zip -r mnist.zip *
cd ..

Run MNIST preparation/tranining/inference on TensorFlowOnSpark

System Variables Configuration

export PYSPARK_PYTHON=/usr/bin/python3
export SPARK_YARN_USER_ENV="PYSPARK_PYTHON=/usr/bin/python3"
export QUEUE=default  # For CPU
#export QUEUE=gpu     # For GPU

find / -name libhdfs.so
/mnt/nvme2/chendi/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/target/target/usr/local/lib/libhdfs.so
find / -name libjvm.so
/usr/java/jdk1.8.0_201-amd64/jre/lib/amd64/server/libjvm.so

export LIB_HDFS=${libhdfs.so location directory}  #/mnt/nvme2/chendi/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/target/target/usr/local/lib/
export LIB_JVM=${libjvm.so location directory}   #/usr/java/jdk1.8.0_201-amd64/jre/lib/amd64/server/
#export LIB_CUDA=/usr/local/cuda-7.5/lib64   #libcuda.so path

Prepare data as TFRecords(CPU mode)

Notice here: When using file path as hdfs:/// the package should be pre-uploaded to hdfs, and hdfs:/// will be translate as hdfs://${ip}:${port}/

${SPARK_HOME}/bin/spark-submit \
--master yarn \
--deploy-mode cluster \
--queue default \
--num-executors 4 \
--executor-memory 4G \
--archives hdfs:///user/${USER}/Python.zip#Python,mnist/mnist.zip#mnist \
--jars hdfs:///user/${USER}/tensorflow-hadoop-1.0-SNAPSHOT.jar \
TensorFlowOnSpark/examples/mnist/mnist_data_setup.py \
--output mnist/tfr \
--format tfr

Prepare data as TFRecords(GPU mode)

${SPARK_HOME}/bin/spark-submit \
--master yarn \
--deploy-mode cluster \
--queue default \
--num-executors 4 \
--executor-memory 4G \
--archives hdfs:///user/${USER}/Python.zip#Python,mnist/mnist.zip#mnist \
--jars hdfs:///user/${USER}/tensorflow-hadoop-1.0-SNAPSHOT.jar \
--conf spark.executorEnv.LD_LIBRARY_PATH=$LIB_CUDA \
--driver-library-path=$LIB_CUDA \
TensorFlowOnSpark/examples/mnist/mnist_data_setup.py \
--output mnist/tfr \
--format tfr

Verification

Check on Spark UI

hdfs dfs -ls mnist
Found 2 items
drwxr-xr-x   - root supergroup          0 2019-04-17 14:01 mnist/tfr

Till now, we can think the TensorFlowOnSpark setup is basically done and verified. And then we can turn to run some real benchmark.

Optimize Spark (pyspark) with Apache Arrow

Tue, 16 Apr 2019 00:00:00 +0000

What is Apache Arrow High level Intro

Apache Arrow is a cross-language development platform for in-memory data. It specifies a standardized language-independent columnar memory format for flat and hierarchical data, organized for efficient analytic operations on modern hardware. It also provides computational libraries and zero-copy streaming messaging and interprocess communication. Languages currently supported include C, C++, C#, Go, Java, JavaScript, MATLAB, Python, R, Ruby, and Rust.

Apache Arrow Official Site

Why we need Apache Arrow:

Apache Arrow simplifies data convertion, from “N to N” to “N to 1”.
Apache Arrow provides a native memory pool for various projects to access, by which way, data is no longer need to be serialized then deserialized to copy from one kind of object to another kind of object, for example, from on-heap object to buffer or on-heap object to python object.
Apache Arrow provides a columnar layout allows applications to avoid unnecessary IO and accelerate analytical processing performance on modern CPUs and GPUs.

Apache Arrow Internal

Apache Arrow is implemented by c++ and based on memory. I think that is root reason of why it is capable of providing high proficiency to compare with other project of using jvm heap memory. Apache Arrow Memory Pool Implementation

The second main reason of the efficiency is because of the columar layout design, it is implemented by using one “null bitmap” + one “offset buffer” + one “value array” to describe all kinds of base one dimentional data structure, such as array.

Implementation of a List (one dimentional) in Apache Arrow.

The Null Bitmap: indicates which slot in offset buffer should be NULL instead of empty. Check the difference between slot 1 -> NULL and slot 3 -> [].

The Offset Buffer: indicates the start index and next_start index of each String mapping as a flat Buffer. see first two int32 in Offset Buffer, which is 0 and 3, mapping to the values array, it ranges “joe”.

The values Array: a flat buffer of data. Notice: instead of using “struct”, real data is putting together in Arrow, easy to copy with zero serialization>

Implementation of two dimentional data – Struct<List, Int32>

A two dimentional struct in Arrow is implemented by 1 * “parent one dimentional struct” + n * “child one dimentional struct”.

An example here is a Struct<List, Int32>, so there is two children in this DataType, a List and a Int32, and to implement a one-dimentional data was explained above.

And the parent only uses one Null Bitmap to indicates if there is Null inside this table.

Apache Arrow RecordBatch

RecordBatch is quite important data type in Apache Arrow, it is mostly used in Arrow RPC. And RecordBatch is also a two-dimentional data. An example is shown as below, I believe it is easy to understand the structure now.

Apache Arrow Library

Let’s then take a look of the rest of library Apache Arrow provides. Based on one-dimentional datatype and two-dimentional datatype, Arrow is capable of providing more complex data type for different use cases. Inside its python/java lib, it provides io interface such as file and stream.

Modules

Besides Apache Arrow core, it provides two important modules: Plasma(For IPC object storage based on shared memory) and Flight(RPC and Messaging framework), both is worth checking but I did’t have a chance to use any of them, skip here.

Use case and verification

Most spoken use case by using Apache Arrow is how it helped improving pyspark performance by change oroginal DataFrame to Pandas DataFrame or defining original python udf to pandas_udf. Either way is helped to indicate the memory layout and cpu to process data by columnar.

Check below pic, original data sharing between spark scala(JVM) and pyspark used pickle. It works while the efficiency is low. And may need double serialization(SourceData to python, then result back to JVM)

When changed to Arrow, data is stored in off-heap memory(No need to transfer between JVM and python, and data is using columnar structure, CPU may do some optimization process to columnar data.)

Only publicated data of testing how Apache Arrow helped pyspark was shared 2016 by DataBricks. Check its link here: Introduce vectorized udfs for pyspark.

So, I did a test by my own cluster to check if the conclusion is still the same. I also used plus_one function to verify the efficiency of different kind of python functions: compared python expression vs python udf vs pandas udf.

The test set is a table contains two columes: word and its count. The test is to plus one to the count, change from 1 to 2.

Data set is 155G, maps to 1520 partitions, each partiton is 100MB data. Only used 1 physical node to run spark with 20 executors * 4 cores. Benchmark Script Codes

Performance shows pandas_udf performance 2.62x better than python udf, aligns the conclusion from Databricks 2016 publication.

To be Expected

Databricks is now working on a Spark JIRA to Use Apache Arrow to optimize Data Exchange between Spark and DL/AI frameworks

Ideas includes things below:

support DL/AI data source load into spark as DataFrame more natively

Example code:

df = spark.read.format("images").load("s3://xxx/imagenet")
df = spark.read.format("tfrecords").load("wasb://xxx/training")

Speed up convertion from “spark dataframe” to “numpy arrays” for tensorflow by expose Arrow to spark as ArrowRDD

Summarize

It is promising of using Apache Arrow to do optimization, besides spark case, it is also well explored in project like Kuda, etc.

Difference between Spark Shuffle vs. Spill

Mon, 15 Apr 2019 00:00:00 +0000

What does spark shuffle do?

Shuffling is a term to describe the procedure between map task and reduce task. And when we say shuffling, it refers to data shuffling.

What data need to be shuffled?

Let’s take an example. Say states in US need to make a ranking of the GDP of each neighborhood. Imagine the final result shall be something like Manhattan, xxx billion; Beverly Hills, xxx billion, etc.

And since there are enormous amount of neighborhood inside US, we are using terasort algorithm to do the ranking.

Each map task input some data from HDFS, and check which city it belongs to. Say if the neighborhood located in NewYork, then put it into a NewYork bucket.
When all map tasks completed, which means all neighborhoods have been put into a corresponding City Bucket. All buckets are showed in left side, different color indicates different city.
These buckets are shuffle data!
Then, reduce tasks begin, each Reduce task is responsible for one city, it read city bucket data from where multiple map tasks wrote. while reading bucket data, it also start to sort those data at meantime.
Once all bucket data read(right side), we would have records of each City in which the GDP of each neighborhood is sorted.

so, in spark UI, when one job requires shuffling, it always being divicded into two stages. One map stage and one reduce stage. Shown as below.

How to estimate how much data will be shuffled?

Summarize here, shuffling is a precedure for spark executors either in same physical node or in different physical nodes to exchange intermedia data generated by map tasks and required by reduce tasks. Map tasks wrote data down, then reduce tasks retrieve data for later on processing. So the data size of shuffle data is related to what result expects.

Assume the result is a ranking, which means we have an unsorted records of neighborhood with its GDP, and output should be a sorted records of neighborhood with its GDP. Then shuffle data should be records with compression or serialization.

While if the result is a sum of total GDP of one city, and input is an unsorted records of neighborhood with its GDP, then shuffle data is a list of sum of each neighborhood’s GDP.

For spark UI, how much data is shuffled will be tracked. Written as shuffle write at map stage.

If you want to do a prediction, we can calculate this way, let’s say we wrote dataset as 256MB block in HDFS, and there is total 100G data. Then we will have 100GB/256MB = 400 maps. And each map reads 256MB data. These 256MB data will then be put into different city buckets with serialization. So we can see shuffle write data is also around 256MB but a little large than 256MB due to the overhead of serialization.

Then, when we do reduce, reduce tasks read its corresponding city records from all map tasks. So the total shuffle read data size should be the size of records of one city.

What does spark spilling do?

Spilling is another reason of spark writing and reading data from disk. And the reason it happens is that memory can’t be always enough.

When doing shuffle, we didn’t write each records to disk everytime, we will write resords to its corresponding city bucket in memory firstly and when memory hit some pre-defined throttle, this memory buffer then flushes into disk.
Besides doing shuffle, there is one operation called External Sorter inside spark, it does a TimSort(insertion sort + merge sort) to the city buckets, since insertion data requires big memory chunk, when memory is not sufficient, it spills data to disk and clean current memory for a new round of insertion sort. Then it does merge sort to merge spilled data and remaining in memory data to get a sorted resords result.

How to estimate how much data will be spilled?

It depends on how much memory JVM can use. Spark set a start point of 5M memorythrottle to try spill in-memory insertion sort data to disk. While when 5MB reaches, and spark noticed there is way more memory it can use, the memorythrottle goes up.

Spark Shuffle DataFlow Detail(codes go through)

After all these explaination, let’s check below dataflow diagram drawed by me, I believe it should be very easy to guess what these module works for.

No matter it is shuffle write or external spill, current spark will reply on DiskBlockObkectWriter to hold data in a kyro serialized buffer stream and flush to File when hitting throttle.

when doing data read from file, shuffle read treats differently to same node read and internode read. Same node read data will be fetched as a FileSegmentManagedBuffer and remote read will be fetched as a NettyManagedBuffer.

For sort spilled data read, spark will firstly return an iterator to the sorted RDD, and read operation is defined in the interator.hasNext() function, so data is read lazily.

Persisten Memory Development Kit(PMDK) Notes 2: Benchmark examples for multiple interfaces(c/c++/java)

Fri, 07 Dec 2018 00:00:00 +0000

This blog records all my steps of building PMDK, PMDK C++ API(libpmemobj++), PMDK JAVA API(llpl) on Centos 7.3. If you want to have a rampup of what PMDK is, and how it works, please refer to PMDK Notes 0: what it is and quick examples

C++ write benchmark codes using libpmemobj++

# compile with
g++ write_bench.cpp -o aep_bench_writer -std=c++11 -lpmemobj -lboost_program_options -lpthread

# run with
./aep_bench_writer -d /dev/dax0.0,/dev/dax1.0 -r 60 -t 1 -b 2
options:
  -h [ --help ]           Print help messages
  -d [ --device ] arg     pmem device path
  -r [ --runtime ] arg    runtime
  -t [ --thread_num ] arg parallel threads number
  -b [ --block_size ] arg block size for each request(MB/s)

libpmemobj implementation

class Writer {
public:
    Writer(Monitor* mon, string dev, int bs, int thread_num):
        mon(mon),
        inflight(0),
        block_size(bs * 1024 * 1024),
        thread_num(thread_num) {
        /* Open and create in persistent memory device */
        printf("benchmark on %s, block size: %dMB, thread number: %d\n", dev.c_str(), bs, thread_num);

        const char *pool_layout_name = "libpmemobj_persistent_heap";
        pmpool = pmemobj_open(dev.c_str(), pool_layout_name);
        if (pmpool == NULL) {
            pmpool = pmemobj_create(dev.c_str(), pool_layout_name, 0, S_IRUSR | S_IWUSR);
        }
        if (pmpool == NULL) {
            printf("Failed to open pool %s\n", pmemobj_errormsg());
            fflush(stdout);
            exit(-1);
        }

        for (int i = 0; i < thread_num; i++) {
            thread_pool.push_back(thread(&Writer::run, this));
        }
    }

    ~Writer() {
        pmemobj_close(pmpool);
    }

    void stop() {
        for (auto it = thread_pool.begin(); it != thread_pool.end(); it++) {
            it->join();
        }
    }

private:
    vector<thread> thread_pool;
    PMEMobjpool *pmpool;
    uint64_t block_size;
    Monitor* mon;
    int thread_num;
    int inflight;
    std::mutex mtx;
    std::condition_variable cv;

    void run() {
        std::unique_lock<std::mutex> lck(mtx);
        while (mon->is_alive()) {
            exec();
            inflight++;
            while (inflight > thread_num) cv.wait(lck);
        }
    }

    void exec() {
        TX_BEGIN(pmpool) {
            char* address = (char*)pmemobj_direct(pmemobj_tx_zalloc(block_size, 0));
            pmemobj_tx_add_range_direct((const void *)address, block_size);
            memset(address, 'a', block_size);

            mon->incCommittedJobs();
            inflight--;
            cv.notify_all();
            
        } TX_ONABORT {
            exit(1);
        } TX_END
    }

};

libpmemobj++ implementation

class Writer {
public:
    Writer(Monitor* mon, string dev, int bs, int thread_num):
        mon(mon),
        inflight(0),
        block_size(bs * 1024 * 1024),
        thread_num(thread_num) {
        /* Open and create in persistent memory device */
        printf("benchmark on %s, block size: %dMB, thread number: %d\n", dev.c_str(), bs, thread_num);
        //pmpool = pool<MEMBLOCK>::open(dev.c_str(), "");
        try {
            pmpool = pool<MEMBLOCK>::open(dev.c_str(), "");
        } catch (...) {
            pmpool = pool<MEMBLOCK>::create(dev.c_str(), "", 0, S_IRWXU);
        }
        block_ptr = pmpool.root();

        for (int i = 0; i < thread_num; i++) {
            thread_pool.push_back(thread(&Writer::run, this));
        }
    }

    ~Writer() {
        pmpool.close();
    }

    void stop() {
        for (auto it = thread_pool.begin(); it != thread_pool.end(); it++) {
            it->join();
        }
    }

private:
    vector<thread> thread_pool;
    pool<MEMBLOCK> pmpool;
    persistent_ptr<MEMBLOCK> block_ptr;
    uint64_t block_size;
    Monitor* mon;
    int thread_num;
    int inflight;
    std::mutex mtx;
    std::condition_variable cv;
    

    void run() {
        std::unique_lock<std::mutex> lck(mtx);
        while (mon->is_alive()) {
            exec();
            inflight++;
            while (inflight > thread_num) cv.wait(lck);
        }
    }

    void exec() {
	    transaction::run(pmpool, [&] {
            /* Data structure looks like below
            MEMBLOCK0(root) -> MEMBLOCK1 -> MEMBLOCK2 -> ...
                ||                 ||           ||
                \/                 \/           \/
              char[]              char[]       char[]
            */
            auto next_block_ptr = make_persistent<MEMBLOCK>();
            auto data_ptr = make_persistent<char[]>(block_size);
            block_ptr->value_ptr = data_ptr;
            block_ptr->size = block_size;
            block_ptr->next = next_block_ptr;
            block_ptr = next_block_ptr;

            /* To write something, just like doing to a memory pointer */
            char* data_in_mem_ptr = data_ptr.get();
            memset(data_in_mem_ptr, 'a', block_size);

            mon->incCommittedJobs();
            inflight--;
            cv.notify_all();
        });

    }
};

JAVA write benchmark codes using [llpl](https://github.com/xuechendi/llpl

# compile with
javac -cp {$llpl_path}/llpl/target/classes:/usr/share/java/apache-commons-cli.jar examples/Writer.java

# create a shortcut script
vim llpl_write_bench
#!/usr/bin/sh
java -ea -cp {$llpl_path}/llpl/target/classes:lib:target/test_classes:./:/usr/share/java/apache-commons-cli.jar -Djava.library.path={$llpl_path}/llpl/target/cppbuild {$llpl_path}/llpl/examples/Writer $@

./llpl_write_bench -bs 2 -d /dev/dax0.0,/dev/dax1.0 -s 80 -t 1

usage: utility-name
 -bs,--block_size <arg>   block size for each request
 -d,--device <arg>        pmem device path
 -s,--size <arg>          input total data size(GB)
 -t,--thread_num <arg>    parallel threads number

llpl implementation

    synchronized MemoryBlock<Transactional> allocateMemory(long length) {
        return this.h.allocateMemoryBlock(Transactional.class, Integer.BYTES + length);
    }

    private void write() {
	int length = this.bytes.length;
	MemoryBlock<Transactional> mr = allocateMemory(length);
        mr.copyFromArray(this.bytes, 0, 0, length);
        this.monitor.incCommittedJobs();
    }

    public void run(String dev, int bs, long size, int thread_num) {
	    this.device = dev;
	    this.size = size;
	    this.thread_num = thread_num;
        System.out.println("Thread Num: " + this.thread_num + ", Data size: " + this.size + "MB, Device: " + dev);
        this.h = Heap.getHeap(dev, 1024*1024*1024L);
	    // multi write to aep for testing

        /*Console c = System.console();
        c.readLine("press Enter to start");*/
        
        bytes = new byte[bs * 1024 * 1024];
        Arrays.fill(this.bytes, (byte)'a');
       
        long total_jobs = this.size / bs;

	    long remained_jobs = total_jobs;


        System.out.println("Start to run, total jobs: " + total_jobs);
	    this.executor = Executors.newFixedThreadPool(this.thread_num);
	    while (remained_jobs > 0) {
          this.executor.submit(this::write);
	      remained_jobs -= 1;
	    }
    }

    public void wait_to_stop() {
	try {
	    this.executor.awaitTermination(60, TimeUnit.SECONDS);
	} catch (InterruptedException ie) {
        this.executor.shutdown();
	}
        System.out.println("Completed!");
    }

    public void stop() {
        this.executor.shutdown();
    }

completed version

#include <iostream>
#include <string>
#include <unistd.h>
#include <stdio.h>
#include <libpmemobj++/make_persistent.hpp>
#include <libpmemobj++/make_persistent_array.hpp>
#include <libpmemobj++/persistent_ptr.hpp>
#include <libpmemobj++/pool.hpp>
#include <libpmemobj++/transaction.hpp>
#include <boost/program_options/options_description.hpp>
#include <boost/program_options/parsers.hpp>
#include <boost/program_options/variables_map.hpp>
#include <boost/program_options/cmdline.hpp>
#include <mutex>
#include <condition_variable>
#include <thread>


using pmem::obj::p;
using pmem::obj::persistent_ptr;
using pmem::obj::make_persistent;
using pmem::obj::transaction;
using pmem::obj::pool;
using namespace std;

struct MEMBLOCK{
    persistent_ptr<MEMBLOCK> next;
    persistent_ptr<char[]> value_ptr;
    p<size_t> size;
};

class ArgParser {
public:
    ArgParser(int ac, char* av[]) {
        boost::program_options::options_description options("options");
        options.add_options()
            ("help,h", "Print help messages")
            ("device,d", boost::program_options::value<string>(), "pmem device path")
            ("runtime,r", boost::program_options::value<string>(), "runtime")
            ("thread_num,t", boost::program_options::value<string>(), "parallel threads number")
            ("block_size,bs", boost::program_options::value<string>(), "block size for each request");
	    boost::program_options::store(boost::program_options::parse_command_line(ac, av, options), vm);
        if (vm.count("help")){ 
            std::cout << "Basic Command Line Parameter App" << std::endl 
                  << options << std::endl; 
            exit(1);
        } 
	    boost::program_options::notify(vm);
    }

    string get(string key) {
        if (vm.count(key)) {
            return vm[key].as<string>();
        }
        return ""; 
    }
private:
    boost::program_options::variables_map vm;

};

class Monitor {
public:
    Monitor(int runtime, int bs):
        committed_jobs(0),
        block_size(bs),
        alive(true) {
        run_thread = new thread(&Monitor::run, this);
    }

    ~Monitor() {
        delete run_thread;
    }

    void incCommittedJobs() {
        std::lock_guard<std::mutex> lock(committed_jobs_mutex);
        committed_jobs += 1;
    }

    bool is_alive() {
        return alive;
    }

    void stop() {
        alive = false;
        run_thread->join();
        elapse_sec -= 1;
        printf("======= Summarize =======\nAvg Bandwidth: %d (MB/s)\nAvg IOPS: %d\nTotal write data:%dMB\n", (committed_jobs * block_size / elapse_sec), (committed_jobs / elapse_sec), (committed_jobs * block_size));
    }

private:
    thread *run_thread;
    uint64_t committed_jobs;
    int block_size;
    std::mutex committed_jobs_mutex;
    bool alive;
	int elapse_sec = 0;

    void run() {
        long last_committed_jobs = 0;
        while (alive) {
            printf("Second %d (MB/s): %d\n", elapse_sec, (committed_jobs - last_committed_jobs) * block_size );
	        last_committed_jobs = committed_jobs;
	        elapse_sec += 1;
	        sleep(1);
	    }
    }

};

class Writer {
public:
    Writer(Monitor* mon, string dev, int bs, int thread_num):
        mon(mon),
        inflight(0),
        block_size(bs * 1024 * 1024),
        thread_num(thread_num) {
        /* Open and create in persistent memory device */
        printf("benchmark on %s, block size: %dMB, thread number: %d\n", dev.c_str(), bs, thread_num);
        //pmpool = pool<MEMBLOCK>::open(dev.c_str(), "");
        try {
            pmpool = pool<MEMBLOCK>::open(dev.c_str(), "");
        } catch (...) {
            pmpool = pool<MEMBLOCK>::create(dev.c_str(), "", 0, S_IRWXU);
        }
        block_ptr = pmpool.root();

        for (int i = 0; i < thread_num; i++) {
            thread_pool.push_back(thread(&Writer::run, this));
        }
    }

    ~Writer() {
        pmpool.close();
    }

    void stop() {
        for (auto it = thread_pool.begin(); it != thread_pool.end(); it++) {
            it->join();
        }
    }

private:
    vector<thread> thread_pool;
    pool<MEMBLOCK> pmpool;
    persistent_ptr<MEMBLOCK> block_ptr;
    uint64_t block_size;
    Monitor* mon;
    int thread_num;
    int inflight;
    std::mutex mtx;
    std::condition_variable cv;
    

    void run() {
        std::unique_lock<std::mutex> lck(mtx);
        while (mon->is_alive()) {
            exec();
            inflight++;
            while (inflight > thread_num) cv.wait(lck);
        }
    }

    void exec() {
	    transaction::run(pmpool, [&] {
            /* Data structure looks like below
            MEMBLOCK0(root) -> MEMBLOCK1 -> MEMBLOCK2 -> ...
                ||                 ||           ||
                \/                 \/           \/
              char[]              char[]       char[]
            */
            auto next_block_ptr = make_persistent<MEMBLOCK>();
            auto data_ptr = make_persistent<char[]>(block_size);
            block_ptr->value_ptr = data_ptr;
            block_ptr->size = block_size;
            block_ptr->next = next_block_ptr;
            block_ptr = next_block_ptr;

            /* To write something, just like doing to a memory pointer */
            char* data_in_mem_ptr = data_ptr.get();
            memset(data_in_mem_ptr, 'a', block_size);

            mon->incCommittedJobs();
            inflight--;
            cv.notify_all();
        });

    }

};

std::vector<std::string> split_string_to_vector(const string& s, char delimiter ) {
   std::vector<std::string> tokens;
   std::string token;
   std::istringstream tokenStream(s);
   while (std::getline(tokenStream, token, delimiter))
   {
      tokens.push_back(token);
   }
   return tokens;
}

int main(int argc, char* argv[]) {
    ArgParser arg_parser(argc, argv);
    int bs = stoi(arg_parser.get("block_size"));
    int thread_num = stoi(arg_parser.get("thread_num"));
    int runtime = stoi(arg_parser.get("runtime"));
    vector<string> device_list = split_string_to_vector(arg_parser.get("device"), ',');

    Monitor monitor(runtime, bs);
    vector<Writer*> writer_list;
    for (int i = 0; i < device_list.size(); i++) {
        writer_list.push_back(new Writer(&monitor, device_list[i], bs, thread_num));
    }
    sleep(runtime);
    monitor.stop();
    for (int i = 0; i < device_list.size(); i++) {
        writer_list[i]->stop();
        delete writer_list[i];
    }
    exit(0);
}

JAVA write benchmark codes using [llpl](https://github.com/xuechendi/llpl

# compile with
javac -cp {$llpl_path}/llpl/target/classes:/usr/share/java/apache-commons-cli.jar examples/Writer.java

# create a shortcut script
vim llpl_write_bench
#!/usr/bin/sh
java -ea -cp {$llpl_path}/llpl/target/classes:lib:target/test_classes:./:/usr/share/java/apache-commons-cli.jar -Djava.library.path={$llpl_path}/llpl/target/cppbuild {$llpl_path}/llpl/examples/Writer $@

./llpl_write_bench -bs 2 -d /dev/dax0.0,/dev/dax1.0 -s 80 -t 1

usage: utility-name
 -bs,--block_size <arg>   block size for each request
 -d,--device <arg>        pmem device path
 -s,--size <arg>          input total data size(GB)
 -t,--thread_num <arg>    parallel threads number

/* 
 * Copyright (C) 2018 Intel Corporation
 *
 * SPDX-License-Identifier: BSD-3-Clause
 * 
 */

package examples;

import lib.llpl.*;
import java.util.Arrays;
import java.util.concurrent.*;
import org.apache.commons.cli.*;


class ArgParser {

    CommandLine cmd;
    Options options = new Options();
    CommandLineParser parser = new DefaultParser();
    HelpFormatter formatter = new HelpFormatter();

    ArgParser (String[] args) {

        Option device = new Option("d", "device", true, "pmem device path");
        device.setRequired(true);
        options.addOption(device);

        Option size = new Option("s", "size", true, "input total data size(GB)");
        size.setRequired(true);
        options.addOption(size);

        Option thread_num = new Option("t", "thread_num", true, "parallel threads number");
        thread_num.setRequired(true);
        options.addOption(thread_num);

        Option block_size = new Option("bs", "block_size", true, "block size for each request");
        block_size.setRequired(true);
        options.addOption(block_size);

        try {
            this.cmd = parser.parse(options, args);
        } catch (ParseException e) {
            System.out.println(e.getMessage());
            formatter.printHelp("utility-name", options);
            System.exit(1);
        }

    }

    public String get(String key) {
	String ret = "";
    ret = this.cmd.getOptionValue(key);
	return ret;
    }
}

class Monitor {
    long committedJobs = 0;
    boolean alive = true;
    int bs;
    ExecutorService monitor_thread;
	Monitor (int bs) {
        this.bs = bs;
	    this.monitor_thread = Executors.newFixedThreadPool(1);
        this.monitor_thread.submit(this::run);
	}

	void run () {
        long last_committed_jobs = 0;
	    int elapse_sec = 0;
        while(alive) {
            System.out.println("Second " + elapse_sec + "(MB/s): " + (this.committedJobs - last_committed_jobs) * this.bs );
	        last_committed_jobs = this.committedJobs;
	        elapse_sec += 1;
	        try {
	            Thread.sleep(1000);
            } catch (InterruptedException e) {
                System.exit(1);
	        }
	    }
    }

    synchronized void incCommittedJobs() {
        this.committedJobs += 1;
    }

	void stop() {
	    this.alive = false;
	    this.monitor_thread.shutdown();
	}
}

public class Writer {
    Heap h = null;
    String device;
    long size;
    int thread_num;
    byte[] bytes;
    ExecutorService executor;
    Monitor monitor;

    Writer (Monitor monitor) {
        this.monitor = monitor;
    }


    synchronized MemoryBlock<Transactional> allocateMemory(long length) {
    //MemoryBlock<Transactional> allocateMemory(long length) {
        return this.h.allocateMemoryBlock(Transactional.class, Integer.BYTES + length);
    }

    private void write() {
	int length = this.bytes.length;
	MemoryBlock<Transactional> mr = allocateMemory(length);
        mr.copyFromArray(this.bytes, 0, 0, length);
        this.monitor.incCommittedJobs();
    }


    public void run(String dev, int bs, long size, int thread_num) {
	this.device = dev;
	this.size = size;
	this.thread_num = thread_num;
        System.out.println("Thread Num: " + this.thread_num + ", Data size: " + this.size + "MB, Device: " + dev);
        this.h = Heap.getHeap(dev, 1024*1024*1024L);
	    // multi write to aep for testing

        /*Console c = System.console();
        c.readLine("press Enter to start");*/
        
        bytes = new byte[bs * 1024 * 1024];
        Arrays.fill(this.bytes, (byte)'a');
       
        long total_jobs = this.size / bs;

	    long remained_jobs = total_jobs;


        System.out.println("Start to run, total jobs: " + total_jobs);
	    this.executor = Executors.newFixedThreadPool(this.thread_num);
	    while (remained_jobs > 0) {
          this.executor.submit(this::write);
	      remained_jobs -= 1;
	    }
    }

    public void wait_to_stop() {
	try {
	    this.executor.awaitTermination(60, TimeUnit.SECONDS);
	} catch (InterruptedException ie) {
        this.executor.shutdown();
	}
        System.out.println("Completed!");
    }

    public void stop() {
        this.executor.shutdown();
    }


    public static void main(String[] args) {
        //String[] device_list = {"/dev/dax0.0", "/dev/dax0.1", "/dev/dax0.2", "/dev/dax0.3", "/dev/dax0.5","/dev/dax1.0", "/dev/dax1.1", "/dev/dax1.2", "/dev/dax1.3", "/dev/dax1.5"};
        ArgParser arg_parser = new ArgParser(args);
        String[] device_list = arg_parser.get("device").trim().split("\\s*,\\s*", -1);
        long size = Integer.parseInt(arg_parser.get("size")) * 1024;
        int thread_num = Integer.parseInt(arg_parser.get("thread_num"));
        int bs = Integer.parseInt(arg_parser.get("block_size"));
        
        Writer[] writer = new Writer[device_list.length];
        Monitor monitor = new Monitor(bs);
        for (int i = 0; i < device_list.length; i++) { 
            writer[i] = new Writer(monitor);
            writer[i].run(device_list[i], bs, size, thread_num);
        }
        writer[0].wait_to_stop();
        for (int i = 1; i < device_list.length; i++) { 
            writer[i].stop();
        }
        monitor.stop();
        System.exit(0);
    }
}

S3A Committer review: how to enable, how to verify and performance

Fri, 19 Oct 2018 00:00:00 +0000

Work in progress.

Running Spark on kubernetes Step by steps

Thu, 18 Oct 2018 00:00:00 +0000

DEPLOYMENT

This blog is practised on CentOS 7, ubuntu reference

Pre-Setup

Upgrade kernel to latest version.

$ yum upgrade -y
$ reboot

setenforce 0
sed -i --follow-symlinks 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/sysconfig/selinux
modprobe br_netfilter
echo '1' > /proc/sys/net/bridge/bridge-nf-call-iptables

Docker and Kubeadm installation

vim /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
        https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg

yum update
yum install kubeadm docker –y

systemctl restart docker && systemctl enable docker
systemctl restart kubelet && systemctl enable kubelet

[optional] some configuration to set proxy and no proxy

sudo mkdir -p /etc/systemd/system/docker.service.d
vim /etc/systemd/system/docker.service.d/http-proxy.conf
[Service]
Environment="HTTP_PROXY=${proxy}" "NO_PROXY=${no_proxy}"

vim /etc/systemd/system/docker.service.d/https-proxy.conf
[Service]
Environment="HTTPS_PROXY=${proxy}" "NO_PROXY=${no_proxy}"

systemctl daemon-reload
systemctl restart docker

Kubernetes nodes create and join

swapoff –a

kubeadm init --pod-network-cidr=10.244.0.0/16 --feature-gates=CoreDNS=false --apiserver-advertise-address=10.1.0.35 --kubernetes-version=v1.11.0
# if kubeadm init failed, use “kubeadm reset && systemctl restart kubelet” to rerun

# if init completed, set permission and check status 
mkdir ~/.kube/
chmod +755 ~/.kube
cp /etc/kubernetes/admin.conf ~/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config

Set up pod flannel networks

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.9.1/Documentation/kube-flannel.yml

$ kubectl get nodes
NAME        STATUS     ROLES     AGE       VERSION
bigdata09   Ready   master    6m        v1.10.3

for kubernetes slave nodes, use below command to join

kubeadm join 192.168.5.9:6443 --token ${provide_by_} --discovery-token-ca-cert-hash sha256:5a549c8e6c1f0e76e96c86b516e830be454a024360c05c08e196ba9bc971284d

spark docker img preparation

In order to run spark benchmark tool HiBench, I also added HiBench jar and configuration inside Spark Docker Img

vim /hadoop/spark/kubernetes/dockerfiles/spark/Dockerfile
# add below lines in Dockerfile
# to copy spark configuration with hibench configuration and jars to the img
COPY HiBench/sparkbench /opt/HiBench/sparkbench
COPY sparkbench.conf /opt/HiBench/sparkbench.conf
# set the configuration path to docker system variable.
RUN echo SPARKBENCH_PROPERTIES_FILES=/opt/HiBench/sparkbench.conf >> /root/.bashrc
ENV SPARKBENCH_PROPERTIES_FILES /opt/HiBench/sparkbench.conf

#Notes: sparkbench.conf is a HiBench generated conf from HiBench.conf
#Sparkbench folder is the one under HiBench contains all SparkBench jars.

$ ./bin/docker-image-tool.sh -r docker.io/xuechendi -t v2.3.0-with-hibench build
$ ./bin/docker-image-tool.sh -r docker.io/xuechendi -t v2.3.0-with-hibench push

Run spark benchmark terasort in Kubenetes pods.

spark-submit --master k8s://https://10.1.0.35:6443 --deploy-mode cluster --properties-file /HiBench/report/terasort/spark/conf/sparkbench/spark.conf --class com.intel.hibench.sparkbench.micro.ScalaTeraSort --spark.executor.instances 25 --executor-memory 17g --conf spark.kubernetes.driver.pod.name=spark-terasort --conf spark.kubernetes.container.image=xuechendi/spark:v2.3.0-with-hibench --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark  local:///opt/HiBench/sparkbench/assembly/target/sparkbench-assembly-7.1-SNAPSHOT-dist.jar "hdfs://10.1.0.35:8020/HiBench/Terasort/Input" "hdfs://10.1.0.35:8020/HiBench/Terasort/Output"

local:// refers to the location inside container.

Run Spark Sql benchmark in Kubernetes pods.

We will also put our application inside HiBench jar.

package com.intel.hibench.sparkbench.sql
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.hive.HiveContext
object ScalaSparkSQLBench{
  def main(args: Array[String]){
    if (args.length < 2){
      System.err.println(
        s"Usage: $ScalaSparkSQLBench <workload name> <SQL sciprt file>"
      )
      System.exit(1)
    }
    val workload_name = args(0)
    val sql_file = args(1)
	// Configure Hive Context inline
    val spark = SparkSession.builder()
    .config("spark.sql.warehouse.dir","hdfs://10.1.0.35:8020/Hive")
    .config("javax.jdo.option.ConnectionURL","jdbc:mysql://10.1.0.35:3306/metastore")
    .config("javax.jdo.option.ConnectionDriverName","com.mysql.jdbc.Driver")
    .config("javax.jdo.option.ConnectionUserName","hive")
    .config("javax.jdo.option.ConnectionPassword","password")
    .config("hive.vectorized.execution.enabled","true")
    .config("hive.vectorized.execution.reduce.enabled","true")
    .config("hive.metastore.warehouse.dir","hdfs://10.1.0.35:8020/Hive")
    .config("hive.exec.scratchdir","hdfs://10.1.0.35:8020/Hive")
    .config("hive.metastore.schema.verification","false")
    .enableHiveSupport()
    .getOrCreate()
    val hc = new HiveContext(spark.sparkContext)

    val _sql = scala.io.Source.fromFile(sql_file).mkString
    _sql.split(';').foreach { x =>
      if (x.trim.nonEmpty)
        hc.sql(x).collect()
}

Compile scala codes with “mvn package” and Build a new docker img with this new jar.

spark-submit --master k8s://https://10.1.0.35:6443 --deploy-mode cluster --properties-file /hadoop/spark/spark.conf --class com.intel.hibench.sparkbench.sql.ScalaSparkSQLBench --num-executors 25 --executor-cores 6 --executor-memory 17g --conf spark.kubernetes.driver.pod.name=spark-sql-tpcds --conf spark.kubernetes.container.image=xuechendi/spark/spark:v2.3.0-with-spark-sql --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark local:///opt/HiBench/sparkbench/assembly/target/sparkbench-assembly-7.1-SNAPSHOT-dist.jar Spark-sql-on-k8s /opt/UC11_K8S/$query_id

Optimization spark in kube performance by mounting a faster media device to container

echo "spark.local.dir /tmp/" >> spark-defaults.conf #so spark will use /tmp as its shuffle & spill directory
echo "VOLUME /tmp" >> Dockerfile #this configuration will tell run-time container to use /var/lib/docker/volumes for /tmp dir inside container.

Debug and check

docker ps
CONTAINER ID        IMAGE                        COMMAND                  CREATED             STATUS              PORTS               NAMES
844ab90bcfb8        71f3e1f14cb6                 "/opt/entrypoint.s..."   4 minutes ago       Up 4 minutes                            k8s_executor_com-intel-hibench-sparkbench-micro-scalaterasort-3ed74fbe8cf935e9a34dfebb8baafbfa-exec-5_d
$ docker inspect 844ab90bcfb8
"Mounts": [
            {
                "Type": "bind",
                "Source": "/var/lib/kubelet/pods/346e134d-8579-11e8-b100-001e677c4f1a/etc-hosts",
                "Destination": "/etc/hosts",
                "Mode": "",
				...

One way I did to debug a spark kube is add a sleep in dockerfile, so once the spark executor pod is up, it won’t die too quickly if this executor hit some error because of the sleep cmdline.

kubectl exec -it ${podname} -- /bin/bash
# this helps to get into the pod for some checking.

[Solved]Hadoop no credential provider issue with Ceph as object store(hadoop 3.1.1)

Wed, 17 Oct 2018 00:00:00 +0000

BACKGROUND

I saw this issue when I upgrade my hadoop to 3.1.1 and my hive to 3.1.0. Didn’t see in hadoop 2.8.5.

I used ceph with ceph radosgw as a replacement to HDFS. Using S3A interface, so it will call some codes in AWSCredentialProviderList.java for a credential checking.

I provided aws s3a.access.key and s3a.secret.key in hadoop core-site.xml, and set defaultFS to s3a://buckets, in that case, it worked fine to run a hadoop mapreduce job and input/output data to s3a.

Also when I can use below command line to manage data in s3a://buckets.

hdfs dfs -ls /

While when I used hive, any metadata operations like create/drop databases/tables worked fine. But when hive scheduled a mapreduce job and connecting with yarn, yarn failed in the credential checking process.

ERROR LOG

2018-10-16 20:28:45,462 INFO [main] org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state INITED
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Error while initializing
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:368)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$6.run(MRAppMaster.java:1760)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1757)
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1691)
Caused by: java.net.SocketTimeoutException: doesBucketExist on tpcds: com.amazonaws.AmazonClientException: No AWS Credentials provided by BasicAWSCredentialsProvider EnvironmentVariableCredentialsProvider InstanceProfileCredentialsProvider : com.amazonaws.SdkClientException: Unable to load credentials from service endpoint
        at org.apache.hadoop.fs.s3a.S3AUtils.translateInterruptedException(S3AUtils.java:330)
        at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:171)
        at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:111)
        at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$3(Invoker.java:260)
        at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:317)
        at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:256)
        at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:231)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.verifyBucketExists(S3AFileSystem.java:372)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:308)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354)
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3403)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3371)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:477)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:226)
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.getFileSystem(MRAppMaster.java:605)
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:315)
        ... 7 more
Caused by: com.amazonaws.AmazonClientException: No AWS Credentials provided by BasicAWSCredentialsProvider EnvironmentVariableCredentialsProvider InstanceProfileCredentialsProvider : com.amazonaws.SdkClientException: Unable to load credentials from service endpoint
        at org.apache.hadoop.fs.s3a.AWSCredentialProviderList.getCredentials(AWSCredentialProviderList.java:139)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1164)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:762)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:724)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4325)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4272)
        at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1337)
        at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:1277)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$verifyBucketExists$1(S3AFileSystem.java:373)
        at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:109)
        ... 21 more
Caused by: com.amazonaws.SdkClientException: Unable to load credentials from service endpoint
        at com.amazonaws.auth.EC2CredentialsFetcher.handleError(EC2CredentialsFetcher.java:183)
        at com.amazonaws.auth.EC2CredentialsFetcher.fetchCredentials(EC2CredentialsFetcher.java:162)
        at com.amazonaws.auth.EC2CredentialsFetcher.getCredentials(EC2CredentialsFetcher.java:82)
        at com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:151)
        at org.apache.hadoop.fs.s3a.AWSCredentialProviderList.getCredentials(AWSCredentialProviderList.java:117)
        ... 35 more
Caused by: java.net.SocketTimeoutException: connect timed out
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:589)

SOLUTION

Add below lines to all yarn nodemanager nodes, and restart yarn service.

After that, yarn will be able to pass the credential check.

cat /etc/profile.d/hive.sh
export PATH=/hadoop/hive/bin:$PATH
export AWS_ACCESS_KEY_ID=${fs.s3a.access.key}
export AWS_SECRET_ACCESS_KEY=${fs.s3a.secret.key}

cat ~/.aws/credentials
[default]
aws_access_key_id = ${fs.s3a.access.key}
aws_secret_access_key = ${fs.s3a.secret.key}

Relative configuration: Get ceph radosgw credential

radosgw-admin user info --uid s3a
{
    "user_id": "s3a",
    "display_name": "s3a",
    "email": "",
    "suspended": 0,
    "max_buckets": 100000,
    "auid": 0,
    "subusers": [
        {
            "id": "s3a:operator",
            "permissions": "full-control"
        }
    ],
    "keys": [
        {
            "user": "s3a",
            "access_key": ${fs.s3a.access.key},
            "secret_key": ${fs.s3a.secret.key}
        }
    ],
    "caps": [],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "temp_url_keys": [],
    "type": "rgw"
}

core-site.xml

<property>
  <name>fs.defaultFS</name>
  <value>s3a://tpcds</value>
</property>
<property>
  <name>hadoop.tmp.dir</name>
  <value>/tmp/hadoop-${user.name}</value>
</property>
<property>
  <name>fs.s3a.access.key</name>
  <value>${fs.s3a.access.key}</value>
</property>
<property>
  <name>fs.s3a.secret.key</name>
  <value>${fs.s3a.secret.key}</value>
</property>
<property>
  <name>fs.s3a.endpoint</name>
  <value>rgw.intel.com:7481</value>
</property>

REASON

Exeption function in AWSCredentialProviderList.java

  /**
   * Iterate through the list of providers, to find one with credentials.
   * If {@link #reuseLastProvider} is true, then it is re-used.
   * @return a set of credentials (possibly anonymous), for authenticating.
   */
  @Override
  public AWSCredentials getCredentials() {
    checkNotEmpty();
    if (reuseLastProvider && lastProvider != null) {
      return lastProvider.getCredentials();
    }

    AmazonClientException lastException = null;
    for (AWSCredentialsProvider provider : providers) {
      try {
        AWSCredentials credentials = provider.getCredentials();
        if ((credentials.getAWSAccessKeyId() != null &&
            credentials.getAWSSecretKey() != null)
            || (credentials instanceof AnonymousAWSCredentials)) {
          lastProvider = provider;
          LOG.debug("Using credentials from {}", provider);
          return credentials;
        }
      } catch (AmazonClientException e) {
        lastException = e;
        LOG.debug("No credentials provided by {}: {}",
            provider, e.toString(), e);
      }
    }

    // no providers had any credentials. Rethrow the last exception
    // or create a new one.
    String message = "No AWS Credentials provided by "
        + listProviderNames();
    if (lastException != null) {
      message += ": " + lastException;
    }
    throw new AmazonClientException(message, lastException);

  }

This function will try to obtain credential iterately from three credential provider: BasicAWSCredentialsProvider EnvironmentVariableCredentialsProvider InstanceProfileCredentialsProvider

We are expecting it found credential by BasicAWSCredentialsProvider, while it failed.

Throught Working with AWS Credentials reference, I noticed add environment configuration works for EC2, so I also tried these in my node, since by checking with BasicAWSCredentialsProvider, hadoop called AmazonHttpClient codes.

And it worked.

Java Program Profiling and Optimization

Thu, 11 Oct 2018 00:00:00 +0000

References:

Java Heap Dump Analyzer

Coding Basics Note 0: Refreshing

Wed, 10 Oct 2018 00:00:00 +0000

Coding Advanced Note 0

Wed, 10 Oct 2018 00:00:00 +0000

Persisten Memory Development Kit(PMDK) Notes 1: How to install

Wed, 10 Oct 2018 00:00:00 +0000

PMDK Installation github page

step1: PREREQUISITES TO BUILD

autoconf

yum install -y autoconf
yum install -y asciidoctor 

pkg-configure

yum groupinstall -y "Development Tools"

libndctl-devel (v60.1 or later), libdaxctl-devel (v60.1 or later), ndctl (v60.1 or later)

git clone https://github.com/pmem/ndctl.git
cd ndctl
git checkout v63

yum install -y acsciidoctor
yum install -y kmod-devel.x86_64
yum install -y libudev-devel
yum install -y libuuid-devel
yum install -y json-c-devel
yum install -y jemalloc-devel

./autogen.sh
./configure CFLAGS='-g -O2' --prefix=/usr --sysconfdir=/etc --libdir=/usr/lib64
make -j
make check
make install
cd ..

step2: PMDK INSTALLATION

yum install -y pandoc
git clone https://github.com/pmem/pmdk.git
cd pmdk
git checkout tags/1.8
make -j
make install
export PKG_CONFIG_PATH=/usr/local/lib64/pkgconfig/:$PKG_CONFIG_PATH
cd ..

libpmemobj++ Installation github page

srep1: PREREQUISITES TO BUILD

cmake >= 3.3 official site

wget https://cmake.org/files/v3.13/cmake-3.13.0-rc1.tar.gz
tar zxf cmake-3.13.0-rc1.tar.gz
cd cmake-3.13.0-rc1/
./bootstrap
make -j
make install
cd ..

cmake --version
cmake version 3.13.0-rc1
CMake suite maintained and supported by Kitware (kitware.com/cmake).

upgrade gcc/g++ to 5.3

sudo yum install -y centos-release-scl
sudo yum install devtoolset-4-gcc*
scl enable devtoolset-4 bash
gcc --version

Persistent Memory Development Kit (PMDK)

step2: COMPILE Libpmemobj++

git clone https://github.com/pmem/libpmemobj-cpp.git
cd libpmemobj-cpp/
git checkout 1.5-rc1
mkdir build
cd build
cmake ..
make -j
make install
cd ../..

Some may happen issue fixing

-- Checking for module 'libpmemobj>=1.4'
--   No package 'libpmemobj' found

export PKG_CONFIG_PATH=/usr/local/lib64/pkgconfig/:$PKG_CONFIG_PATH

llpl(Low Level Persistence Library)JAVA Lib Installation github page

step1: PREREQUISITES TO BUILD

Persistent Memory Development Kit (PMDK)
Java 8 or above

java -version
java version "1.8.0_171"
Java(TM) SE Runtime Environment (build 1.8.0_171-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.171-b11, mixed mode)

Build tools - g++ compiler and make

g++ -v
Using built-in specs.
COLLECT_GCC=g++
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-redhat-linux/4.8.5/lto-wrapper
Target: x86_64-redhat-linux
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-bootstrap --enable-shared --enable-threads=posix --enable-checking=release --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --with-linker-hash-style=gnu --enable-languages=c,c++,objc,obj-c++,java,fortran,ada,go,lto --enable-plugin --enable-initfini-array --disable-libgcj --with-isl=/builddir/build/BUILD/gcc-4.8.5-20150702/obj-x86_64-redhat-linux/isl-install --with-cloog=/builddir/build/BUILD/gcc-4.8.5-20150702/obj-x86_64-redhat-linux/cloog-install --enable-gnu-indirect-function --with-tune=generic --with-arch_32=x86-64 --build=x86_64-redhat-linux
Thread model: posix
gcc version 4.8.5 20150623 (Red Hat 4.8.5-28) (GCC)

step2: BUILD AND RUN

compile

git clone https://github.com/pmem/llpl.git
cd llpl/
make -j

mount a EMULATING PERSISTENT MEMORY or PERSISTENT MEMORY and run tests

[Pmem official doc: How to emulate Persistent Memory]https://pmem.io/2016/02/22/pm-emulation.html

# sudo vi /etc/default/grub
GRUB_CMDLINE_LINUX="memmap=60G!4G"  #60G is pmem size, 4G is start offset
On BIOS-based machines:
# sudo grub2-mkconfig -o /boot/grub2/grub.cfg
On UEFI-based machines:
# sudo grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg

# REBOOT system

sudo mkdir /mnt/mem
sudo mkfs.xfs /dev/pmem0
sudo mount -o dax /dev/pmem0 /mnt/mem
make tests
cd ..

Persisten Memory Development Kit(PMDK) Notes 0: What it is and Quick Examples

Wed, 10 Oct 2018 00:00:00 +0000

What is PMDK, some cool reference below:

For me, PMDK helps me to use persistent memory just like using memory. I can use a special pointer named “persistent_ptr” to access every byte of the big chunk of persistent memory I allocated. And use lots of std::/boost:: data structures there.

PART ONE

So, let’s firstly think about why Persistent Memory is cool?

If we want to use some data in the program life time, we can use any in memory structures like c based array, or std/boost data structures like map, list, vector, etc, and of course self-designed data structure to keep track of where data is.

While if we want to use those data after programs restart or share between programs. We need to store data somewhere not lives inside this program’s context/userspace. Which is to say, options are files, databases(actually another type of file), sending to some long-run services. Any option results in a thousands of times greater latency than using memory. Beacuse we need either disk driver or network driver here.

While using persistent memory, it means we can access data by spending memory-speed latency, we can reuse the data after programs restart, system reboot and even data center servers migration!!!

Why need a new lib to access persistent memory?

If the speed is that fast, why don’t mount it as a normal disk (just use it as a SSD)? You must ask? Yes, of course you can use the PM device that way, and what’s more, PMDK also tells you how to mount as an ordinaty device to your system by DAX mode. DAX is a directly access mode, so you won’t need to go through page cache system which is a default option for most filesystem to speed up. Because you already have a memory(Persistent memory) device, you don’t need another memory as your page cache.

But, the cooler way of using this device is by the lib “PMDK” !!

When you use PMDK to access Persisten memory or even a in-memory temp file for emulation, you can use it just like memory. Malloc, pointers, hashmap, map, tree, graph, yes, you don’t need to worry how to serialize these data structure and how to make write and read efficiently, just mind the real algorithm, and data is consistenly and reliably staying in a non-volatile pereistent memory after you commit transactions.

Transaction Support for PMDK

As a foundation, PMDK uses transation to read and write, we submit our lambda function to PMDK, it will keep data being read and written consistenly (In the right sequence, and no partially dirty issue), once the data committed, PMDK calls our lambda function to call our next step (Or wake/notify up your waiting thread).

PART TWO

PMDK c++ example and a java example

I think now, you get why I say PMDK is dope. And in this intro note, I want to show two most importand API (c++ and java). And PMDK provides so many APIs, and I will cover in the third part for a brief introduction.

C++ EXAMPLE

C++ PMDK API called pmemobjc++, you can refer to PMDK Notes 1: How to install for install details.

#include <iostream>
#include <string>
#include <unistd.h>
#include <stdio.h>
#include <libpmemobj++/make_persistent.hpp>
#include <libpmemobj++/make_persistent_array.hpp>
#include <libpmemobj++/persistent_ptr.hpp>
#include <libpmemobj++/pool.hpp>
#include <libpmemobj++/transaction.hpp>

using pmem::obj::p;
using pmem::obj::persistent_ptr;
using pmem::obj::make_persistent;
using pmem::obj::transaction;
using pmem::obj::pool;
using namespace std;

#define PMEMPOOLSIZE 1024*1024*1024
#define DATASIZE 32
#define LOG(msg) std::cout << msg << "\n"

struct MEMBLOCK{
    persistent_ptr<MEMBLOCK> next;
    persistent_ptr<char[]> value_ptr;
    p<size_t> size;
};

int main() {
    const size_t size = PMEMPOOLSIZE;
    string path = "/mnt/mem/pmempool";
    pool<MEMBLOCK> pmpool;
	
    /* Open and create in persistent memory device */
    if ((access(path.c_str(), F_OK) != 0) && (size > 0)) {
        LOG("Creating filesystem pool, path=" << path << ", size=" << to_string(size));
        pmpool = pool<MEMBLOCK>::create(path.c_str(), "", size, S_IRWXU);
    } else {
        LOG("Opening pool, path=" << path);
        pmpool = pool<MEMBLOCK>::open(path.c_str(), "");
    }

    /* Get root persistent_ptr, 
    the first struct pointer 
    and link to the next one */
    auto block_ptr = pmpool.root();

    /* Cool part, submit your lambda func to pmpool
    , and when the func is ready to run (consistency design),
    it executes below lines.*/
	transaction::run(pmpool, [&] {
        /* Data structure looks like below
        MEMBLOCK0(root) -> MEMBLOCK1 -> MEMBLOCK2 -> ...
            ||                 ||           ||
            \/                 \/           \/
          char[]              char[]       char[]
        */
        auto next_block_ptr = make_persistent<MEMBLOCK>();
        auto data_ptr = make_persistent<char[]>(DATASIZE);
        block_ptr->value_ptr = data_ptr;
        block_ptr->size = DATASIZE;
        block_ptr->next = next_block_ptr;
        block_ptr = next_block_ptr;

        /* To write something, just like doing to a memory pointer */
		char* data_in_mem_ptr = data_ptr.get();
        memset(data_in_mem_ptr, 'a', DATASIZE);

        string data_str(data_in_mem_ptr, DATASIZE);
        LOG("Input Data: " << data_str);
    });

    pmpool.close();
}

Compile

g++ example_libpmemobj++.cpp -o example_pmem++ -std=c++11 -lpmemobj

Verify lib links

ldd example_pmem++
linux-vdso.so.1 =>  (0x00007ffdaac7e000)
libpmemobj.so.1 => /usr/local/lib64/libpmemobj.so.1 (0x00007ff9d0ca3000)
libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007ff9d099c000)
libm.so.6 => /lib64/libm.so.6 (0x00007ff9d069a000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007ff9d0484000)
libc.so.6 => /lib64/libc.so.6 (0x00007ff9d00b7000)
libpmem.so.1 => /usr/local/lib64/libpmem.so.1 (0x00007ff9cfe98000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007ff9cfc94000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007ff9cfa78000)
/lib64/ld-linux-x86-64.so.2 (0x00007ff9d0eda000)

Running

./example_pmem++

[Program output]:

Creating filesystem pool, path=/mnt/mem/pmempool, size=1073741824
Input Data: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa

Verify

./example_pmem++_verify

[Program output]:

Opening pool, path=/mnt/mem/pmempool
Find data: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa

java EXAMPLE

Java PMDK API called llpl, you can refer to PMDK Notes 1: How to install for install details.

package examples.string_store;

import lib.llpl.*;
import java.io.Console;
import java.util.Arrays;
import sun.misc.Unsafe;

public class Writer {
    public static void main(String[] args) {
        /*
        open and create persistent_pool file
        */
        Heap h = Heap.getHeap("/mnt/mem/persistent_pool", 1024*1024*1024L);

        Console c = System.console();
        c.readLine("press Enter to start");

        byte[] bytes = {(byte)'a', (byte)'b', (byte)'c', (byte)'d', (byte)'e',
                        (byte)'f', (byte)'g', (byte)'h', (byte)'i', (byte)'j',
                        (byte)'k', (byte)'l', (byte)'m', (byte)'n', (byte)'o',
                        (byte)'p', (byte)'q', (byte)'r', (byte)'s', (byte)'t'};
        int length = bytes.length;

        /*
        allocate a memory chunk from persistent pool
        */
        MemoryBlock<Transactional> mr = h.allocateMemoryBlock(Transactional.class, Integer.BYTES + length);
		
        /*
        copy bytes array to persistent memory
        */
        mr.copyFromArray(bytes, 0, 0, length);

        /*
        set the memory block as root, 
        and this memory block should point to the next, 
        so you can find any memory block after restart program
        */
        h.setRoot(mr.address());

        System.out.println("String successfully written.");

        /*
        get the actual address in memory of our data
        To check if it is correct.
        we check the value indexed of 10. Which is k
        */
        long addr = mr.payloadAddress(10);
        Unsafe UNSAFE;
        try {
            java.lang.reflect.Field f = Unsafe.class.getDeclaredField("theUnsafe");
            f.setAccessible(true);
            UNSAFE = (Unsafe)f.get(null);
        }
        catch (Exception e) {
            throw new RuntimeException("Unable to initialize UNSAFE.");
        }
        byte res = UNSAFE.getByte(addr);
        System.out.println("Address 10 is " + (char)res);
    }
}

Compile

cd llpl/src
javac -cp ../target/classes examples/string_store/Writer.java

Running

cd llpl/src
java -ea -cp ../target/classes:lib:target/test_classes:./ -Djava.library.path=../target/cppbuild examples/string_store/Writer

[Program output]:

press Enter to start
String successfully written.
Address 10 is k

PART THREE

Besides C++ and JAVA, PMDK provides bunch of other APIs

pmemkv : read and right as a key-value store, it provides C++/C/Java/Ruby/Node.js apis.
librpmem : Write and read data from a persistent memory device over RDMA.

Lessons Learned by ajax request http from https

Wed, 10 Oct 2018 00:00:00 +0000

I am building my jekyll blog portal recently on github, and since github is static, while I still want to add some cool features who can not be simply implemented by javascript inside browser, I came up this thought, can I ajax to my amazon EC2 virtual machine to handle some dynamic requests?

And I have this image in my mind.

Then let’s consider this idea for a bit:

Exists Conditions:

github is now using https protocol for hosting page site, which means my browser is a https client.
It’s easy to host a http server on EC2.

[IDEA 1]

Then I came up with a very naive idea, Is it possible to ajax to a http website from https client?

The answer is NO! Because to request http site data from https downgraded this https website’s security level. Reference

A live example, press to test

function get_subscribe_status () {
	$.ajax({
		url: "http://ec2-18-191-213-11.us-east-2.compute.amazonaws.com/subscribe_status",
		type: 'get',
		dataType: 'text'
	}).done(function(data) {
		$("div.subscribe-block-status").text(data);
	});
}

Result:

[IDEA 2]

Ok, but since this is my own website, how about I simply downgraded my github site to http?

The Answer is also NO!! GITHUB OFFICIAL HELP

HTTPS enforcement is required for GitHub Pages sites using a github.io domain that were created after June 15, 2016. If you created your GitHub Pages site before June 15, 2016, you can manually enable HTTPS enforcement.

I closed my github pages and re-open in 2018… Hum… So I can’t downgrade it… [Sad Face]

[IDEA 3]

Hum, but I can upgrade my EC2 site to provide HTTPS!! How about that?

The Answer is YES to someone but NO to me….

Firstly, it is possible to lauch a HTTPS server by you own. All you need to do is to use openssl to create a private key, then public key and use the key to assign a certificate.

openssl genrsa -des3 -out server.key 1024
openssl req -new -key server.key -out server.csr
openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt 

And then provide this key to your web server process.

But, a self-assigned key is just like a self-assigned job title, no one believes you since the title is not authorized.

So, if I hosting a https website with a self-assigned certificate, Browser will try to warn visitors and even worse, if I browser such a site with my iphone, safari simply declined with no asking…

And in order to have an AUTHORIZED SSL Certificate, I need a domain name which is under my own control.

Since I registered a free-trial amazonaws account, and I can’t have a domain name for my EC2 virtual machine. This is a NO-WAY to me.

[IDEA 4]

I gave up and went to sleep, and while laying in my bed, there is one question arised. Do I have to use http or https? Can I use a socket request? Or other protocol?

And, yes, there is another protocol called “websockets”.

But, quote by this link

When the page is accessed through HTTP, you can use WS or WSS (WebSocket secure: WS over TLS) . However, when your page is loaded through HTTPS, you can only use WSS - browsers don’t allow to “downgrade” security. You can’t use WebSockets over HTTPS, but you can use WebSockets over TLS (HTTPS is HTTP over TLS).

Alright, then how about using socket? I can’t find my reference, but seems this is a experimental feature in many browser, and I didn’t really try it…

After all, I gave up, but I think all these info I got deserved my time, so want to share my experience here.

Bye!

Ways to post codes on Jekyll

Wed, 03 Oct 2018 04:00:00 +0000

[Display with iframe and leetcode]

[Display with embed and gist using jupyter output style]

[Display with embed and gist using plain c output style]

New Journey

Wed, 03 Oct 2018 04:00:00 +0000

It’s been a while since my last post, I am hoping to pick up this habbit and record my daily findings, ideas and tech summarization here.

I am gonna turn 30 in less than 6 months, and I feel I am lost. There are so many things to learn and so many goals to reach. And some people I know at my age had already made something, while I am still working and expecting my monthly payment to keep my life move forward.

Recently, my husband started his 3 years study in UMich, and we started our overseas long-distance relationship life, I tried to apply jobs in Ann Arbor and resulted in zero response. Which makes me to reconsider my current life, why I acted like a good employee with valuable outcomes and improved myself quite much during these years, I still feel I am being trapped.

The conclusion is that, I am trapped by my comfort zone, and there isn’t zero other options, while I choose to ignore them. I want to keep every nice thing I have and want to jump from one safe circle to another with zero gap window, and that is what trapped me.

So, I need to make a choice, to keep my current routine or to put myself to something unknown… …

Hope I will have my conclusion before my 30 birthday.

How much we can see from what we are listening

Wed, 23 Apr 2014 13:01:00 +0000

Just write this post to see(literally see) how much differences there are in a compressed mp3 with lossless FLAC and WAV.

I compared two sessions in TIME(from 《inception》 OST) with three compressed version: 44.1kHZ MP3 184kbps 44.1KHZ 16bit FLAC 608kbps VBR 44.1kHZ 16bit WAV 1141kbps

First bite: mp3

flac

wav

Hum…I swear they are three different screen snapshot… But I did barely see no difference from those waves… Dead eyes…

So, maybe I should find some session more dynamic, and I choose the the first rise in TIME, see below

mp3

flac

wav

Er… still nothing? Look closer…

mp3

flac

wav

put the waves together

MP3 vs WAV: mp3(red line), wav(green line)

FLAC vs WAV:flac(red line),wav(green line)

closer… 300x MP3 vs WAV: mp3(red line), wav(green line)

300x FLAC vs WAV:flac(red line),wav(green line)

Yes! There does have some differences between mp3 with wav!!! And it’s so apparently mp3 lost some tops and bottoms…

Seems I spend another 500% disk space for wav versions does worth something!!! Yep…

But Flac is nearly 50% space occupation of WAV… Still feels like listening to WAV. Er….

Don’t ask me if I can hear the difference between all these version… That’s what I’ll never tell… XOXO, geek girl

VM Creation by Virsh

Thu, 12 Dec 2013 14:34:00 +0000

###xml setting could be like below:

    <domain type ="kvm">
      <uuid>1b87fb8f-442c-4682-b693-b38612342cd1</uuid>      // should be unique
      <name>node3_1 </name>
      <memory>524288 </memory>
      <vcpu>1 </vcpu>
      <cputune>                                               // virsh vcpupin 
        <vcpupin vcpu='0' cpuset ='4'/>
      </cputune>
      <os>
        <type>hvm </type>
        <boot dev="hd" />
      </os>
      <features>
        <acpi/>
      </features>
      <clock offset="utc" >
        <timer name="pit" tickpolicy ="delay"/>
        <timer name="rtc" tickpolicy ="catchup"/>
      </clock>
      <cpu mode="host-model" match="exact" />
      <devices>                                           
        <disk type="file" device ="disk">                     // os disk
          <driver name ="qemu" type="raw" cache="none" />
          <source file ="/var/lib/instances/node3_1/instance.img" />
          <target bus ="virtio" dev="vda"/>
        </disk>
        <hostdev mode='subsystem' type ='pci' managed='yes'>  // nic setting
          <source>
            <address domain ='0x0000' bus='0x04' slot='0x10' function ='0x0'/>
          </source>
          <address type ='pci' domain='0x0000' bus='0x00' slot ='0x05' function='0x0' />
        </hostdev>
        <serial type="pty" />
        <input type="tablet" bus ="usb"/>
        <graphics type="vnc" autoport ="yes" keymap="en-us" listen="0.0.0.0" />
      </devices>
    </domain>

###VM Creation by Virsh

make nic setting is right(SR-IOV, OpenVSwitch)
BIOS enables VT
virsh create libvirt.xml
virsh attach-device domain disk.xml 4.1 Create your own rbd device(rbd info volume-name) 4.2 According to below xml , attach your device

###disk xml could be like below:

    <disk type='network' device ='disk'>
      <driver name ='qemu' type='raw' cache='none' />
      <source protocol ='rbd' name='xiaoxi/volume-79ee1e33-0353-4e22-bf98-d81a91b2c283:debug_rbd=1:debug_client=1:debug_objecter=1:debug_striper=1:log_to_stderr=true' />
      <target dev ='vdb' bus='virtio'/>
      <serial>79ee1e33-0353-4e22-bf98-d81a91b2c283</serial>
      <alias name ='virtio-disk1'/>
      <address type ='pci' domain='0x0000' bus='0x00' slot ='0x06' function='0x0' />
    </disk>

###Using bridge as network interface:

#The primary network interface
auto br0
iface br0 inet dhcp
        bridge_ports eth0
        bridge_stp off
        bridge_fd 0
        bridge_maxwait 0

/etc/init.d/networking restart

<domain type ="kvm">
  <uuid>1b87fb8f-442c-4682-b693-b3866dc20cd3 </uuid>
  <name>node2_1 </name>
  <memory>524288 </memory>
  <vcpu>1 </vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset ='2'/>
  </cputune>
  <os>
    <type>hvm </type>
    <boot dev="hd" />
  </os>
  <features>
    <acpi/>
  </features>
  <clock offset="utc" >
    <timer name="pit" tickpolicy ="delay"/>
    <timer name="rtc" tickpolicy ="catchup"/>
  </clock>
  <cpu mode="host-model" match="exact" />
  <devices>
    <disk type="file" device ="disk">
      <driver name ="qemu" type="raw" cache="none" />
      <source file ="/opt/instances/node2_1/instance.img" />
      <target bus ="virtio" dev="vda"/>
    </disk>
     <interface type='bridge' >
          <source bridge ='br0'/>
          <mac address ='00:11:22:33:44:55'/>
        </interface>
    <serial type="pty" />
    <input type="tablet" bus ="usb"/>
    <graphics type="vnc" autoport ="yes" keymap="en-us" listen="0.0.0.0" />
  </devices>
</domain>

###extra setting to apparmor:

root@CephXCD1:~# tail -3 /etc/apparmor.d/abstractions/libvirt-qemu
  # for rbd
  /etc/ceph/ceph.conf r,
  /usr/local/lib/* rmix,
root@CephXCD1:~# service apparmor restart
Setting vncserver & vncviewer: 

###install vncserver in your hyperviser

root@CephXCD1:~# apt-get install vnc4server
root@CephXCD1:~# adduser vnc
root@CephXCD1:~# /etc/init.d/vncserver
#!/bin/bash
PATH="$PATH:/usr/bin/"
export USER= "vnc"
DISPLAY="1"
DEPTH="16"
GEOMETRY="1024x768"
OPTIONS="-depth ${DEPTH} -geometry ${GEOMETRY} :${DISPLAY}"
. / lib/lsb /init-functions
case "$1" in
start)
log_action_begin_msg "Starting vncserver for user '${USER}' on localhost:${DISPLAY}"
su ${USER} -c "/usr/bin/vncserver ${OPTIONS}"
;;
stop)
log_action_begin_msg "Stoping vncserver for user '${USER}' on localhost:${DISPLAY}"
su ${USER} -c "/usr/bin/vncserver -kill :${DISPLAY}"
;;
restart)
$0 stop
$0 start
;;
esac
exit 0
root@CephXCD1:~# chmod +x vncserver
root@CephXCD1:~# service vncserver start

###configure the client install vnc viewer in a computer with 图形界面, like windows

with best luck, you will enter the VM

System analysis and tools

Fri, 15 Nov 2013 01:09:00 +0000

It’s really clear of these 2 graphs to remind me all those tools can help to analyse the system

Device-Mapper deep dive

Thu, 14 Nov 2013 19:40:00 +0000

I used to stuck in all these terms and concepts for like really long time, just trying to get everything clear by writing.

##Storage subsystem ###Overview

BIO is the unit to map data in the memory to generic block offset. When generic block layer gets bio units, it calls io scheduler to combine bios into request to specific device. Then requests can be sent to real device or virtual block device like software raid or logic volume(using MD or Device Mapper modules).

Actually BIO units point to a much smaller unit named bio_vec which is the exactly unit point to the memory, and BIO also has one field record which block device and which sector it wanna to read/write.(Notice, the block device here is kind of a generic idea, could be some virtual block device)

The smart use of bio_vec help kernel to support scatter/Gather I/O, so that BIO can map some scatter part in mem to some continuous part in block device.

When BIO unit received by generic block layer, kernel will do some “merge and sort” operations then hand the combined BIOs to block device. All these work can be done in the IO scheduler layer and then all BIO units are combined into one and one request, which also be linked by a pointer named “request_queue” store in bdev struct(gendisk).

Then the whole idea is pretty clear, there is a picture shows some important function to translate a fs syscall into requests to block devices.

Submit_bio is a generic api to submit bio to generic block layer(of course by its name…)

generic_make_request puts BIO into bio_list, then __generic_make_request will see if this bio is suitable to make request or it is delivered to some stack device like Device Mapper(in this situation, __generic_make_request will produce a new bio and call generic_make_request).

__make_request_fn() then pus BIO into request_queue, if this function returns 0, the BIO is delivered to the real block device, or it may continues to call __make_request_fn until it delivered to real block device(like Device Mapper).

In fact, the request_queue also will not be directly tackled by block device, the device will use some method named “Plugging/Unplugging” to tackles these requests.

Now let’s see a linked graph of some important structs in block-level subsystem. Every block device has a field named gendisk(generic disk), and generic disk has a field to record its request_queue, and since each type of block device has its own implementation of read and write, the gendisk also has a field named private_data to point to the corresponding block device(also, the block device can be real or virtual).

Then, after all the general idea of block-layer subsystem, let’s talk about `Device Mapper`

What is Device-Mapper

A block device mapping facility available in Linux Kernel.
A component required by LVM2 to support the mapping between logical volumes and physical storage devices.
Device-mapper provides a generic way to create virtual layers of block devices that can do different things on top of real block devices like striping, concatenation, mirroring, snapshotting, etc…

Here is the Usecase

Before lv creation

After lv creation

The thing should notice here is that a dm-0 device in /sys/block is created, which indicates that Logic Volume is a “device mapper” device.

another device mapper usecase

Here is the Device Mapper Overview

DM Devices in Ubuntu 12.10,kernel 3.6.3

Linear Device target-type example

static struct target_type linear_target = {   
	.name   = "linear",   
	.version = {1, 1, 0},   
	.module = THIS_MODULE,   
	.ctr    = linear_ctr,   
	.dtr    = linear_dtr,   
	.map    = linear_map,   
	.status = linear_status,   
	.ioctl  = linear_ioctl,   
	.merge  = linear_merge,   
	.iterate_devices = linear_iterate_devices,
}

Let’s see the codes

int dm_create(int minor, struct mapped_device **result){
   	struct mapped_device *md;
    	md = alloc_dev(minor);   	
	if (!md)       		
		return -ENXIO;   	
	dm_sysfs_init(md);   	
	*result = md;   	
	return 0;
}

static void dm_request(struct request_queue *q, struct bio *bio)
{
	struct mapped_device *md = q->queuedata;

	if (dm_request_based(md))
		blk_queue_bio(q, bio);  //Using dm_target rules to reconstruct the bio
	else
		_dm_request(q, bio);    //split and process this bio
}