Showing posts with label spark. Show all posts
Showing posts with label spark. Show all posts

Saturday, December 23, 2017

Developers Needs SDKMAN Not Super-Man


Every developer has pain for setup development environment to his/her machine with lots of the setups. Sometimes, the pain goes beyond while we need to test same application on multiple versions of sdks or virtual machines.


If you are a Mac user, you have the best option called brew installer.




But if you are Linux user, your pain is unpredictable. 






We are Java developers and Linux users and have the same pain for setting development environment with lots of configuration and different versions virtual machines.

For the sake of innocent developers, for the sake of time, we are going to introduce our superhero called SDKMAN. Which saves us from the cruel world of setup developments tools.



Technical Introduction:  


SDKMAN! is a tool for managing parallel versions of multiple Software Development Kits on most Unix based systems. It provides a convenient Command Line Interface (CLI) and API for installing, switching, removing and listing Candidates. SDKMAN is primary used for JVM based languages and framework. In future, they were plan to move SDKMAN for other environments as well. Currently SDKMAN have a huge list of sdks, which we get from here

Install SDKMAN

$ curl -s "https://get.sdkman.io" | bash

$ source "$HOME/.sdkman/bin/sdkman-init.sh"

$ sdk version

Install Java

For installing java, SDK provide simple and easy command as below:


$ sdk install java

Downloading: java 8u152-zulu

In progress...

######################################################################## 100.0%

Repackaging Java 8u152-zulu...

Done repackaging...

Installing: java 8u152-zulu
Done installing!


Setting java 8u152-zulu as default.
root@a33316a976d9:~/.sdkman# java -version
openjdk version "1.8.0_152"
OpenJDK Runtime Environment (Zulu 8.25.0.1-linux64) (build 1.8.0_152-b16)
OpenJDK 64-Bit Server VM (Zulu 8.25.0.1-linux64) (build 25.152-b16, mixed mode)

By default, sdkman download the zulu or open source JDK of java. But if we require installing some specific version of JDK or Specific Oracle JDK, what can we do???


SDKMAN gave us the way to download sdk's with specific versions as well. We can easily list out the existing SDK's which SDKMAN support and install it as per requirements.


$ sdk list

================================================================================
Available Candidates
================================================================================
q-quit                                  /-search down
j-down                                  ?-search up
k-up                                    h-help

--------------------------------------------------------------------------------
Ant (1.10.1)                                             https://ant.apache.org/

Apache Ant is a Java library and command-line tool whose mission is to drive
processes described in build files as targets and extension points dependent
upon each other. The main known usage of Ant is the build of Java applications.
Ant supplies a number of built-in tasks allowing to compile, assemble, test and
run Java applications. Ant can also be used effectively to build non Java

So on ...................

$ sdk list java

================================================================================
Available Java Versions
================================================================================
     9.0.1-zulu                                                                    
     9.0.1-oracle                                                                  
     9.0.0-zulu                                                                    
 > * 8u152-zulu                                                                    
     8u151-oracle                                                                  
     8u144-zulu                                                                    
     8u131-zulu                                                                    
     7u141-zulu                                                                    
     6u93-zulu                              
	 
$ sdk install java 8u151-oracle

Oracle requires that you agree with the Oracle Binary Code License Agreement
prior to installation. The license agreement can be found at:

  http://www.oracle.com/technetwork/java/javase/terms/license/index.html

Do you agree to the terms of this agreement? (Y/n): y


Downloading: java 8u151-oracle

In progress...

######################################################################## 100.0%

Repackaging Java 8u151-oracle...

Done repackaging...

Installing: java 8u151-oracle
Done installing!

Do you want java 8u151-oracle to be set as default? (Y/n): y

Setting java 8u151-oracle as default.

$ java -version
java version "1.8.0_151"
Java(TM) SE Runtime Environment (build 1.8.0_151-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode)

As it shows, we can install oracle java successfully. But, at the start of this blog, as we discussed we can install multiple version of the same SDK easily and manage easily. If we go through the blog again, first we install OpenJDK after we are installing OracleJDK, single machine multiple JDKS and we also set OracleJDK as default, so how can we use OpenJDK as per our requirements??

Below are powerfull and ease commands of SDKMAN which help us to achieve this functionality.


$ sdk list java

================================================================================
Available Java Versions
================================================================================
     9.0.1-zulu                                                                    
     9.0.1-oracle                                                                  
     9.0.0-zulu                                                                    
   * 8u152-zulu                                                                    
 > * 8u151-oracle                                                                  
     8u144-zulu                                                                    
     8u131-zulu                                                                    
     7u141-zulu                                                                    
     6u93-zulu                                                                     
                                                                                   
                                                                                   
                                                                                   
                                                                                   
                                                                                   
                                                                                   

================================================================================
+ - local version
* - installed
> - currently in use
================================================================================


$ java -version
java version "1.8.0_151"
Java(TM) SE Runtime Environment (build 1.8.0_151-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode)

$ sdk use java 8u152-zulu

$ java -version
openjdk version "1.8.0_152"
OpenJDK Runtime Environment (Zulu 8.25.0.1-linux64) (build 1.8.0_152-b16)
OpenJDK 64-Bit Server VM (Zulu 8.25.0.1-linux64) (build 25.152-b16, mixed mode)

I am sure, now you can feel the power of SDKMAN and how easy is using this tool. This makes developers life happy and safe.


References: 

  1. http://sdkman.io/index.html
  2. Thanks to google funny images.

Wednesday, January 25, 2017

Short Interview With SMACK Tech Stack !!!

Hello guy's, today's we conduct short interview with SMACK about its architecture and there uses. Let's start with of some introduction.

Interviewer: How would you describe your self ?
SMACK: I am SMACK (Spark, Mesos, Akka, Cassandra and Kafka) and belongs to all open source technologies. Mesosphere and Cisco collaboration bundles these technologies together and create a product called Infinity.  Which is used to solved pipeline data challenges where the speed of response is matters like fraud detection system.

Interviewer: Why SMACK ?
SMACK: Now day's modern data-processing challenges are :
  • Data is getting bigger or more accurately, the number of data source is increasing.
  • Today, many modern business models data from one hour ago is practically obsolete.
  • Data analysis becomes to slow to get any return on investment info.
  • One modern requirement is to have horizontal scaling with low cost.
  • We live in a age where data freshness matters many times more than the amount or size of data.
There are many challenges we are facing, SMACK exist because one technology doesn't make an architecture. SMACK is a pipelined architecture model for data processing.

Interviewer: Lambda Architecture is data processing architecture and have advantage of both batch and stream processing methods, So, how SMACK different ?
SMACK: Yes, Lambda Architecture have these features, but most of the lambdas solutions cannot meet two needs at the same time :
  1. Handles a massive data stream in real time.
  2. Handles multiple and different data models from multiple data source.
For these. Apache Spark is responsible for real time analysis for both historical and recent and from massive information torrent and all such information and analysis results are persisted in Apache Cassandra. So, in the case of failure we can recover the real time data from any point of time. With lambda Architecture it's not always possible.

Interviewer: SMACK can you briefly describe about you technologies ?
SMACK: Yes sure, as we discussed SMACK is basically used for Pipeline data architecture for online data stream processing. There are lots of books and articles are available on each and every technology but we are using every technology for some specific purpose like:
  • Apache Spark: Processing Engine.
  • Akka: The Model.
  • Apache Kafka: The Broker.
  • Apache Cassandra: The Storage.
  • Apache Mesos: The Container.
See, all are Apache projects with the exception of Akka.

Interviewer: Is, SMACK is only solution ?
SMACK: No, You can replace individual components as per you requirements like Yarn could be used as the cluster scheduler instead of Mesos and Apache Flink would be suitable batch and stream processing alternatives to Akka. There are many alternatives to SMACK.

Interviewer: Could you discuss one of your case study with us ?
SMACK: Yes, But not know. I need to go with my technologies for some hangout, that we will discuss further in our next interview.

Interviewer: Can I take one picture of you ?
SMACK: Yes sure, cheeezzzz .....


References: