grep [pattern] [file_name]

grep -r [pattern] [directory_name]

locate [name]

find [/folder/location] -name [a]

find [/folder/location] -size [+100M]

ls

ls -a

pwd

mkdir [directory]

rm [file_name] 

rm -r [directory_name]

rm -rf [directory_name]

cp [file_name1] [file_name2]

cp -r [directory_name1] [directory_name2]

mv [file_name1] [file_name2]

ln -s /path/to/[file_name] [link_name]

touch [file_name]

more [file_name]

cat [file_name]

cat [file_name1] >> [file_name2]

head [file_name]

tail [file_name]

gpg -c [file_name]

gpg [file_name.gpg]

wc

dmesg

cat /proc/cpuinfo

free -h

lshw

lsblk

lspci -tv

lsusb -tv

dmidecode

hdparm -i /dev/disk

hdparm -tT /dev/[device]

badblocks -s /dev/[device]

cd ..

cd

cd /chosen/directory

tar cf [compressed_file.tar] [file_name]

tar xf [compressed_file.tar]

tar czf [compressed_file.tar.gz]

gzip [file_name]

scp [file_name.txt] [server/tmp]

rsync -a [/your/directory] [/backup/] 

id

last

who

w

groupadd [group_name]

adduser [user_name]

usermod -aG [group_name] [user_name]

sudo [command_to_be_executed_as_superuser]

userdel [user_name] 

usermod

yum list installed

yum search [keyword]

yum info [package_name]

yum install [package_name.rpm]

dnf install [package_name.rpm]

apt-get install [package_name]

rpm -i  [package_name.rpm]

rpm -e [package_name.rpm]

tar zxvf [source_code.tar.gz]
cd [source_code]
./configure
make
make install

ps

pstree

pmap

top

kill [process_id]

pkill [proc_name]

killall [proc_name]

bg

fg

fg [job]

lsof

ip addr show

ip address add [IP_address]

ifconfig

netstat -pnltu

netstat -nutlp

whois [domain]

dig [domain] 

dig -x host

dig -x [ip_address]

host [domain]

hostname -I

wget [file_name]

Ctrl + C

Ctrl + Z

Ctrl + W

Ctrl + U

Ctrl + K

Ctrl + Y

Ctrl + R

Ctrl + O

Ctrl + G

!!

exit

chmod 777 [file_name]

chmod 755 [file_name]

chmod 766 [file_name]

chown [user] [file_name]

chown [user]:[group] [file_name]

df -h

df -i

fdisk -l

du -ah

du -sh

findmnt

mount [device_path] [mount_point]

ssh user@host

ssh host

ssh -p [port] user@host

telnet host

The post Linux Commands for Beginners appeared first on iExpertify.

Related posts:

Linux vs Unix

Linux is a multi-operating system that can be accessed through numerous users concurrently. So, it has to be authenticated to ensure people from accessing other confidential files. Linux can also be used in mainframes and servers without any modifications. It uses the concept of ownership and permissions to enhance the security of the directories and files. 

Linux File Permissions

Types of permissions

Every directories and file in Linux have three basic permission types. They are discussed as follows:

#1 Read Permission

The read permission enables you to open and read a file. For a directory, the read permission enables the user to list the contents of the directory.

#2 Write Permission

The write permission allows the user to modify the file and write new data to the file. For a directory, the write permission allows the user to modify the content of the directory. The user can add, remove, or rename files that belong to a particular directory. 

#3 Execute Permissions

The execute permission allows the user to execute the file as a shell script or a program. For the directory, the execute permission enables the user to access the files in a directory and enter it by using the cd command but it does not allow to list the content.

Viewing the permissions

The view permission allows the user to check the directory or file in the GUI file manager or by reviewing the output using the command given below.

ls -l 

Permissions groups

Every directory and file on Linux is owned by a specific user and group and are defined separately as three user based permission groups. They are as follows:

# User

A user is a person who owns the directory or file. By default, the user who creates the file or directory will be the owner. 

# Group

The user group that owns the directory or file will not affect the actions of other users. All the users who belong to the group that owns the directory or file will have the same permission to access the file or directory.

# Other

The user who is not the owner of the directory or file and doesn’t belong to the same group of the directory or file. Simply, if we set the permission for the ‘other’ category, by default it will affect everyone. 

If you want to view the users on the system, you can view the user using the command below:

cat /etc/passwd

Similarly, you can view the group on the system by using the command below:

cat /etc/group

-rw-rw-r– is a code that represents the permissions given to the owner, user group, and the world.

Here, the ‘-’ represents the selected file. For the directory, it is denoted as ‘d’. 

The characters are simple to remember and understand.

• r- read permission
• w- write permission
• x- execute permission
• _- no permission

The first part of the code ‘rw-’ represents the owner can read the file, write the file, but cannot execute the file since the execute bit is set to ‘-’. Several Linux distributions such as CentOS, Ubuntu, Fedora, etc. will add users to the group of the same group name as the username. 

The second part of the code ‘rw-’ represents for the user group and group members can read the file, write the file.

The third part of the code ‘r–’ represents any user and the user can only read the file.

Changing file permission using chmod

With the help of the change mode ‘chmod’ command, we can set the permissions such as read, write, and execute on a directory or file for the owner, user, and the group.

chmod <permission-number> <file-name>

Here, the permission number is calculated by using the assigned values for r, w, and x. The basic permission number includes three digits. Some special cases can use four digits as a permission number. 

There are two ways to use the commands. They are as follows: 

1. Numeric mode

In a numeric mode, file permissions do not denote as characters but as a three-digit octal number. The following table provides the numbers for all permission types.

For example, see the command below.

$ chmod 764 sample

ls -l sample

rwxrw-r-- 1 iExpertify iExpertify 20 June 20 06:00 sample

chmod 764 and checking permission. In the above command, we have changed the file permissions to 764. 764 represents the following:

• The owner can read, write, and execute
• The user group can read and write
• Any user can only read

2. Symbolic mode

In this mode, we can change permissions for all three owners. We can modify the permissions of a specific owner. With the use of mathematical symbols, we can modify the file permissions.

Let’s see the example:

Current file

ls -l sample

-rw-rw-r-- 1 iExpertify iExpertify 22 2020-06-29 13:45 sample?

Setting permission to other users

$ chmod 0=rwx sample

$ ls -l sample

-rw-rw-rwx 1 iExpertify iExpertify 22 2020-06-29 13:45 sample

Adding execute permission to the user group

$ chmod g+x sample

$ ls -l sample

-rw-rwxrwx 1 iExpertify iExpertify 22 2020-06-29 13:45 sample?

Removing read permission for the user

$ chmod u-r sample

$ ls -l sample

--w-rwxrwx 1 iExpertify iExpertify 22 2020-06-29 13:45 sample

Changing ownership and group

For changing the ownership of a directory or file, use the command below:

chown user

If you want to change the user along with the group for a directory or file, use the command below

chown user: group filename

If you wish to change group owner only, use the command below

chgrp group_name filename

Here, chgrp represents for change group

Advanced permissions

The special permissions that are used to access directories or files are as following:

• _ – It represents there are no special permissions.
• d- It represents the directory.
• l-  It represents the symbolic link of a directory or a file.
• t- It represents the sticky bit permissions. It represents ‘t’ in the executable portion of all user permissions.
• s- It indicates the setuid or setgid permissions. It represents ‘s’ in the read portion of the owner or group permissions.

Setuid or setgid special permissions

The setuid or setgid permissions are used to assign the system to run an executable as the owner with the owner’s permissions. We can assign this permission by explicit defining permissions.

The character that represents the setuid or setgid is ‘s’. To set the setuid or setgid bit on file1.sh, use the command below:

chmod g+s file1.sh

Be careful while using setuid or setgid permissions. If you assign the permissions incorrectly, then your system goes to intrusion.

Sticky bit special permissions

The sticky bit can be useful in a shared environment because when it is assigned to the permissions on a directory it sets permissions for the file owner only to rename or delete the file.

The character for the sticky bits is ‘t’. To set the sticky bits on a directory, use the command below:

chmod+t dir1

Tip

• The file /etc/group contains all the groups defined in the system
• You can use the command “groups” to find all the groups you are a member of
• You can use the command newgrp to work as a member a group other than your default group
• You cannot have 2 groups owning the same file.
• You do not have nested groups in Linux. One group cannot be sub-group of other
• x- eXecuting a directory means Being allowed to “enter” a dir and gain possible access to sub-dirs
• There are other permissions that you can set on Files and Directories which will be covered in a later advanced tutorial

Summary:

• Linux being a multi-user system uses permissions and ownership for security.
• There are three user types on a Linux system viz. User, Group and Other
• Linux divides the file permissions into read, write and execute denoted by r,w, and x
• The permissions on a file can be changed by ‘chmod’ command which can be further divided into Absolute and Symbolic mode
• The ‘chown’ command can change the ownership of a file/directory. Use the following commands: chown user file or chown user:group file
• The ‘chgrp’ command can change the group ownership chrgrp group filename
• What does x – eXecuting a directory mean? A: Being allowed to “enter” a dir and gain possible access to sub-dirs.

The post Linux File Permissions -chmod appeared first on iExpertify.

What are Regular Expressions?

Grep Regex is one of the most popular command-line utilities to find and search strings in a text file.

Using the grep command with regular expressions makes it even more powerful.

Regular expressions come in the picture when you want to search for a text containing a particular pattern.

It simplifies your search operation by searching the patterns on each line of the file.

Types of Regular expressions

For ease of understanding let us learn the different types of Regex one by one.

Basic Regular expressions

Some of the commonly used commands with Regular expressions are tr, sed, vi and grep. Listed below are some of the basic Regex.

Let’s create a sample test.txt file with the following content:

cat test.txt

Output:

This record ends in 2021
2021 is the start of the record

2021
2020

This record ends in 2020
2020 is the start of the record

For example, find all the lines which end with the word 2021:

grep "2021$" test.txt

You should see the following output:

This record ends in 2021

2021

Next, find all the lines which start and end with the word 2021:

grep "^2021$" test.txt

You should see the following output:

2021

For example, find all the lines which end with the word 2020 or 2021(The dot allows any single character in the place):

grep "202.$" test.txt

You should see the following output:

This record ends in 2021

2021

This record ends in 2020

2020

Now, lets display all the lines that start with the string balaram:

grep "^2021" test.txt

You should see the following output:

2021 is the start of the record
2021

Next, find the number of blank lines in the file test.txt:

grep "^$" test.txt

You should see the following output:

<3 empty lines>

Interval Regular expressions

These expressions tell us about the number of occurrences of a character in a string. They are

Example:

Filter out all lines that contain character ‘p’

We want to check that the character ‘p’ appears exactly 2 times in a string one after the other. For this the syntax would be:

cat sample | grep -E p\{2}

Note: You need to add -E with these regular expressions.

Extended regular expressions

These regular expressions contain combinations of more than one expression. Some of them are:

Example:

Searching for all characters ‘t’

Suppose we want to filter out lines where character ‘a’ precedes character ‘t’

We can use command like

cat sample|grep "a\+t"

The regular expression (\) used to search for special characters.

Let’s create a sample test.txt file with the following contents:

cat test.txt

Output:

1.1.1.1
1a1a1a1
1b1c1d1

Now, search for all the lines which matches the pattern “1.1.1.1“:

grep "1.1.1.1" test.txt

This command does not show the proper result as “.” matches any single character:

1.1.1.1
1a1a1a1
1b1c1d1

You can use the regular expression “\” to resolve this issue:

grep "1\.1\.1\.1" test.txt

Output:

1.1.1.1

Brace expansion

The syntax for brace expansion is either a sequence or a comma separated list of items inside curly braces “{}”. The starting and ending items in a sequence are separated by two periods “..”.

Some examples:

In the above examples, the echo command creates strings using the brace expansion.

Let’s create a sample test.txt file with the following contents:

cat test.txt

Output:

apple
appple
appppple

Now, search for all the lines which match a character “p” two times:

grep -E "ap{2}l" test.txt

You should see the following output:

apple

Next, search for all the lines which match a character “p” two or more times:

grep -E "ap{2,}l" test.txt

You should see the following output:

apple
appple
appppple

Next, search for all the lines which match a character “p” two or three times:

grep -E "ap{2,3}l" test.txt

You should see the following output:

apple
appple

Square bracket expansion

The regular expression [] can be used to match any one character found within the bracket group.

search for all the lines which match any range character found within the “test” group.

grep "test[x-z]" test.txt

You should see the following output:

testx
testy
testz

The post Grep Regex Example Linux appeared first on iExpertify.

What is Data Modeling?

The method of creating a data store model is called data processing in a database. Data modeling is the process of identifying the entities in our domain, the relationships between these entities and how they will be stored in the database. This introduces theoretical data objects and connections between different data objects. Data modelling is a data formulation process in a standardized format in an information system. It helps to quickly analyze data, which helps to meet business needs. The data modelling process requires data modelers who work correctly with stakeholders and prospective IT users. Data modelling ends with the development of a model of data supporting the infrastructure for the business information system

Understanding Data Modeling / Scope

It occurs at three different layers:

• Physical model: It is a schema which says how data is stored physically in the database
• Conceptual model: It is the user view of the data i.e. the high level which the user sees.
• Logical model: It sits between the Physical model and conceptual model and it represents the data logically, separate from its physical stores.

Hierarchical Data Modeling: These models were used to replace file-based systems. The data was kept in a tree like one too many arrangements.

Relational Data Modeling: It’s true that the hierarchical model helped us to move from file-based systems which reduced complexity but still one had known the specific physical data storage employed. The relational database follows the relational model where data is stored in tables, unlike Hierarchical database where it is stored in a tree-like structure. In short, it reduced the complexity more when compared to the hierarchical model.

How does Data Modeling make work so easy/why should we use it?

It helps us in a visual representation of data and enforces business logic, regulations, policies, etc on data. It is a guide which is used by scientists and analysts in the designing and implementation of a database. So, without data modeling the job of analysts and scientists to implement the business requirements on database becomes difficult.

• Data modeling is query based, that is, we think of the application workflow and the queries early on in the data model process
• A Table is how databases stores data and can be thought of as a set of rows and columns
• After designing the entities (or tables) that we need, we decide how the tables would be related to each other
• The next thought process is to define the primary and foreign keys for the tables. The primary key uniquely identifies a single record on the table. A foreign key is used to relate to other tables within the data model.
• For operational databases/transactional systems, the goal of data model is to reduce any duplication so that we don’t store the same data in multiple places.
• For dimensional models, data warehouses, data marts, etc, the goal is to optimize the select query. Duplication of data is not considered an issue. We do not optimize for writes as writes are considered cheaper than select statements. so, we design the database to solve all the fields that are needed in a select query the BI tools run, the report that gets generated or the model that the data scientists need to work with.
• One of our goals in data modeling for data warehouses is even data distribution. As the data is dispersed across nodes (servers), the optimal distribution of data so as to query within one node or query disperses equal load across all nodes may be considered based on what database we are working with (Say, it depends on Teradata vs Netezza)
• Selecting the Primary Key is very important and has a huge impact on query performance

Why do we need Data Modeling? / What can you do with it?

The main goal of using it is:

• To ensure that all data objects are represented correctly as if it is not done correctly we would get incorrect results.
• It helps as stated earlier to design database at conceptual, physical and logical levels.
• It helps to design the relational tables, primary keys, foreign keys, etc.
• Database developers can create a better physical database with a good model as it becomes a guiding tool for them.
• It helps to identify missing and redundant data.
• It helps us to have a better IT infrastructure and to have easy and cheap maintenance when required in the long run though it’s time-consuming initially.

Working

Now let’s create a sample data model to understand how to work with a model. To do this we have to follow certain steps:

• First we have to understand the requirements, In this case, we will create a model for an online store. So, keeping that in mind we need two tables a) customers b) products
• Next step is to get the attributes of the tables or entities

a. customer table can have attributes like:

• Id
• Name
• Email
• Address

b. Product table can have attributes like:

• Id
• Name

In the customer table, we can have Id as Primary key and similarly Product Id in Product table will be the primary key.

Now, we will design the relationship between these two tables. So to connect the customer and product table we will create a table called purchase which will be like an order table (i.e. which customer ordered which product).

Who is the right audience for learning this technology?

It is very essential. The right audiences for learning modeling techniques are individuals who are data architects and data analysts. Most individuals start as data analysts and then move up the ladder.

How this technology will help you in career growth?

According to Glassdoor, the average salary in the market for modelers is projected to earn about $78,601 on an average. So you can see that it is a well-paid job. Most big companies invest in modelers as they are very essential for keeping the integrity of data.

Conclusion

In conclusion, we can say that the model created by modelers ensure consistency in naming conventions, integrity, and security of data. because good data will enable the business in the correct efficient utilization of their data.

The post What is Data Modeling? appeared first on iExpertify.

Just as the output of a command can be redirected to a file, so can the input of a command be redirected from a file. As the greater-than character > is used for output redirection, the less-than character < is used to redirect the input of a command.

We may want a file to be the input for a command that normally wouldn’t accept a file as an option. This redirecting of input is done using the “<” (less-than symbol) operator.

Below is an example of sending a file to somebody, using input redirection.

> mail john.dba@organization.com < sample_ddl

This reads a bit more difficult than the beginner’s cat file | mail someone, but it is of course a much more elegant way of using the available tools.

File Descriptors (FD)

In Linux/Unix, everything is a file. Regular file, Directories, and even Devices are files. Every File has an associated number called File Descriptor (FD).

Your screen also has a File Descriptor. When a program is executed the output is sent to File Descriptor of the screen, and you see program output on your monitor. If the output is sent to File Descriptor of the printer, the program output would have been printed.

Whenever you execute a program/command at the terminal, 3 files are always open, viz., standard input, standard output, standard error.

Following is a complete list of commands which you can use for redirection −

The post Input Redirection in Linux/Unix Examples appeared first on iExpertify.

Q) What is a Snowflake cloud data warehouse?

Ans. Snowflake is an analytic data warehouse implemented as a SaaS service. It is built on a new SQL database engine with a unique architecture built for the cloud. This cloud-based data warehouse solution was first available on AWS as software to load and analyze massive volumes of data. The most remarkable feature of Snowflake is its ability to spin up any number of virtual warehouses, that means the user can operate an unlimited number of independent workloads against the same data without any risk of contention.

Q) What is Unique about Snowflake Cloud Data Warehouse?

Ans. Snowflake is cloud native (built for the cloud).So, It takes advantage of all the good things about the cloud and brings exciting new features like,

• Auto scaling
• Zero copy cloning
• Dedicated virtual warehouses
• Time travel
• Military grade encryption and security
• Robust data protection features

Snowflake is a poetry. It’s beautifully crafted with smart defaults –

• All the data is compressed by default
• All the data is encrypted
• Its Columnar, thereby making the column level analytical operations a lot faster

Not to mention the number of innovations in the product – eg. Intelligent Services layer, data shares, tasks & streams. Snowflake also has a simple and transparent pricing, which makes it very easier even for smaller businesses to afford a cloud datawarehouse

Snowflake Full Course

Q) How is data stored in Snowflake?

Ans. Snowflakes store the data in multiple micro partitions which are internally optimized and compressed. The data is stored in a columnar format in the cloud storage of Snowflake. The data objects stored by Snowflake cannot be accessed or visible to the users. By running SQL query operations on Snowflake, you can access them.

Q4) What type of database is Snowflake?

Ans. Snowflake is built entirely on a SQL database. It’s a columnar-stored relational database that works well with Excel, Tableau, and many other tools. Snowflake contains its query tool, supports multi-statement transactions, role-based security, etc., which are expected in a SQL database.

Q) What is a Columnar database and what are its benefits?

Ans. Columnar databases organize data at Column level instead of the conventional row level. All Column level operations will be much faster and consume less resources when compared to a row level relational database

Q) What are the different ways to access the Snowflake Cloud Datawarehouse?

Ans. You can access the Snowflake Data Warehouse using

• Web User Interface
• ODBC Drivers
• JDBC Drivers
• SnowSQL Command line Client
• Python Libraries

Q) Can AWS glue connect to Snowflake? 

Ans. Definitely. AWS glue presents a comprehensive managed environment that easily connects with Snowflake as data warehouse service. These two solutions collectively enable you to handle data ingestion and transformation with more ease and flexibility.

Q) Explain Snowflake editions.

Ans. Snowflake offers multiple editions depending on your usage requirements.

• Standard edition – Its introductory level offering provides unlimited access to Snowflake’s standard features.
• Enterprise edition – Along with Standard edition features and services, offers additional features required for the large-scale enterprises.
• Business-critical edition – Also, called Enterprise for Sensitive Data (ESD). It offers high-level data protection for sensitive data to organization needs.
• Virtual Private Snowflake (VPS) – Provides high-level security for organizations dealing with financial activities.

Q) What is Snowflake Caching?

Ans. Imagine executing a query that takes 10 minutes to complete. Now if you re-run the same query later in the day while the underlying data hasn’t changed, you are essentially doing again the same work and wasting resources

Instead Snowflake caches the results of every query you ran and when a new query is submitted, it checks previously executed queries and if a matching query exists and the results are still cached, it uses the cached result set instead of executing the query. This can greatly reduce query times because Snowflake retrieves the result directly from the cache.

Q) Define the Snowflake Cluster.

Ans. In Snowflake, data partitioning is called clustering, that specifies cluster keys on the table. The method by which you manage clustered data in a table is called re-clustering.

Q) Explain Snowflake architecture.

Ans. Snowflake is built on a patented, multi-cluster, shared data architecture created for the cloud. Snowflake architecture is comprised of storage, compute, and services layers that are logically integrated but scale infinitely and independent from one another. Snowflake is built on AWS/Azure/GCP cloud data warehouse and is truly Saas offering. There is no software, hardware, ongoing maintenance, tuning, etc. needed to work with Snowflake.

Three main layers make the Snowflake architecture – database storage, query processing, and cloud services.

• Data storage – In Snowflake, the stored data is reorganized into its internal optimized, columnar, and optimized format. 
• Query processing – Virtual warehouses process the queries in Snowflake.
• Cloud services – This layer coordinates and handles all activities across the Snowflake. It provides the best results for Authentication, Metadata management, Infrastructure management, Access control, and Query parsing.

Q) What are the features of Snowflake? 

Ans. Unique features of the Snowflake data warehouse are listed below:

• Database and Object Closing
• Support for XML
• External tables
• Hive metastore integration
• Supports geospatial data
• Security and data protection
• Data sharing
• Search optimization service
• Table streams on external tables and shared tables
• Result Caching

Q) What is Time Travel in Snowflake?

Ans. Time travel is a cool feature which lets you access data as of any time in the past. For example, if you have an Employee table and if you delete the table accidentally you can use time travel and go back 5 minutes and retrieve the data back. Snowflake Time Travel enables accessing historical data (i.e., data that has been changed or deleted) at any point within a defined period. It serves as a powerful tool for performing the following tasks:

• Query data in the past that has since been updated or deleted
• Create clones of entire tables, schemas, and databases at or before specific points in the past
• Restore tables, schemas, and databases that have been dropped

Q) Tell me something about Snowflake AWS?

Ans. For managing today’s data analytics, companies rely on a data platform which offers rapid deployment, compelling performance, and on-demand scalability. Snowflake on the AWS platform serves as a SQL data warehouse, which makes modern data warehousing effective, manageable, and accessible to all data users. It enables the data-driven enterprise with secure data sharing, elasticity and per-second pricing.

Q) Describe Snowflake computing. 

Ans. Snowflake cloud data warehouse platform provides instant, secure, and governed access to the entire data network and a core architecture to enable various types of data workloads, including a single platform for developing modern data applications.  

Q) What is the schema in Snowflake?

Ans. Schemas and databases used for organizing data stored in the Snowflake. A schema is a logical grouping of database objects such as tables, views, etc. The benefits of using Snowflake schemas are it provides structured data and uses small disk space.

Q) What kind of SQL does Snowflake use?

Ans. Snowflake supports the most common standardized version of SQL, i.e., ANSI for powerful relational database querying.

Q) What are the cloud platforms currently supported by Snowflake?

Ans. 

• Amazon Web Services (AWS)
• Google Cloud Platform (GCP)
• Microsoft Azure (Azure)

Q) What ETL tools do you use with Snowflake?

Ans. Following are the best ETL tools for Snowflake:

• Matillion
• Blendo
• Hevo Data
• StreamSets
• Etleap
• Apache Airflow 

The post Snowflake Interview Questions appeared first on iExpertify.

ROLAP

This methodology relies on manipulating the data stored in the relational database to give the appearance of traditional OLAP’s slicing and dicing functionality. In essence, each action of slicing and dicing is equivalent to adding a “WHERE” clause in the SQL statement

Advantages of ROLAP Cube

• High data efficiency. It offers high data efficiency because query performance and access language are optimized particularly for the multidimensional data analysis.
• Scalability. This type of OLAP system offers scalability for managing large volumes of data, and even when the data is steadily increasing.

Disadvantages of ROLAP Cube

• Demand for higher resources: ROLAP needs high utilization of manpower, software, and hardware resources.
• Aggregately data limitations. ROLAP tools use SQL for all calculation of aggregate data. However, there are no set limits to the for handling computations.
• Slow query performance. Query performance in this model is slow when compared with MOLAP

Hybrid OLAP

HOLAP technologies attempt to combine the advantages of MOLAP and ROLAP. For summary-type information, HOLAP leverages cube technology for faster performance. When detail information is needed, HOLAP can “drill through” from the cube into the underlying relational data

1. Aggregated or computed data is stored in a multidimensional OLAP cube
2. Detailed information is stored in a relational database.

MOLAP

MOLAP uses array-based multidimensional storage engines to display multidimensional views of data. Basically, they use an OLAP cube. Data is pre-computed,pre-summarized, and stored in a MOLAP (a major difference from ROLAP).

Using a MOLAP, a user can use multidimensional view data with different facets. Multidimensional data analysis is also possible if a relational database is used. By that would require querying data from multiple tables. On the contrary, MOLAP has all possible combinations of data already stored in a multidimensional array. MOLAP can access this data directly. Hence, MOLAP is faster compared to Relational Online Analytical Processing (ROLAP).

MOLAP Tools

1. Essbase – Tools from Oracle that has a multidimensional database.
2. Express Server – Web-based environment that runs on Oracle database.
3. Yellowfin – Business analytics tools for creating reports and dashboards.
4. Clear Analytics – Clear analytics is an Excel-based business solution.
5. SAP Business Intelligence – Business analytics solutions from SAP

Implementation considerations is MOLAP

• In MOLAP it’s essential to consider both maintenance and storage implications to creating strategy for building cubes.
• Proprietary languages used to query MOLAP. However, it involves extensive click and drag support for example MDX by Microsoft.
• Difficult to scale because the number and size of cubes required when dimensions increase.
• API’s should provide for probing the cubes.
• Data structure to support multiple subject areas of data analyses which data can be navigated and analyzed. When the navigation changes, the data structure needs to be physically reorganized.
• Need different skill set and tools for Database administrator to build, maintain the database.

MOLAP Advantages

• MOLAP can manage, analyze and store considerable amounts of multidimensional data.
• Fast Query Performance due to optimized storage, indexing, and caching.
• Smaller sizes of data as compared to the relational database.
• Automated computation of higher level of aggregates data.
• Help users to analyze larger, less-defined data.
• MOLAP is easier to the user that’s why It is a suitable model for inexperienced users.
• MOLAP cubes are built for fast data retrieval and are optimal for slicing and dicing operations.
• All calculations are pre-generated when the cube is created.

MOLAP Disadvantages

• One major weakness of MOLAP is that it is less scalable than ROLAP as it handles only a limited amount of data.
• The MOLAP also introduces data redundancy as it is resource intensive
• MOLAP Solutions may be lengthy, particularly on large data volumes.
• MOLAP products may face issues while updating and querying models when dimensions are more than ten.
• MOLAP is not capable of containing detailed data.
• The storage utilization can be low if the data set is highly scattered.
• It can handle the only limited amount of data therefore, it’s impossible to include a large amount of data in the cube itself.

The post MOLAP, ROLAP and HOLAP appeared first on iExpertify.

Database and data warehouse are both systems that store data. But they serve very different purposes.

What is a Data Warehouse?

A data warehouse is an information system which stores historical and commutative data from single or multiple sources. Data flows into a data warehouse from transactional systems, relational databases, and other sources, typically on a regular cadence. Business analysts, data engineers, data scientists, and decision makers rely on reports, dashboards, and analytics tools(business intelligence (BI) tools, SQL clients) to extract insights from their data, monitor business performance, and support decision making. It is designed to analyze, report, integrate transaction data from different sources.

Data Warehouse eases the analysis and reporting process of an organization. It is also a single version of truth for the organization for decision making and forecasting process.

What is a Database?

A database is a computerized system that makes it easy to search, select and store information. Databases are used in many different places. A database (DB), in the most general sense, is an organized collection of data. Most common examples are relational databases like Oracle, SQL SERVER and MySQL, etc. 

How do data warehouses and databases co-exist?

Why use a Database?

Here, are prime reasons for using Database system:

• It offers the security of data and its access
• A database offers a variety of techniques to store and retrieve data.
• Database act as an efficient handler to balance the requirement of multiple applications using the same data
• A DBMS offers integrity constraints to get a high level of protection to prevent access to prohibited data.
• A database allows you to access concurrent data in such a way that only a single user can access the same data at a time.

Why Use Data Warehouse?

Here, are Important reasons for using Data Warehouse:

• Data warehouse helps business users to access critical data from some sources all in one place.
• It provides consistent information on various cross-functional activities
• Helps you to integrate many sources of data to reduce stress on the production system.
• Data warehouse helps you to reduce TAT (total turnaround time) for analysis and reporting.
• Data warehouse helps users to access critical data from different sources in a single place so, it saves user’s time of retrieving data information from multiple sources. You can also access data from the cloud easily.
• Data warehouse allows you to stores a large amount of historical data to analyze different periods and trends to make future predictions.
• Enhances the value of operational business applications and customer relationship management systems
• Separates analytics processing from transactional databases, improving the performance of both systems
• Stakeholders and users may be overestimating the quality of data in the source systems. Data warehouse provides more accurate reports.

Processing Types: OLAP vs OLTP

The most significant difference between databases and data warehouses is how they process data. While most databases are OLTP application files, most data warehouses are online application processing (OLAP) files. OLAP gets information by gathering data from OLTP and other database files. Because of how OLAP files are structured, it’s far easier to run queries and analyses on the data they contain, and anyone can query the data warehouse with either data warehouse software or knowledge of SQL. Individual subsections of the data warehouse, which are typically relevant to an individual team or department, are called “data marts.”

The data in databases are normalized. The goal of normalization is to reduce and even eliminate data redundancy, i.e., storing the same piece of data more than once. This reduction of duplicate data leads to increased consistency and, thus, more accurate data as the database stores it in only one place.

Difference between Database and Data Warehouse

Databases aren’t better than data warehouses, or vice versa. They perform very different functions from one another, and each is very powerful.

Applications of Database

Database Use Cases

Databases process the day-to-day transactions in an organization. Some examples of database applications include:

• An ecommerce website creating an order for a product it has sold
• An airline using an online booking system
• A hospital registering a patient
• A bank adding an ATM withdrawal transaction to an account

Applications of Data Warehousing

Data Warehouse Use Cases

Data warehouses provide high-level reporting and analysis that empower businesses to make more informed business. Use cases include:

• Segmenting customers into different groups based on their past purchases to provide them with more tailored content
• Predicting customer churn using the last ten years of sales data
• Creating demand and sales forecasts to decide which areas to focus on next quarter

Summary

Now you understand the difference between a database and a data warehouse and when to use which one. Your business needs both an effective database and data warehouse solution to truly succeed in today’s economy.

The post Difference between Database and Data Warehouse appeared first on iExpertify.

The post Linux vs Unix appeared first on iExpertify.

In the Datastage Director, we can:

• run, schedule and monitor jobs
• view job status, logs and schedules
• filter the displayed events

Click on the datastage director icon to open the application:

Fill out the server details, user credentials and choose the project name

The DataStage Director window is divided into two panes:

• The Job Category pane lists all of the jobs in the repository.
• The right pane shows one of three views: Status view, Schedule view, or Log view.

This table describes DataStage Director menu options:

DataStage Director has three view options:

• The Status view displays the status, date and time started, elapsed time, and other run information about each job in the selected repository category.
• The Schedule view displays job scheduling details.
• The Log view displays all of the events for a particular run of a job.

To check for job completions, these can be checked in Status column ( Complied, Aborted, Finished, etc.). The start time and end times are also listed in the director

TO see summary of a particular job run, double click the job and the below window with job parameters, status, etc will pop up

For debugging, we would need to look at the detailed log. click on the log icon

The log would have info records, warnings and Fatal errors that would help in debugging the issues.

The post Introduction to DataStage Director appeared first on iExpertify.