Bash tricks for data analysis

In the realm of data analysis, the command-line shell Bash stands as an invaluable tool that empowers analysts and researchers to swiftly process, manipulate, and glean insights from their datasets. With its powerful combination of text-processing tools and scripting capabilities, Bash provides an arsenal of essential tricks that can expedite data tasks and facilitate quick exploratory analysis.

This article unveils a collection of essential Bash tricks that are indispensable for anyone seeking to harness the potential of command-line data analysis. Whether you’re a seasoned data professional or an aspiring analyst, these tricks will not only enhance your efficiency but also unlock new ways of handling and understanding your data.

From reading and previewing data to filtering, aggregating, and transforming it, we’ll delve into a range of techniques that cover fundamental aspects of data analysis. By the end of this article, you’ll be equipped with a toolkit of Bash tricks that will empower you to efficiently process, explore, and manipulate data directly from the command line.

Prepare to embark on a journey through the world of Bash data analysis, where the command line becomes your canvas, and a few simple commands wield the power to transform raw data into valuable insights.

Reading and Previewing Data

# Display the first few lines of a file 
head data.csv 
 
# Count the number of lines in a file 
wc -l data.csv

Filtering Data

# Filter lines containing a specific keyword 
grep "keyword" data.txt 
 
# Filter lines by column value (assuming CSV format) 
awk -F ',' '$2 > 50' data.csv

Aggregating Data

# Calculate sum/average of a specific column (assuming CSV format) 
awk -F ',' '{sum+=$3} END {print sum}' data.csv 
 
# Group by a specific column and calculate aggregates 
cut -d ',' -f 2 data.csv | sort | uniq -c

Replacing Text

# Replace occurrences of a specific word in a file 
sed -i 's/old_word/new_word/g' data.txt

Joining Data

# Join two CSV files based on a common column (using join command) 
join -t ',' -1 2 -2 1 file1.csv file2.csv

Sorting Data

# Sort lines based on a specific column 
sort -t ',' -k 3n data.csv

Extracting Columns

# Extract specific columns from a CSV file 
cut -d ',' -f 2,4 data.csv

Piping and Chaining Commands

# Pipe output of one command as input to another 
cat data.txt | grep "keyword" 
 
# Chain multiple commands together 
cat data.csv | grep "keyword" | awk -F ',' '{print $2}'

Looping and Scripting

# Iterate over files in a directory and perform an operation 
for file in *.txt; do 
    echo "Processing $file" 
    # Your operation here 
done

Extracting Unique Values

# Get unique values from a column (assuming CSV format) 
cut -d ',' -f 2 data.csv | sort -u

Remember that while Bash is powerful for data manipulation, it’s generally more suitable for quick data tasks and preprocessing. For more complex data analysis tasks, you might consider using dedicated data analysis tools and programming languages like Python, R, or specialized data analysis platforms.

Join Medium with my referral link - Konstantinos Patronas

As a Medium member, a portion of your membership fee goes to writers you read, and you get full access to every story…