December 01, 2021

Data Engineering Analogy

Data is the new oildata is the economic  fuel of the future. Like oil, data must be refined, cleansed.

Data is the new energy. You consume data irrespective of from which source it came and enriched similar to energy , it come from windmills stored in power station and consumed by end user and pay per use.

Data is the new sunlight. Like solar rays, Data will be everywhere and underlie everything



Creating DataFrames from CSV in Apache Spark

 from pyspark.sql import SparkSession spark = SparkSession.builder.appName("CSV Example").getOrCreate() sc = spark.sparkContext Sp...