January 13, 2019

MongoDB: Schemaless does not mean you have no Schema


RDBMSs usually have a pre-defined schema: tables with columns, each with names and a data type. When working with a RDBMSs, we’re often confronted with complex schemas that define the structure of the data. When we want to make changes to the database, we may have to wrestle with schema changes as well. The implications of making a schema change include being sure that existing data fits the new schema. Or, more commonly, that the existing application programming won’t break when we modify the database schema. The strict controls and exactness imposed by schemas and by typed languages allow you to keep large groups of developers on the same page, and can allow you to catch bugs earlier in the development cycle.

Document databases are a flexible alternative to the pre-defined schemas of relational databases. Each document in a collection can have a unique set of fields, and those fields can be added or removed from documents once they are inserted which makes document databases, and MongoDB in particular, an excellent way to prototype applications. However, this flexibility is not without cost and the most underestimated cost is that of predictability.

MongoDB is an open-source, non-relational database developed by MongoDB, Inc. MongoDB stores data as documents in a binary representation called BSON (Binary JSON).Fields can vary from document to document; there is no need to declare the structure of documents to the system – documents are self-describing. If a new field needs to be added to a document then the field can be created without affecting all other documents in the collection, without updating a central system catalog, and without taking the system offline. MongoDB’s document data model maps naturally to objects in application code, making it simple for developers to learn and use. With a schemaless database, 90% of the time adjustments to the database become transparent and automatic. This makes rapid development and changes easy.

When you say “schemaless”, you actually say “dynamically typed schema” – as opposed to statically typed schemas as they are available from SQL databases. JSON is still a completely schema free data structure standard.

A good technique  is to have a schema definition which can be shared among programs and tools. Different programs can then agree on the schema.We can also represent a schema definition which can be useful in design and modeling, data validation, and schema migration.

Also having a properly designed schema will allow you to get the best performance from MongoDB and  Data Analysts will also know of different policies and metadata that are required to understand the data.

January 05, 2019

Who Moved My Cheese?



It is a well-known fact that any technology can fade in no time due to new and rapid advancements rolling out every now or then. Hence, you have to be aware of the advancements in the technology industry and trends that can benefit your business in the future.

Anticipate Change

Even if we are doing well, we should always have a Plan B. Better be prepared than regret later.

Monitor Change: 

Smell the Cheese Often So You Know When It Is Getting Old
You should be receptive to your environment in order to observe the changes around you. 
Managed database services moved Cheese from Database developers but they can always look for role in Big Data for another and better career.

Adaption:

The Quicker You Let Go Of Old Cheese, the Sooner You Can Enjoy New CheeseAs soon as we notice that there is no cheese left we should not waste any time, and left in search of a new cheese.So what do you do when change happens? Just go with it. Nothing good will happen if you deny it or just complain about it. It is safer to search in the maze than to remain in a cheese-less situation.

e.g Cloud technology moved Cheese from System Administrator, Network Administrators but they can always look for DevOps, Cloud architect roles

Move with the Cheese

Hem was hesitant to leave Cheese Station C because that was his comfort zone. What is holding Hem back? Fear. He was frightened of the unknown that’s why he refused to move on. On the other hand, Haw realises that he has to overcome this fear in order to find a new cheese.

Enjoy Change! : 

Savour the Adventure and Enjoy the Taste Of New Cheese!
Project Leader/Project Manager having 15+ years of experience are in comfort zone and and given up  Hands-on and now suddenly with project management tools, there would be not main role of these project managers. Now that you accepted change, enjoy the new cheese! You need to  appreciated the change of pace and the thrill of the hunt. Now, you should  ready to embark on a new journey in your  life as a Scrum master in Agile Development.

Be Ready To Change Quickly and Enjoy It Again: They Keep Moving the Cheese

Again, as you enjoy the new cheese, be wary of your surroundings because another one will come soon. An AI bot creates log-in tests and then goes throughout the SDLC. Today CI/CD and streamlining long-term, this will be very commonplace. Testing will be as natural as writing code which will be done by machines and Automation testing role soon outdated. You’ll experience success with new role by steering continuous delivery teams towards quality instead of being a quality gatekeeper, monitoring impact of code changes in production, eliminating testing bottlenecks, improve slow feedback loops in continuous delivery pipelines

Like the characters in the "Who Moved My Cheese?” story, we will learn that these lessons are applicable to the circumstances we encounter in our career. It will change your thinking and attitude towards change. It’s ideal to be like Sniff and Scurry who are always ready and adaptive, but it’s not too late to be like Hem who learned embraced change.

Creating DataFrames from CSV in Apache Spark

 from pyspark.sql import SparkSession spark = SparkSession.builder.appName("CSV Example").getOrCreate() sc = spark.sparkContext Sp...