Spark SQL Programming Interview Questions & Answers

  1. Question 1. What Is Shark?

    Answer :

    Most of the data users know only SQL and are not good at programming. Shark is a tool, developed for people who are from a database background – to access Scala MLib capabilities through Hive like SQL interface. Shark tool helps data users run Hive on Spark – offering compatibility with Hive metastore, queries and data.

  2. Question 2. Most Of The Data Users Know Only Sql And Are Not Good At Programming. Shark Is A Tool, Developed For People Who Are From A Database Background – To Access Scala Mlib Capabilities Through Hive Like Sql Interface. Shark Tool Helps Data Users Run Hive On Spark – Offering Compatibility With Hive Metastore, Queries And Data.

    Answer :

    1. Sensor Data Processing –Apache Spark’s ‘In-memory computing’ works best here, as data is retrieved and combined from different sources.
    2. Spark is preferred over Hadoop for real time querying of data
    3. Stream Processing – For processing logs and detecting frauds in live streams for alerts, Apache Spark is the best solution.
  3. Python Interview Questions

  4. Question 3. What Is A Sparse Vector?

    Answer :

    sparse vector has two parallel arrays –one for indices and the other for values. These vectors are used for storing non-zero entries to save space.

  5. Question 4. What Is Rdd?

    Answer :

    RDDs (Resilient Distributed Datasets) are basic abstraction in Apache Spark that represent the data coming into the system in object format. RDDs are used for in-memory computations on large clusters, in a fault tolerant manner. RDDs are read-only portioned, collection of records, that are –

    • Immutable – RDDs cannot be altered.
    • Resilient – If a node holding the partition fails the other node takes the data.
  6. Python Tutorial

  7. Question 5. Explain About Transformations And Actions In The Context Of Rdds.

    Answer :

    Transformations are functions executed on demand, to produce a new RDD. All transformations are followed by actions. Some examples of transformations include map, filter and reduceByKey.

    Actions are the results of RDD computations or transformations. After an action is performed, the data from RDD moves back to the local machine. Some examples of actions include reduce, collect, first, and take.

  8. Core Java Interview Questions

  9. Question 6. What Are The Languages Supported By Apache Spark For Developing Big Data Applications?

    Answer :

    Scala, Java, Python, R and Clojure

  10. Question 7. Can You Use Spark To Access And Analyse Data Stored In Cassandra Databases?

    Answer :

    Yes, it is possible if you use Spark Cassandra Connector.

  11. Core Java Tutorial
    JDBC Interview Questions

  12. Question 8. Is It Possible To Run Apache Spark On Apache Mesos?

    Answer :

    Yes, Apache Spark can be run on the hardware clusters managed by Mesos.

  13. Question 9. Explain About The Different Cluster Managers In Apache Spark

    Answer :

    The 3 different clusters managers supported in Apache Spark are:

    • YARN
    • Apache Mesos -Has rich resource scheduling capabilities and is well suited to run Spark along with other applications. It is advantageous when several users run interactive shells because it scales down the CPU allocation between commands.
    • Standalone deployments – Well suited for new deployments which only run and are easy to set up.
  14. DB2 Using SQL Interview Questions

  15. Question 10. How Can Spark Be Connected To Apache Mesos?

    Answer :

    To connect Spark with Mesos-

    • Configure the spark driver program to connect to Mesos. Spark binary package should be in a location accessible by Mesos. (or)
    • Install Apache Spark in the same location as that of Apache Mesos and configure the property ‘spark.mesos.executor.home’ to point to the location where it is installed.
  16. JDBC Tutorial

  17. Question 11. How Can You Minimize Data Transfers When Working With Spark?

    Answer :

    Minimizing data transfers and avoiding shuffling helps write spark programs that run in a fast and reliable manner. The various ways in which data transfers can be minimized when working with Apache Spark are:

    1. Using Broadcast Variable- Broadcast variable enhances the efficiency of joins between small and large RDDs.
    2. Using Accumulators – Accumulators help update the values of variables in parallel while executing.
    3. The most common way is to avoid operations ByKey, repartition or any other operations which trigger shuffles.
  18. Domain Name System(DNS) Interview Questions

  19. Question 12. Why Is There A Need For Broadcast Variables When Working With Apache Spark?

    Answer :

    These are read only variables, present in-memory cache on every machine. When working with Spark, usage of broadcast variables eliminates the necessity to ship copies of a variable for every task, so data can be processed faster. Broadcast variables help in storing a lookup table inside the memory which enhances the retrieval efficiency when compared to an RDD lookup ().

  20. Python Interview Questions

  21. Question 13. Is It Possible To Run Spark And Mesos Along With Hadoop?

    Answer :

    Yes, it is possible to run Spark and Mesos with Hadoop by launching each of these as a separate service on the machines. Mesos acts as a unified scheduler that assigns tasks to either Spark or Hadoop.

  22. DB2 Using SQL Tutorial

  23. Question 14. What Is Lineage Graph?

    Answer :

    The RDDs in Spark, depend on one or more other RDDs. The representation of dependencies in between RDDs is known as the lineage graph. Lineage graph information is used to compute each RDD on demand, so that whenever a part of persistent RDD is lost, the data that is lost can be recovered using the lineage graph information.

  24. Question 15. How Can You Trigger Automatic Clean-ups In Spark To Handle Accumulated Metadata?

    Answer :

    You can trigger the clean-ups by setting the parameter ‘spark.cleaner.ttl’ or by dividing the long running jobs into different batches and writing the intermediary results to the disk.

  25. Hibernate Interview Questions

  26. Question 16. Explain About The Major Libraries That Constitute The Spark Ecosystem

    Answer :

    • Spark MLib- Machine learning library in Spark for commonly used learning algorithms like clustering, regression, classification, etc.
    • Spark Streaming – This library is used to process real time streaming data.
    • Spark GraphX – Spark API for graph parallel computations with basic operators like joinVertices, subgraph, aggregateMessages, etc.
    • Spark SQL – Helps execute SQL like queries on Spark data using standard visualization or BI tools.
  27. Hibernate Tutorial

  28. Question 17. What Are The Benefits Of Using Spark With Apache Mesos?

    Answer :

    It renders scalable partitioning among various Spark instances and dynamic partitioning between Spark and other big data frameworks.

  29. MYSQL DBA Interview Questions

  30. Question 18. What Is The Significance Of Sliding Window Operation?

    Answer :

    Sliding Window controls transmission of data packets between various computer networks. Spark Streaming library provides windowed computations where the transformations on RDDs are applied over a sliding window of data. Whenever the window slides, the RDDs that fall within the particular window are combined and operated upon to produce new RDDs of the windowed DStream.

  31. Core Java Interview Questions

  32. Question 19. What Is A Dstream?

    Answer :

    Discretized Stream is a sequence of Resilient Distributed Databases that represent a stream of data. DStreams can be created from various sources like Apache Kafka, HDFS, and Apache Flume. DStreams have two operations –

    • Transformations that produce a new DStream.
    • Output operations that write data to an external system.
  33. Java Swing Tutorial

  34. Question 20. When Running Spark Applications, Is It Necessary To Install Spark On All The Nodes Of Yarn Cluster?

    Answer :

    Spark need not be installed when running a job under YARN or Mesos because Spark can execute on top of YARN or Mesos clusters without affecting any change to the cluster.

  35. Java Swing Interview Questions

  36. Question 21. What Is Catalyst Framework?

    Answer :

    Catalyst framework is a new optimization framework present in Spark SQL. It allows Spark to automatically transform SQL queries by adding new optimizations to build a faster processing system.

  37. Question 22. Name A Few Companies That Use Apache Spark In Production.

    Answer :

    Pinterest, Conviva, Shopify, Open Table

  38. Scala Tutorial

  39. Question 23. Which Spark Library Allows Reliable File Sharing At Memory Speed Across Different Cluster Frameworks?

    Answer :


    Work On Interesting Data Science Projects using Spark to build an impressive project portfolio!

  40. SQLite Interview Questions

  41. Question 24. Why Is Blinkdb Used?

    Answer :

    BlinkDB is a query engine for executing interactive SQL queries on huge volumes of data and renders query results marked with meaningful error bars. BlinkDB helps users balance ‘query accuracy’ with response time.

  42. JDBC Interview Questions

  43. Question 25. How Can You Compare Hadoop And Spark In Terms Of Ease Of Use?

    Answer :

    Hadoop MapReduce requires programming in Java which is difficult, though Pig and Hive make it considerably easier. Learning Pig and Hive syntax takes time. Spark has interactive APIs for different languages like Java, Python or Scala and also includes Shark i.e. Spark SQL for SQL lovers – making it comparatively easier to use than Hadoop.

  44. Apache ZooKeeper Tutorial

  45. Question 26. What Are The Common Mistakes Developers Make When Running Spark Applications?

    Answer :

    Developers often make the mistake of-

    • Hitting the web service several times by using multiple clusters.
    • Run everything on the local node instead of distributing it.

    Developers need to be careful with this, as Spark makes use of memory for processing.

  46. Apache Spark Interview Questions

  47. Question 27. What Is The Advantage Of A Parquet File?

    Answer :

    Parquet file is a columnar format file that helps –

    • Limit I/O operations
    • Consumes less space
    • Fetches only required columns.
  48. DB2 Using SQL Interview Questions

  49. Question 28. What Are The Various Data Sources Available In Sparksql?

    Answer :

    • Parquet file
    • JSON Datasets
    • Hive tables
  50. Hyper SQL Database Tutorial

  51. Question 29. How Spark Uses Hadoop?

    Answer :

    Spark has its own cluster management computation and mainly uses Hadoop for storage.

  52. Scala Interview Questions

  53. Question 30. What Are The Key Features Of Apache Spark That You Like?

    Answer :

    • Spark provides advanced analytic options like graph algorithms, machine learning, streaming data, etc
    • It has built-in APIs in multiple languages like Java, Scala, Python and R
    • It has good performance gains, as it helps run an application in the Hadoop cluster ten times faster on disk and 100 times faster in memory.
  54. Question 31. What Do You Understand By Pair Rdd?

    Answer :

    Special operations can be performed on RDDs in Spark using key/value pairs and such RDDs are referred to as Pair RDDs. Pair RDDs allow users to access each key in parallel. They have a reduceByKey () method that collects data based on each key and a join () method that combines different RDDs together, based on the elements having the same key.

  55. Question 32. Which One Will You Choose For A Project –hadoop Mapreduce Or Apache Spark?

    Answer :

    As it is known that Spark makes use of memory instead of network and disk I/O. However, Spark uses large amount of RAM and requires dedicated machine to produce effective results. So the decision to use Hadoop or Spark varies dynamically with the requirements of the project and budget of the organization.

  56. Apache ZooKeeper Interview Questions

  57. Question 33. Explain About The Different Types Of Transformations On Dstreams?

    Answer :

    • Stateless Transformations- Processing of the batch does not depend on the output of the previous batch. Examples – map (), reduceByKey (), filter ().
    • Stateful Transformations- Processing of the batch depends on the intermediary results of the previous batch. Examples –Transformations that depend on sliding windows.
  58. Domain Name System(DNS) Interview Questions

  59. Question 34. Explain About The Popular Use Cases Of Apache Spark

    Answer :

    Apache Spark is mainly used for

    • Iterative machine learning.
    • Interactive data analytics and processing.
    • Stream processing
    • Sensor data processing
  60. Question 35. Is Apache Spark A Good Fit For Reinforcement Learning?

    Answer :

    No. Apache Spark works well only for simple machine learning algorithms like clustering, regression, classification.

  61. Question 36. What Is Spark Core?

    Answer :

    It has all the basic functionalities of Spark, like – memory management, fault recovery, interacting with storage systems, scheduling tasks, etc.

  62. Hibernate Interview Questions

  63. Question 37. How Can You Remove The Elements With A Key Present In Any Other Rdd?

    Answer :

    Use the subtractByKey () function

  64. Question 38. What Is The Difference Between Persist() And Cache()

    Answer :

    persist () allows the user to specify the storage level whereas cache () uses the default storage level.

  65. Question 39. What Are The Various Levels Of Persistence In Apache Spark?

    Answer :

    Apache Spark automatically persists the intermediary data from various shuffle operations, however it is often suggested that users call persist () method on the RDD in case they plan to reuse it. Spark has various persistence levels to store the RDDs on disk or in memory or as a combination of both with different replication levels.

    The various storage/persistence levels in Spark are –

    • OFF_HEAP
  66. Question 40. How Spark Handles Monitoring And Logging In Standalone Mode?

    Answer :

    Spark has a web based user interface for monitoring the cluster in standalone mode that shows the cluster and job statistics. The log output for each job is written to the work directory of the slave nodes.

  67. MYSQL DBA Interview Questions

  68. Question 41. Does Apache Spark Provide Check Pointing?

    Answer :

    Lineage graphs are always useful to recover RDDs from a failure but this is generally time consuming if the RDDs have long lineage chains. Spark has an API for check pointing i.e. a REPLICATE flag to persist. However, the decision on which data to checkpoint – is decided by the user. Checkpoints are useful when the lineage graphs are long and have wide dependencies.

  69. Question 42. How Can You Launch Spark Jobs Inside Hadoop Mapreduce?

    Answer :

    Using SIMR (Spark in MapReduce) users can run any spark job inside MapReduce without requiring any admin rights.

  70. Java Swing Interview Questions

  71. Question 43. How Spark Uses Akka?

    Answer :

    Spark uses Akka basically for scheduling. All the workers request for a task to master after registering. The master just assigns the task. Here Spark uses Akka for messaging between the workers and masters.

  72. Question 44. How Can You Achieve High Availability In Apache Spark?

    Answer :

    • Implementing single node recovery with local file system
    • Using StandBy Masters with Apache ZooKeeper.
  73. Question 45. Hadoop Uses Replication To Achieve Fault Tolerance. How Is This Achieved In Apache Spark?

    Answer :

    Data storage model in Apache Spark is based on RDDs. RDDs help achieve fault tolerance through lineage. RDD always has the information on how to build from other datasets. If any partition of a RDD is lost due to failure, lineage helps build only that particular lost partition.

  74. Question 46. Explain About The Core Components Of A Distributed Spark Application.

    Answer :

    • Driver- The process that runs the main () method of the program to create RDDs and perform transformations and actions on them.
    • Executor –The worker processes that run the individual tasks of a Spark job.
    • Cluster Manager-A pluggable component in Spark, to launch Executors and Drivers. The cluster manager allows Spark to run on top of other external managers like Apache Mesos or YARN.
  75. Question 47. What Do You Understand By Lazy Evaluation?

    Answer :

    Spark is intellectual in the manner in which it operates on data. When you tell Spark to operate on a given dataset, it heeds the instructions and makes a note of it, so that it does not forget – but it does nothing, unless asked for the final result. When a transformation like map () is called on a RDD-the operation is not performed immediately. Transformations in Spark are not evaluated till you perform an action. This helps optimize the overall data processing workflow.

  76. Question 48. Define A Worker Node.

    Answer :

    A node that can run the Spark application code in a cluster can be called as a worker node. A worker node can have more than one worker which is configured by setting the SPARK_ WORKER_INSTANCES property in the file. Only one worker is started if the SPARK_ WORKER_INSTANCES property is not defined.

  77. Question 49. What Do You Understand By Schemardd?

    Answer :

    An RDD that consists of row objects (wrappers around basic string or integer arrays) with schema information about the type of data in each column.

  78. Question 50. What Are The Disadvantages Of Using Apache Spark Over Hadoop Mapreduce?

    Answer :

    Apache spark does not scale well for compute intensive jobs and consumes large number of system resources. Apache Spark’s in-memory capability at times comes a major roadblock for cost efficient processing of big data. Also, Spark does have its own file management system and hence needs to be integrated with other cloud based data platforms or apache hadoop.

  79. Question 51. Is It Necessary To Install Spark On All The Nodes Of A Yarn Cluster While Running Apache Spark On Yarn ?

    Answer :

    No , it is not necessary because Apache Spark runs on top of YARN.

  80. Question 52. What Do You Understand By Executor Memory In A Spark Application?

    Answer :

    Every spark application has same fixed heap size and fixed number of cores for a spark executor. The heap size is what referred to as the Spark executor memory which is controlled with the spark.executor.memory property of the –executor-memory flag. Every spark application will have one executor on each worker node. The executor memory is basically a measure on how much memory of the worker node will the application utilize.

  81. Question 53. What Does The Spark Engine Do?

    Answer :

    Spark engine schedules, distributes and monitors the data application across the spark cluster.

  82. Question 54. What Makes Apache Spark Good At Low-latency Workloads Like Graph Processing And Machine Learning?

    Answer :

    Apache Spark stores data in-memory for faster model building and training. Machine learning algorithms require multiple iterations to generate a resulting optimal model and similarly graph algorithms traverse all the nodes and edges.These low latency workloads that need multiple iterations can lead to increased performance. Less disk access and  controlled network traffic make a huge difference when there is lots of data to be processed.

  83. Question 55. Is It Necessary To Start Hadoop To Run Any Apache Spark Application ?

    Answer :

    Starting hadoop is not manadatory to run any spark application. As there is no seperate storage in Apache Spark, it uses Hadoop HDFS but it is not mandatory. The data can be stored in local file system, can be loaded from local file system and processed.

  84. Question 56. What Is The Default Level Of Parallelism In Apache Spark?

    Answer :

    If the user does not explicitly specify then the number of partitions are considered as default level of parallelism in Apache Spark.

  85. Question 57. Explain About The Common Workflow Of A Spark Program

    Answer :

    • The foremost step in a Spark program involves creating input RDD’s from external data.
    • Use various RDD transformations like filter() to create new transformed RDD’s based on the business logic.
    • persist() any intermediate RDD’s which might have to be reused in future.
    • Launch various RDD actions() like first(), count() to begin parallel computation , which will then be optimized and executed by Spark.
  86. Question 58. Name A Few Commonly Used Spark Ecosystems.

    Answer :

    Spark SQL (Shark)

    Spark Streaming




  87. Question 59. What Is “spark Sql”?

    Answer :

    Spark SQL is a Spark interface to work with structured as well as semi-structured data. It has the capability to load data from multiple structured sources like “text files”, JSON files, Parquet files, among others. Spark SQL provides a special type of RDD called SchemaRDD. These are row objects, where each object represents a record.

  88. Question 60. Can We Do Real-time Processing Using Spark Sql?

    Answer :

    Not directly but we can register an existing RDD as a SQL table and trigger SQL queries on top of that.

  89. Question 61. What Is Spark Sql?

    Answer :

    SQL Spark, better known as Shark is a novel module introduced in Spark to work with structured data and perform structured data processing. Through this module, Spark executes relational SQL queries on the data. The core of the component supports an altogether different RDD called SchemaRDD, composed of rows objects and schema objects defining data type of each column in the row. It is similar to a table in relational database.

  90. Question 62. What Is A Parquet File?

    Answer :

    Parquet is a columnar format file supported by many other data processing systems. Spark SQL performs both read and write operations with Parquet file and consider it be one of the best big data analytics format so far.

  91. Question 63. List The Functions Of Spark Sql.

    Answer :

    Spark SQL is capable of:

    • Loading data from a variety of structured sources
    • Querying data using SQL statements, both inside a Spark program and from external tools that connect to Spark SQL through standard database connectors (JDBC/ODBC). For instance, using business intelligence tools like Tableau
    • Providing rich integration between SQL and regular Python/Java/Scala code, including the ability to join RDDs and SQL tables, expose custom functions in SQL, and more
  92. Question 64. What Is Spark?

    Answer :

    Spark is a parallel data processing framework. It allows to develop fast, unified big data application combine batch, streaming and interactive analytics.

  93. Question 65. What Is Hive On Spark?

    Answer :

    Hive is a component of Hortonworks’ Data Platform (HDP). Hive provides an SQL-like interface to data stored in the HDP. Spark users will automatically get the complete set of Hive’s rich features, including any new features that Hive might introduce in the future.

    The main task around implementing the Spark execution engine for Hive lies in query planning, where Hive operator plans from the semantic analyzer which is translated to a task plan that Spark can execute. It also includes query execution, where the generated Spark plan gets actually executed in the Spark cluster.

  94. Question 66. What Is A “parquet” In Spark?

    Answer :

    “Parquet” is a columnar format file supported by many data processing systems. Spark SQL performs both read and write operations with the “Parquet” file.

  95. Question 67. What Are Benefits Of Spark Over Mapreduce?

    Answer :

    Due to the availability of in-memory processing, Spark implements the processing around 10-100x faster than Hadoop MapReduce. MapReduce makes use of persistence storage for any of the data processing tasks.

    • Unlike Hadoop, Spark provides in-built libraries to perform multiple tasks form the same core like batch processing, Steaming, Machine learning, Interactive SQL queries. However, Hadoop only supports batch processing.
    • Hadoop is highly disk-dependent whereas Spark promotes caching and in-memory data storage
    • Spark is capable of performing computations multiple times on the same dataset. This is called iterative computation while there is no iterative computing implemented by Hadoop.
  96. Question 68. How Sparksql Is Different From Hql And Sql?

    Answer :

    SparkSQL is a special component on the spark Core engine that support SQL and Hive Query Language without changing any syntax. It’s possible to join SQL table and HQL table.