Sparkmagic is a set of tools for interactively working with remote Spark clusters in Jupyter notebooks. Sparkmagic interacts with remote Spark clusters through a REST server. Automatic visualization of SQL queries in the PySpark, Spark and SparkR kernels; use an easy visual interface to interactively construct visualizations, no code required. Ability to capture the output of SQL queries as Pandas dataframes to interact with other Python libraries (e.g. matplotlib). Send local files or dataframes to a remote cluster (e.g. sending pretrained local ML model straight to the Spark cluster) Authenticate to Livy via Basic Access authentication or via Kerberos.

Features

  • For running interactive sessions on Yarn
  • For running interactive sessions on Yarn or Kubernetes (only PySpark sessions are supported)
  • For running interactive sessions on Yarn or Kubernetes
  • The Sparkmagic project includes a set of magics for interactively running Spark code in multiple languages
  • Run Spark code in multiple languages against any remote Spark cluster through Livy
  • Automatic SparkContext (sc) and HiveContext (sqlContext) creation
  • Easily execute SparkSQL queries with the %%sql magic

Project Samples

sparkmagic Screenshot 1

Project Activity

See All Activity >

Categories

Operating System Kernels

License

MIT License

Follow sparkmagic

sparkmagic Web Site

Other Useful Business Software
Jscrambler: Pioneering Client-Side Protection Platform Icon
Jscrambler: Pioneering Client-Side Protection Platform

Jscrambler offers an exclusive blend of cutting-edge first-party JavaScript obfuscation and state-of-the-art third-party tag protection.

Jscrambler is the leader in Client-Side Protection and Compliance. We were the first to merge advanced polymorphic JavaScript obfuscation with fine-grained third-party tag protection in a unified Client-Side Protection and Compliance Platform. Our integrated solution ensures a robust defense against current and emerging client-side cyber threats, data leaks, and IP theft, empowering software development and digital teams to innovate securely. With Jscrambler, businesses adopt a unified, future-proof client-side security policy all while achieving compliance with emerging security standards including PCI DSS v4.0. Trusted by digital leaders worldwide, Jscrambler gives businesses the freedom to innovate securely.
Learn More
Rate This Project
Login To Rate This Project

User Reviews

Be the first to post a review of sparkmagic!

Additional Project Details

Programming Language

Python

Related Categories

Python Operating System Kernels

Registered

2023-12-18
Report inappropriate content