Apache Ranger Integrations

Solr is highly reliable, scalable, and fault-tolerant. It provides distributed indexing, replication, load-balanced querying and automated failover and recovery. Solr powers many of the largest internet sites around the world's search and navigation functions. Solr allows powerful matching capabilities, including phrases, wildcards and joins, grouping, and much more across any type of data. Solr has been proven on large scales all over the globe. Solr uses the same tools that you use to make application development easy. Solr ships with an intuitive, responsive administrative interface that makes it easy to manage your Solr instances. Want to know more about your instances? Solr publishes loads metric data via JMX. Solr is built on Apache Zookeeper, which has been proven to scale well. Solr is a complete solution for replication, distribution, rebalancing, and fault tolerance.

Apache Hive™, a data warehouse software, facilitates the reading, writing and management of large datasets that are stored in distributed storage using SQL. Structure can be projected onto existing data. Hive provides a command line tool and a JDBC driver to allow users to connect to it. Apache Hive is an Apache Software Foundation open-source project. It was previously a subproject to Apache® Hadoop®, but it has now become a top-level project. We encourage you to read about the project and share your knowledge. To execute traditional SQL queries, you must use the MapReduce Java API. Hive provides the SQL abstraction needed to integrate SQL-like query (HiveQL), into the underlying Java. This is in addition to the Java API that implements queries.

Apache Kafka®, is an open-source distributed streaming platform.

Unlike most data management systems, PHEMI Health DataLab is built with Privacy-by-Design principles, not as an add-on. This means privacy and data governance are built-in from the ground up, providing you with distinct advantages: Lets analysts work with data without breaching privacy guidelines Includes a comprehensive, extensible library of de-identification algorithms to hide, mask, truncate, group, and anonymize data. Creates dataset-specific or system-wide pseudonyms enabling linking and sharing of data without risking data leakage. Collects audit logs concerning not only what changes were made to the PHEMI system, but also data access patterns. Automatically generates human and machine-readable de- identification reports to meet your enterprise governance risk and compliance guidelines. Rather than a policy per data access point, PHEMI gives you the advantage of one central policy for all access patterns, whether Spark, ODBC, REST, export, and more

Apache HBase™, is used when you need random, real-time read/write access for your Big Data. This project aims to host very large tables, billions of rows and X million columns, on top of clusters of commodity hardware.

Apache Hadoop is a software library that allows distributed processing of large data sets across multiple computers. It uses simple programming models. It can scale from one server to thousands of machines and offer local computations and storage. Instead of relying on hardware to provide high-availability, it is designed to detect and manage failures at the application layer. This allows for highly-available services on top of a cluster computers that may be susceptible to failures.

Apache Storm is an open-source distributed realtime computing system that is free and open-source. Apache Storm makes it simple to process unbounded streams and data reliably, much like Hadoop did for batch processing. Apache Storm is easy to use with any programming language and is a lot fun! Apache Storm can be used for many purposes: realtime analytics and online machine learning. It can also be used with any programming language. Apache Storm is fast. A benchmark measured it at more than a million tuples per second per node. It is highly scalable, fault-tolerant and guarantees that your data will be processed. It is also easy to set up. Apache Storm can be integrated with the queueing and databases technologies you already use. Apache Storm topology processes streams of data in arbitrarily complex ways. It also partitions the streams between each stage of the computation as needed. Learn more in the tutorial.

The Knox API Gateway is a reverse proxy that allows pluggability to be made in areas such as policy enforcement, through providers, and backend services. Policy enforcement ranges from authentication/federation, authorization, audit, dispatch, hostmapping and content rewrite rules. A chain of providers is used to enforce policy. This chain is defined in the topology deployment description for each Apache Hadoop cluster gated with Knox. The topology deployment descriptor also defines the cluster definition. This provides the Knox Gateway with the layout and configuration of the cluster to allow for routing and translation between user-facing URLs and internals. Knox protects Apache Hadoop clusters with a single application context path that represents their set of REST APIs. This allows Knox Gateway to protect multiple clusters at once and provide a single endpoint for the REST API user.

The fundamental idea of YARN is to split up the functionalities of resource management and job scheduling/monitoring into separate daemons. The idea is to have a global ResourceManager, (RM), and a per-application ApplicationMaster, (AM). An application can be a single job, or a DAG (distributed array of jobs). The data-computation framework is formed by the NodeManager and the ResourceManager. The ResourceManager is the ultimate authority who arbitrates the allocation of resources among all applications in the system. The NodeManager is the per-machine framework agent who is responsible for containers, monitoring their resource usage (cpu, memory, disk, network) and reporting the same to the ResourceManager/Scheduler. The per-application ApplicationMaster, which is in essence a framework-specific library, is responsible for negotiating resources from ResourceManager and working with NodeManagers to execute and monitor tasks.