HDF/Nifi Cluster Architect

Location: Dallas, TX, United States
Date Posted: 04-25-2018
Position Role/Tile: HDF/Nifi Cluster Architect
Location: Dallas, TX​.


We are looking for a HDF/Nifi Architect to perform the HDF tasks below.  The Architect should also have HDP expertise.  This person will join another Client resource who will be focussing on the HDP efforts, but they will collaborate to finish all the tasks.  This person should have significant HDF Cluster installation, configuration, tuning and optimization experience are well as securing the cluster as our customers expect us to be the experts.  Start date is ASAP, preferably Monday, April 30, but May 7 is also an option.  The work is onsite in Dallas for 3 weeks.  

Cluster Design Guidance
  1. Provide verbal and written guidance to Customer to design the architecture and configuration of the HDP  cluster.
  2. Provide verbal and written guidance to Customer to design the architecture and configuration of the HDF  cluster.
  3. Provide verbal and written guidance to Customer to design HDP/HDF Security layout as per the HDP/HDF security configuration in the  Migration cluster.
  4. Provide verbal and written guidance to Customer to compose an architecture diagram of the desired  cluster. This Diagram shall outline which services are running on which nodes, as well as node CPU, memory and storage specifications.
 
Assumptions
  1. Customer resources will be available to participate in all necessary sessions and discussions with Client.
  2. The same Ambari instance will manage both the HDP and HDF nodes.
  3. HDP and HDF will be installed on separate nodes.
  4. This will be a brand-new cluster installation.
  5. HDP version will be 2.6.x. HDF version will be 3.x.
  6. Customer to compose all design documentation.
HDP and HDF Cluster Installation ( Cluster)
1.      Perform pre-installation validation
2.      Install Ambari 2.6.x.
3.      Install HDP 2.6.x with SmartSense on the  cluster.
4.      Install Spark on the  cluster.
5.      Install HDF 3.x on the  cluster.
6.      Perform installation configuration and validation, as per Client checklist.
  1. Configure Knox, Ranger and Kerberos in the HDP cluster as per the
Knox, Kerberos and Ranger configurations in the  Migration HDP cluster.
8.      Configure SSL as per the SSL configuration in the  Migration cluster:
  1. Ambari
b.      Ranger
c.      Knox
d.      Zeppelin
e.      Grafana
f.       Nifi
9.      Test security as per configuration
10.   Perform HDP and HDF configuration backup
10.  Compose installation runbook, as per Client standard format.
 
 
Assumptions
 
  1. All the hardware will be available for installation at the start of this task.
2.      There are 300 nodes in the  cluster.
3.      Customer will provide all necessary cluster access and configurations to the Client team prior to the start of this task.
4.      The goal in this task is to install the HDP/HDF  cluster to be identical in configuration to  Migration HDP/HDF cluster.
  1.  cluster hardware will meet all HDP and HDF installation
requirements prior to the start of this task.
6.      Customer resources will be available to assist Client as needed as part of the configuration activities (Active Directory, network security, etc.)
  1. Data ingestion is out of scope for this task.
8.      Disaster Recovery configuration is out of scope for this task.
9.      Kerberos will integrate with MIT KDC.
10.   Customer will provide SSL certificates to Client at the start of this task.
  1. IP tables, SELinux and firewalld will be disabled by Customer at
the start of this task.
Cluster Configuration Assistance
  1. Perform the following activities with Customer to copy HDP configurations from the  Migration cluster to the  cluster. This includes:
    1. HDFS configuration
    2. Yarn configuration
    3. Ambari configuration
    4. Hive and LLAP configuration
    5. Kafka configuration
    6. NiFi configuration
    7. Ranger policies
    8. Advise Customer on needed file permissions and Linux group memberships
  2. Copy HDFS data (up to 500TB) from the  Migration cluster to HDFS in  cluster.
  3. Copy and migrate up to one hundred (100) NiFi data flows from the  HDP 2.3.x stand-alone NiFi node to the  HDF NiFi cluster.
  1. Import existing NiFi data flow configuration files from the  HDP 2.6.x cluster to the  HDF Nifi cluster.
  2. Provide verbal configuration guidance to Customer with respect to Yarn queue configuration and tuning activities.
  3. Run Terasort and Teragen tests on the  cluster and assist with HDP and HDF configuration activities.
  4. Perform HDP and HDF issue resolution activities as part of Customer performed testing activities in the  cluster.
 
Assumptions
  1. HDF data flow migration is limited to one hundred (100) data flows.
  2. No new functionality will be added to the  cluster (that does not exist in the  Migration cluster).
  3. Customer is responsible for all testing and test scripts activities.
Client will perform issue resolution activities only.
  1. Customer IT will be performing the testing. Customer business testing is outside the scope of this task.
  2. Customer will configure appropriate file permissions and Linux group memberships as needed.


 


Central Business Solutions, Inc,
37600 Central Ct.
Suite #214
Newark, CA 94560.
Central Business Solutions, Inc(A Certified Minority Owned Organization)
Checkout our excellent assessment tool: http://www.skillexam.com/
Checkout our job board : http://www.job-360.net/
=====================================================
Central Business Solutions, Inc
37600 Central Court Suite 214 Newark CA, 94560
Phone: (510)-713-9900, 510-573-5500 Fax: (510)-740-3677
Web: http://www.cbsinfosys.com
=====================================================
or
this job portal is powered by CATS