Position Role/Tile: CDP Architect
Location: Remote

Client Data Platform (CDP-DC 7.x) (HBASE and related components)

Install and Configure Client Data Platform (CDP-DC) v7.x (Approximately 80 hours)

• Architecture Review
o Discuss key points that will dictate deployment decisions
o Review pre-engagement documentation
o Determine software layout for solution
o Work with Sandia security team to understand Authority to Operate (ATO) implications.
o Discuss sizing and data requirements
• Pre-Installation
o Determine installation type
o Validate environment readiness
o Install Client Manager agents
• CDP-DC Installation and Deployment
o Use Client Manager to deploy to the agreed upon architecture
o High-availability installation for all necessary services (HBase)
o All included CDP-DC 7.x applications are to be installed, including H389 Directory Server, Spark, HBase, Phoenix, Knox, Client Manager, Mapreduce, Atlas, Yarn, Zookeeper, Hue and Ranger.
o Integrate and configure with existing Kerberos and 389 DIRECTORY SERVER
o Install Phoenix (SQL) server for SSL to HBase
o Test integration of Splunk and DBConnect with Phoenix/HBase with SQL over SSL (assuming client has licensing to Splunk/DBConnect products)
o Demonstrate querying HBase/Phoenix from Hue over SSL
o Integrate with Pure Storage storage using S3 API
o Apache Ranger HBase plug-in (for CSP broker authorization)
o Enable multi-tiered storage with HDFS
• CDP-DC High Level Overview
o Provide overview of each subsystem
o Ensure smoke test worked on each subsystem
o Shut down and restart all services
• CDP-DC Cluster Tuning
o Review cluster tuning parameters
o Run TestDFSIO
o Run Teragen and Terasort scripts
o Tune Yarn
• Cluster Management Overview
o Review add/remove Nodes / Services
• CDP-DC Configurations Backup
o Backup important site xml files
o Backup Client Manager server configurations
o Backup Client Manager agent configurations
o Backup Postgres Database
• Cluster Monitoring
o Provide Client Manager alert overview
o Provide Client Manager metrics overview
• Knowledge Transfer and Plan Next Steps
o Knowledge transfer with End Customer team on activities performed and best practices in the form of information sharing
o Conduct client meeting to discuss next engagement and use cases
o Document findings, thoughts, impressions, and provide knowledge transfer (written report)
o Provide recommendations on areas not covered for personal self-paced learning, future Client training, or other avenues
o Analyze performance metrics for HBase with Pure Storage
• Transition to Support
o Update Client customer support team on all activities performed

Documentation:

Operational Runbook Document based on a Client-developed template. An operation run book contains foundational information required to successfully maintain and operate the installed CDP-DC Cluster. It covers the below items in detail:

• Architectural overview and configuration of the existing cluster
• Day to day operational procedures for managing & monitoring the cluster
• Backup & recovery procedures
• Long-term operational procedures needed for running and expanding clusters

CFM (Client Flow Management) (Approximately 40 hours) (Nifi and Related Components)

CDF Cluster Install and Data Flow Implementation

Phase 1: Setup:
• Finalize requirements including data sources, transformation, destination and success criteria
• Validate environment readiness and pre-requisites before installation
• Overall solution including ingest design pattern, transformation, and analytics
• CDF Cluster design including applicable CDF components
• Installation and verification of CDF 3.0 Cluster per design
• Initial tuning of CDF cluster based on best practices
• Integrate with existing 389 Directory and Kerberos [assumes existing 389 DIRECTORY SERVER/KERB + installer needs access to OU]
• Configure data in motion security / SSL using customer provided CA/Security certificates
• Performance baseline of CDF/NiFi cluster using standard scripts
• Designed for redundancy / failover
• Installation and config of NiFi registry

Phase 2: Testing and Evaluation:

• Provide High level overview of CDF/NiFi Cluster
• Enable CDF service monitoring and alerting mechanisms
• Show benefits of code-driven data-engineering practices
• Test saving and versioning processor groups in the Nifi registry
• Test ability to kick off work in Nifi via a REST web-service
• Execute test cases, troubleshoot and resolve issues

Phase 3: Data Ingest and Flow Design:

• Design, build and deploy Data Flows using CDF components and out of the box NiFi processors
• Configure ingest from rabbitmq server

Deliverables:

• Documentation and provide knowledge transfer (written report)
• Provide recommendations on areas not covered for personal self-paced learning, future Client training, or other avenues
• Provide generic example code for custom Nifi processor in Java

CSA (Client Streaming Analytics) (Approximately 40 hours) (Flink and related components)

Phase 1: Setup:
• Finalize requirements including data sources, transformation, destination and success criteria
• Validate environment readiness and pre-requisites before installation
• Installation and verification of CSA per design
• Integrate with CDF cluster
• Initial tuning of CSA based on best practices
• Integrate with existing Kerberos and 389 DIRECTORY SERVER
• Configure CSA with user client provided SSL/CA Security Certificates
• Configure CSA for resilience to failure base on snapshot/restore
• Install Druid on a host and integrate with CSP

Phase 2: Testing and Evaluation:

• Provide High level overview of CSA with CDF Cluster
• Enable CDF service monitoring and alerting mechanisms
• Execute test cases, troubleshoot and resolve issues

Phase 3: Integrate with CSP:

• Design and build connections to CSP
• Ensure reading and writing from CSA to Kafka topics

Phase 4: Integrate with HBase:

• Design and build connections to HBase
• Ensure reading from CSA to HBase
• Read/Write to/from HBase

Deliverables:

• Documentation and provide knowledge transfer (written report)
• Provide recommendations on areas not covered for personal self-paced learning, future Client training, or other avenues

CSP (Client Streams Processing) (Approximately 40 hours) (Kafka and Related Components)

Phase 1: Setup:
• Finalize requirements including data sources, transformation, destination and success criteria
• Validate environment readiness and pre-requisites before installation
• Installation and verification of CSP per design
• Installation and verification of Apache Druid per design
• Integrate with CDF cluster
• Initial tuning of CSP based on best practices
• Configure CSP with user client provided SSL/CA Security Certificates
• Configure CSP brokers for HA

Phase 2: Testing and Evaluation:

• Provide High level overview of CSP with CDF Cluster
• Enable CDF service monitoring and alerting mechanisms
• Execute test cases, troubleshoot and resolve issues

Phase 3: Integrate with CFM:

• Design and build connections to CFM
• Ensure reading and writing from CFM to Kafka topics

Deliverables:

• Documentation and provide knowledge transfer (written report)
• Provide recommendations on areas not covered for personal self-paced learning, future Client training, or other avenues
• Demonstrate read/write of CSP brokers with generic syslog data

Use Case Development - (40 - 120 hours)

Phase 1: Use Case Discovery Session:
• Work with Client PS to Client desired use cases
• Scope and estimate engineering required for each of the discovered use cases
• Prioritize and selection of use case(s) based on the importance to the cyber SNL team

Phase 2: Implementation and Testing:

• Build and implement use cases selected use cases
• Execute test cases, troubleshoot and resolve issues

Deliverables:

• Documentation and provide knowledge transfer (written report)
• Provide recommendations on areas not covered for personal self-paced learning, future Client training, or other avenues

Sandia Cyber Pilot Reference Architecture:

POC Success Criteria:

1. Everything will be installed on-prem on SNL's Azure Stack on VMs.
2. CDP-DC Operational DB (HBase / Phoenix) will be HA.
3. Splunk can query Operational DB (HBase+Phoenix [JDBC] ) in CDP-CD.
NOTE: Splunk JDBC connector is required - licensed by Splunk.
4. CSA (Flink) will be available to query live streams from stored in CSP (Kafka).
5. Services will be delivered remotely.

Central Business Solutions, Inc,
37600 Central Ct.
Suite #214
Newark, CA 94560.

Jobs at Central Business Solutions, Inc

404 Not Found