https://cloud.google.com/certification/data-engineer

A Professional Data Engineer enables data-driven decision making by collecting, transforming, and publishing data. A Data Engineer should be able to design, build, operationalize, secure, and monitor data processing systems with a particular emphasis on security and compliance; scalability and efficiency; reliability and fidelity; and flexibility and portability. A Data Engineer should also be able to leverage, deploy, and continuously train pre-existing machine learning models.

How Google Invented An Amazing Datacenter Network Only They Could Create

http://highscalability.com/blog/2015/8/10/how-google-invented-an-amazing-datacenter-network-only-they.html

The Datacenter as a Computer: An Introduction to the Design of Warehouse-Scale Machines https://www.morganclaypool.com/doi/abs/10.2200/S00193ED1V01Y200905CAC006

What is Google Cloud Dataproc? https://cloud.google.com/dataproc/docs/concepts/overview

Security Configuration https://cloud.google.com/dataproc/docs/concepts/configuring-clusters/security

Streaming 101: The world beyond batch https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101

Dataflow: A Unified Model for Batch and Streaming Data Processing https://youtu.be/3UfZN59Nsk8

After Lambda: Exactly-once processing in Google Cloud Dataflow, Part 1 https://cloud.google.com/blog/products/gcp/after-lambda-exactly-once-processing-in-google-cloud-dataflow-part-1

Cloud Data Fusion https://cloud.google.com/data-fusion/

Cloud Data Fusion is a fully managed, cloud-native data integration service that helps users efficiently build and manage ETL/ELT data pipelines. With a graphical interface and a broad open source library of preconfigured connectors and transformations, Cloud Data Fusion shifts an organization’s focus away from code and integration to insights and action.

The biggest barrier to enterprise data analysis and machine learning is data integration.It is not uncommon to see companies struggle to get their data in one place, move it around, transform it, and make sense out of it.

Companies struggle to get their data in one place, move, transform, and make sense out of it. Cloud Data Fusion shifts an organization’s focus away from code and integration to insights and action.

What is Cloud Data Fusion? https://cloud.google.com/data-fusion/docs/concepts/overview

A Cloud Data Fusion instance is a unique deployment of Cloud Data Fusion. To get started with Cloud Data Fusion, you create a Cloud Data Fusion instance through the Cloud Console.

Cloud Data Fusion creates ephemeral execution environments to run pipelines when you manually run your pipelines or when pipelines run through a time schedule or a pipeline state trigger. Cloud Data Fusion supports Dataproc as an execution environment, in which you can choose to run pipelines as MapReduce, Spark, or Spark Streaming programs. Cloud Data Fusion provisions an ephemeral Dataproc cluster in your customer project at the beginning of a pipeline run, executes the pipeline using MapReduce or Spark in the cluster, and then deletes the cluster after the pipeline execution is complete.

A pipeline is a way to visually design data and control flows to extract, transform, blend, aggregate, and load data from various on-premises and cloud data sources. Building pipelines allows you to create complex data processing workflows that can help you solve data ingestion, integration, and migration problems. You can use Cloud Data Fusion to build both batch and real-time pipelines, depending on your needs.

A plugin is a customizable module that can be used to extend the capabilities of Cloud Data Fusion. Cloud Data Fusion provides plugins for sources, transforms, aggregates, sinks, error collectors, alert publishers, actions, and post-run actions.

A compute profile specifies how and where a pipeline is executed. A profile encapsulates any information required to set up and delete the pipeline's physical execution environment.

Architecture components https://cloud.google.com/data-fusion/docs/concepts/architecture

Build natural and rich conversational experiences https://dialogflow.com/

Give users new ways to interact with your product by building engaging voice and text-based conversational interfaces, such as voice apps and chatbots, powered by AI. Connect with users on your website, mobile app, the Google Assistant, Amazon Alexa, Facebook Messenger, and other popular platforms and devices.

Dialogflow documentation https://cloud.google.com/dialogflow/docs/

Dialogflow is a natural language understanding platform that makes it easy to design and integrate a conversational user interface into your mobile app, web application, device, bot, interactive voice response system, and so on. Using Dialogflow, you can provide new and engaging ways for users to interact with your product.

Dialogflow can analyze multiple types of input from your customers, including text or audio inputs (like from a phone or voice recording). It can also respond to your customers in a couple of ways, either through text or with synthetic speech.

Building Amazing Apps With Cloud Firestore (Cloud Next '19) https://youtu.be/ah5tQ7yOh2s

Pipeline fundamentals for the Apache Beam SDKs https://cloud.google.com/dataflow/docs/guides/beam-creating-a-pipeline

BigQuery

The 8 Requirements of Real-Time Stream Processing http://cs.brown.edu/~ugur/8rulesSigRec.pdf

Migrating On-Premises Hadoop Infrastructure to Google Cloud Platform https://cloud.google.com/solutions/migration/hadoop/hadoop-gcp-migration-overview

Migrating from an on-premises Hadoop solution to GCP requires a shift in approach. A typical on-premises Hadoop system consists of a monolithic cluster that supports many workloads, often across multiple business areas. As a result, the system becomes more complex over time and can require administrators to make compromises to get everything working in the monolithic cluster. When you move your Hadoop system to GCP, you can reduce the administrative complexity. However, to get that simplification and to get the most efficient processing in GCP with the minimal cost, you need to rethink how to structure your data and jobs.