Data Engineering

Map vs FlatMap (Scala) val data = Seq ( "his is sample string" , "one more sample string" , "third line" , "line" , "this is one more line" ) data: Seq[String] = List(his is sample string, one more sample string, third line, line, this is one more line) val df = spark . sparkContext . parallelize ( data ) df: org.apache.spark.rdd.RDD[String] = ParallelCollectionRDD[7] at parallelize at command-2343346390787948:1 df . map ( x => ( x , x . length )). collect () res0: Array[(String, Int)] = Array((his is sample string,20), (one more sample string,22), (third line,10), (line,4), (this is one more line,21)) df . map ( x => x . split ( " " )). collect () res1: Array[Array[String]] = Array(Array(his, is, sample, string), Array(one, more,...

SSH key setup for Gitlab on Windows On Windows 10, Windows 8.1 and Windows 7 The easiest way to install Git and the SSH client on Windows 8.1 and Windows 7 is Git for Windows . It provides a Bash emulation (Git Bash) used for running Git from the command line and the ssh-keygen command that is useful to create SSH keys as you'll learn below. GitLab supports RSA, DSA, ECDSA, and ED25519 keys. Their difference lies on the signing algorithm, and some of them have advantages over the others. ED25519 SSH keys Following best practices , you should always favor ED25519 SSH keys, since they are more secure and have better performance over the other types. They were introduced in OpenSSH 6.5, so any modern OS should include the option to create them. If for any reason your OS or the GitLab instance you interact with doesn't support this, you can fallback to RSA. RSA SSH keys RSA keys are the most common ones and therefore the most compatible with servers th...

Revoke a Token This section describes how to revoke personal access tokens using the Databricks UI. You can also generate and revoke access tokens using the  Token API . Click the user profile icon   in the upper right corner of your Databricks workspace. Click  User Settings . Go to the  Access Tokens  tab. Click  x  for the token you want to revoke. On the Revoke Token dialog, click the  Revoke Token  button. Use tokens for API authentication In this section: Store token in .netrc file and use in  curl Pass token to Bearer authentication Store token in .netrc file and use in  curl Create a  .netrc  file with  machine ,  login , and  password  properties: machine <databricks-instance> login token password <personal-access-token-value> Replace  <databricks-instance>  with the <account>.cloud.databricks.com domain name of y...

Generating Databricks token Token-based authentication is enabled by default for all Databricks accounts launched after January 2018. If it is disabled, your administrator must enable it before you can perform the tasks described in this topic. See  Enable Token-based Authentication . Generate a token This section describes how to generate a personal access token in the Databricks UI. You can also generate and revoke tokens using the  Token API . Click the user profile icon   in the upper right corner of your Databricks workspace. Click  User Settings . Go to the  Access Tokens  tab. Click the  Generate New Token  button. Optionally enter a description (comment) and expiration period. Click the  Generate  button. Copy the generated token and store in a secure location. Revoke a token This section describes how to revoke personal access tokens using the Databricks UI. You can al...