General Readings

• https://reactjs.org/
• https://reactnative.dev/
• https://facebook.github.io/metro/
• https://expo.io/

Create App

Expo CLI Quickstart

Assuming that you have Node 12 LTS or greater installed, you can use npm to install the Expo CLI command line utility:

npm install -g expo-cli

Then run the following commands to create a new React Native project called “GettingStarted”:

expo init GettingStarted
cd GettingStarted
npm start # you can also use: expo start

React Native CLI Quickstart

npx react-native init GettingStarted

or create new project based on a template

npx react-native init GettingStarted --template react-native-template-typescript

Start App

Step 1: Start Metro

First, you will need to start Metro (Docs), the JavaScript bundler that ships with React Native. Metro “takes in an entry file and various options, and returns a single JavaScript file that includes all your code and its dependencies.”

npx react-native start

Step 2: Start your application

Let Metro Bundler run in its own terminal. Open a new terminal inside your React Native project folder. Run the following:

npx react-native run-ios

Installation

Install Python 3

Install Ansible

pip install ansible

Setup virtual machines

In this example, we use Vagrant. But using docker is also a good choice

Create Vagrantfile

Change IP Address and Hostname for each virtual machine

Vagrant.configure("2") do |config|
  config.vm.box = "hashicorp/bionic64"
  config.vm.network "public_network", ip: "172.16.2.191", bridge: "en0: Ethernet"

  config.vm.synced_folder "./data", "/DATA"
  config.vm.hostname = "host1"
end

Start virtual machine

vagrant up
vagrant ssh

Configure virtual machine

Install required programm sshpass for running ansible commands as root on target machines.

sudo apt-get install sshpass

Setup SSH permissions

Create ssh key

ssh-keygen -t rsa

Add id_rsa_<user>.pub to virtual machines into $HOME/.ssh/authorized_keys

Setup config files

etc/ansible/hosts

ansible1
ansible2
ansible3

etc/ansible/ansible.cfg

Create Ansible configuration file in current directory with this content:

[defaults]
interpreter_python = auto

Ansible searches for configuration files in the following order, processing the first file it finds and ignoring the rest:

1. $ANSIBLE_CONFIG if the environment variable is set.
2. ansible.cfg if it’s in the current directory.
3. ~/.ansible.cfg if it’s in the user’s home directory.
4. /etc/ansible/ansible.cfg, the default config file.

First Test

 ansible -i etc/ansible/hosts all -m ping -u vagrant

Set default Python version on Ubuntu

update-alternatives --install /usr/bin/python python /usr/bin/python3.6 1

Working with Playbooks

Create simple playbook

Create file httpd.yaml

---
- hosts: webservers
  remote_user: ansible
  tasks:
  - name: Ensure apache is installed and updated
    yum:
      name: httpd
      state: latest
    become: yes

Run playbook

ansible-playbook -i etc/ansible/hosts  httpd.yaml -kK

Troubleshooting

Allow SSH root login

$ sudo sed -i 's/#PermitRootLogin prohibit-password/PermitRootLogin yes/' /etc/ssh/sshd_config

Certifications: 1 2

Azure Solutions Architect Expert

Qualification

• Implementierung von Management- und Sicherheitslösungen
• Implementieren und Überwachen einer Azure-Infrastruktur
• Implementieren Sie Lösungen für Apps
• Implementierung und Verwaltung von Datenplattformen
• Entwurfsüberwachung
• Entwurf Identität und Sicherheit
• Entwerfen eines Datenspeichers
• Entwicklung von Geschäftskontinuität
• Infrastruktur entwerfen

Exams

• AZ-303: Microsoft Azure Architect Technologies
• AZ-304: Microsoft Azure Architect Design

Exams

• AZ-204: Developing Solutions for Microsoft Azure
• AZ-303: Microsoft Azure Architect Technologies

• AZ-304: Microsoft Azure Architect Design

Exam Preparation

• AZ-300 Exam Preparation: Technologies for Microsoft Azure Architects
• AZ-301 Exam Preparation: Designing a Microsoft Azure Architecture

Readings

• How to Become a Microsoft Certified Azure Solutions Architect

Learning Path

AZ-303 / AZ-304

Introduction

NestJS (just Nest from here on out), is a Node framework meant to build server-side applications. Not only is it a framework, but it is also a platform to meet many backend application needs, like writing APIs, building microservices, or doing real-time communications through web sockets.

Nest is also heavily influenced by Angular, and you will immediately find its concepts familiar. The creators of Nest strived to make the learning curve as small as possible, while still taking advantage of many higher level concepts such as modules, controllers, and dependency injection.

Installation

Install NodeJS

Download NodeJS from here and install as described here.

For example, on macOS using Homebrew

brew install node

Or download the package

curl "https://nodejs.org/dist/latest/node-${VERSION:-$(wget -qO- https://nodejs.org/dist/latest/ | sed -nE 's|.*>node-(.*)\.pkg</a>.*|\1|p')}.pkg" > "$HOME/Downloads/node-latest.pkg" && sudo installer -store -pkg "$HOME/Downloads/node-latest.pkg" -target "/"

Install NextJS

npm i -g @nestjs/cli

Create server App

Create new server App

nest new demo.server

Start server App

cd demo.server

npm run start:dev

Now open browser on http: localhost:3000

App Structure

main.ts

import { NestFactory } from '@nestjs/core';
import { AppModule } from './app.module';

async function bootstrap() {
  const app = await NestFactory.create(AppModule);
  await app.listen(3000);
}
bootstrap();

app.module.ts

import { Module } from '@nestjs/common';
import { AppController } from './app.controller';
import { AppService } from './app.service';

@Module({
  imports: [],
  controllers: [AppController],
  providers: [AppService],
})
export class AppModule {}

app.controller.ts

import { Controller, Get } from '@nestjs/common';
import { AppService } from './app.service';

@Controller()
export class AppController {
  constructor(private readonly appService: AppService) {}

  @Get()
  getHello(): string {
    return this.appService.getHello();
  }
}

app.service.ts

import { Injectable } from '@nestjs/common';

@Injectable()
export class AppService {
  getHello(): string {
    return 'Hello World!';
  }
}

Add Functionality

Create service and controller

nest g service missions
nest g controller missions

Modify mission service

@Injectable()
export class MissionsService {
  missions: Mission[] = [
    { id: 1, title: 'Rescue cat stuck in asteroid', reward: 500, active: true, },
    { id: 2, title: 'Escort Royal Fleet', reward: 5000, active: true, },
    { id: 3, title: 'Pirates attacking the station', reward: 2500, active: false, },
  ];

  async getMissions(): Promise<Mission[]> {
    return this.missions;
  }
}

Modify mission controller

@Controller('missions')
export class MissionsController {
  constructor(private missionsService: MissionsService) {}

  @Get()
  getMissions() {
    return this.missionsService.getMissions();
  }
}

Open in browser: http://localhost:3000

Create Frontend App

Create new frontend App

ionic start demo.frontend sidemenu

Working with Database and TypeORM

Create / Sync database with schema

Add command to package.json

"scripts": {
    "typeorm": "ts-node -r tsconfig-paths/register ./node_modules/typeorm/cli.js"
}

npm run typeorm schema:sync

TypeORM Commands

schema:sync         
Synchronizes your entities with database schema. It runs schema update queries on all connections you have. To run update queries on a concrete connection use -c option.
schema:log          
Shows sql to be executed by schema:sync command. It shows sql log only for your default connection. To run update queries on a concrete connection use -c option.
schema:drop         
Drops all tables in the database on your default connection. To drop table of a concrete connection's database use -c option.
query               
Executes given SQL query on a default connection. Specify connection name to run query on a specific connection.
entity:create       
Generates a new entity.
subscriber:create   
Generates a new subscriber.
migration:create    
Creates a new migration file. [Aliase: migrations:create]
migration:generate  Generates a new migration file with sql needs to be executed to update schema. [Aliase: migrations:generate]
migration:run       
Runs all pending migrations. [Aliase: migrations:run]
migration:show      
Show all migrations and whether they have been run or not
migration:revert    
Reverts last executed migration. [Aliase: migrations:revert]
version             
Prints TypeORM version this project uses.
cache:clear         
Clears all data stored in query runner cache.
init                
Generates initial TypeORM project structure. If name specified then creates files inside directory called as name. If its not specified then creates files inside current directory.

Additional readings

• Full-Stack TypeScript with Ionic, Angular, and NestJS Part 1
• Nest.js Tutorial: Build your First REST API CRUD App with TypeORM/
• NestJS TypeORM Pagination – Step By Step Guide/
• Getting started with NestJS & TypeORM (+bonus NestJS Admin)
• TypeORM With NEST JS Basic Tutorial
• Learning nestjs I
• NestJS, TypeORM and PostgreSQL — full example development and project setup working with database migrations.

Installation Overview

NodeJS

Angular

Ionic

React / ReactJS

React Native

Stencil

npm init stencil
npm install --save-exact @stencil/core@latest 
npm install --save-dev @types/jest@26.0.12 jest@26.4.2 jest-cli@26.4.2 
npm install --save-dev @types/puppeteer@3.0.1 puppeteer@5.2.1
npm test
npm start

Gatsby

NextJS

NestJS

Usefull Libraries

More to read

• https://medium.com/better-programming/heres-a-list-of-top-css-front-end-libraries-eb0330c04148

Introduction

From Wikipedia, the free encyclopedia:

Rust is a multi-paradigm programming language focused on performance and safety, especially safe concurrency. Rust is syntactically similar to C++, and provides memory safety without using garbage collection.

Rust was originally designed by Graydon Hoare at Mozilla Research.

It has gained increasing use in industry and is now Microsoft’s language of choice for secure and safety-critical software components.

Rust has been the “most loved programming language” in the Stack Overflow Developer Survey every year since 2016.

Rust could be used in different areas

• Web Assembly
• Embedded Programming
• Command Line Utilitiesls
• Operating System Developmend (see Redox OS)

Read more

• On Stack Overflow : What is Rust and why is it so popular?
• Awesome Lists of Rust Examples:
  • rust-unofficial/awesome-rust
  • not-yet-awesome-rust/not-yet-awesome-rust
  • phoenixenero/awesome-rust

Installation

Rustup

Download install script and run it

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

Modify .bashrc to add Rust path to PATH

source $HOME/.cargo/env

Other way to install Rust on MacOS

brew install rust

Create and run your first App

Create the app

$ cargo new hello_world
$ cd hello_world

Show folder structure

$ tree .
.
├── Cargo.lock
├── Cargo.toml
└── src
    └── main.rs

1 directory, 3 files

Show main source file

fn main() {
    println!("Hello, world!");
}

Build your app

$ cargo build
Compiling hello_world v0.1.0 (.../hello_world)
Finished dev [unoptimized + debuginfo] target(s) in 6.32s

Or build a production ready version

$ cargo build --release<br>Finished release [optimized] target(s) in 0.19s

Run your app

$ cargo run
Finished dev [unoptimized + debuginfo] target(s) in 0.04s
Running `target/debug/hello_world
Hello, world!

Add functionality to your app

Add Dependencies

Let’s add a dependency to our application. You can find all sorts of libraries on crates.io, the package registry for Rust. In Rust, we often refer to packages as “crates.”

In this project, we’ll use a crate called ferris-says.

In our Cargo.toml file we’ll add this information (that we got from the crate page):

[dependencies]
ferris-says = "0.1"

Modify main source

Now let’s write a small application with our new dependency. In our main.rs, add the following code:

use ferris_says::say; // from the previous step
use std::io::{stdout, BufWriter};

fn main() {
    let stdout = stdout();
    let message = String::from("Hello fellow Rustaceans!");
    let width = message.chars().count();

    let mut writer = BufWriter::new(stdout.lock());
    say(message.as_bytes(), width, &mut writer).unwrap();
}

Run App

$ cargo build
    Updating crates.io index
  Downloaded object v0.20.0
  Downloaded textwrap v0.11.0
  Downloaded adler v0.2.3
  Downloaded ansi_term v0.11.0
  Downloaded miniz_oxide v0.4.1
  Downloaded gimli v0.22.0
  Downloaded strsim v0.8.0
  Downloaded error-chain v0.10.0
  Downloaded vec_map v0.8.2
  Downloaded clap v2.33.3
  Downloaded smallvec v0.4.5
  Downloaded ferris-says v0.1.2
  Downloaded backtrace v0.3.50
  Downloaded rustc-demangle v0.1.16
  Downloaded addr2line v0.13.0
  Downloaded 15 crates (1.4 MB) in 1.65s
   Compiling libc v0.2.76
   Compiling bitflags v1.2.1
   Compiling gimli v0.22.0
   Compiling adler v0.2.3
   Compiling rustc-demangle v0.1.16
   Compiling unicode-width v0.1.8
   Compiling object v0.20.0
   Compiling cfg-if v0.1.10
   Compiling strsim v0.8.0
   Compiling vec_map v0.8.2
   Compiling ansi_term v0.11.0
   Compiling smallvec v0.4.5
   Compiling textwrap v0.11.0
   Compiling miniz_oxide v0.4.1
   Compiling addr2line v0.13.0
   Compiling atty v0.2.14
   Compiling backtrace v0.3.50
   Compiling clap v2.33.3
   Compiling error-chain v0.10.0
   Compiling ferris-says v0.1.2
   Compiling hello_world v0.1.0 (.../hello_world)
    Finished dev [unoptimized + debuginfo] target(s) in 14.73s

Run your app

$ cargo run
     Finished dev [unoptimized + debuginfo] target(s) in 0.14s
       Running`target/debug/hello_world`

----------------------------
| Hello fellow Rustaceans! |
----------------------------
              \
               \
                  _~^~^~_
              \) /  o o  \ (/
                '_   -   _'
                / '-----' \

Next steps

Read Getting Started on rust homepage

Explore Learn Rust

Introduction

Problem

Like any other program, Azure Databricks notebooks should be tested automatically to ensure code quality.

Using standard Python Test Tools is not easy because these tools are based on Python files in a file system. And a notebook doesn’t correspond to a Python file.

Solution

To enable automated testing with unittest (documentation), we proceed as follows:

• Create a test class that contains all the tests you want
• Execution of all defined tests

Create Notebook with the Code

We will create a simple Notebook for our test.

This notebook will implement a simple calculator, so that we can test the basic calculator operations like add ad multiply.

Create a new Notebook with the name Calculator:

class Calculator:

	def __init__(self, x = 10, y = 8):
		self.x = x
		self.y = y
		
	def add(self, x = None, y = None):
		if x == None: x = self.x
		if y == None: y = self.y			
          
		return x+y

	def subtract(self, x = None, y = None):
		if x == None: x = self.x
		if y == None: y = self.y	
          
		return x-y

	def multiply(self, x = None, y = None):
		if x == None: x = self.x
		if y == None: y = self.y			
          
		return x*y

	def divide(self, x = None, y = None):
		if x == None: x = self.x
		if y == None: y = self.y			
          
		if y == 0:
			raise ValueError('cannot divide by zero')
		else:
			return x/y

The notebook should look like this

To use this class, write the following lines:

c = Calculator()
print(c.add(20, 10), c.subtract(20, 10), c.multiply(20, 10), c.divide(20, 10))

Create Notebook with the Tests

Create a new Notebook in the same folder with the name Calculator.Tests.

The name is not important, but it is convenient to name the test program like the program to be tested with the suffix ‘Tests’.

Create the first command to import the Calculator Notebook

%run "./Calculator"

Create the Test Class

import unittest

class CalculatorTests(unittest.TestCase):
  
  @classmethod
  def setUpClass(cls):
    cls.app = Calculator()

  def setUp(self):
    # print("this is setup for every method")
    pass

  def test_add(self):
    self.assertEqual(self.app.add(10,5), 15, )

  def test_subtract(self):
    self.assertEqual(self.app.subtract(10,5), 5)
    self.assertNotEqual(self.app.subtract(10,2), 4)

  def test_multiply(self):
    self.assertEqual(self.app.multiply(10,5), 50)

  def tearDown(self):
    # print("teardown for every method")
    pass

  @classmethod
  def tearDownClass(cls):
    # print("this is teardown class")
    pass

Create the code to run the tests

suite =  unittest.TestLoader().loadTestsFromTestCase(CalculatorTests)
unittest.TextTestRunner(verbosity=2).run(suite)

Databricks CLI

Export all Notebooks

databricks workspace list | ForEach { databricks workspace export_dir /$_ $_ }

Troubleshooting
Problem

Error in SQL statement: AnalysisException: Can not create the managed table('`demo`'). The associated location('dbfs:/user/hive/warehouse/demo') already exists.;

Solution

dbutils.fs.rm("dbfs:/user/hive/warehouse/demo/", true)

TL;DR

Don’t want to read the post, then explore this Azure Notebook

Requirements

Define needed moduls and functions

from datetime import datetime

import pyspark.sql.functions as F

Create DataFrame for this post:

df = spark.sql("select * from diamonds")
df.show()

Working with Widgets

Default Widgets

dbutils.widgets.removeAll()

dbutils.widgets.text("W1", "1", "Text")
dbutils.widgets.combobox("W2", "3", [str(x) for x in range(1, 10)], "Combobox")
dbutils.widgets.dropdown("W3", "4", [str(x) for x in range(1, 10)], "Dropdown")

Multiselect Widgets

list = [ f"Square of {x} is {x*x}" for x in range(1, 10)]
dbutils.widgets.multiselect("W4", list[0], list, "Multi-Select")

Monitor the changes when selection values

print("Selection: ", dbutils.widgets.get("W4"))
print("Current Time =", datetime.now().strftime("%H:%M:%S"))

Filter Query by widgets

Prepare widgets

dbutils.widgets.removeAll()

df = spark.sql("select * from diamonds")

vals = [ str(x[0]) for x in df.select("cut").orderBy("cut").distinct().collect() ]
dbutils.widgets.dropdown("Cuts", vals[0], vals)

vals = [ str(x[0]) for x in df.select("carat").orderBy("carat").distinct().collect() ]
dbutils.widgets.dropdown("Carat", vals[0], vals)

Now, change some values

filter_cut = dbutils.widgets.get("Cuts")
df=spark.sql(f"select * from diamonds where cut='{filter_cut}'").show()

Arbeiten mit dem Header

Schreibweise ändern

Grossschreibung/Kleinschreibung/CamelCase

= Table.TransformColumnNames(RenameColumns, Text.Upper)
= Table.TransformColumnNames(RenameColumns, Text.Lower)
= Table.TransformColumnNames(RenameColumns, Text.Proper)

Bestimmte Zeichen entfernen (z. B. _)

= Table.TransformColumnNames(Source,each Text.Proper(Replacer.ReplaceText( _ , "_", " ")))

Aufteilen in Worte

= Table.TransformColumnNames(Source, each Text.Combine(
                    Splitter.SplitTextByCharacterTransition({"a".."z"},{"A".."Z"})(_), " "))

Als Function

(columnNames as text) =>
let 
    splitColumn = Splitter.SplitTextByCharacterTransition({"a".."z"}, {"A".."Z"})(columnNames)
in
    Text.Combine(splitColumn, " ")

Daten transformieren

Zeilen gruppenweise pivotieren

Aufgabenstellung

Werden Daten angeliefert, in denen das Gruppierungsmerkmal in den Zeilen vorhanden ist und somit mehrere Zeilen pro Datensatz vorhanden, wünscht man sich meist eine kompaktere Darstellung.

Für den Datensatz mit dem Wert “Daten 1” werden also vier Zeilen mit unterschiedlichen Werten in GRUPPE und Wert angeliefert.

Gewünscht ist aber eine kompaktere Darstellung mit den vorhandenen Gruppen als Spalten:

Die Aufgabenstellung ist somit die Umwandlung der angelieferten Daten:

Eine Beispieldatei liegt hier. Das Endergebnis liegt hier. Speichern sie beide Datein im Order C: \TMP, dann stimmt der Verweis in Query.xlsx auf die Daten Daten.xlsx.

Schritt 1: Daten vorbereiten

Im ersten Schritt erstellen wir eine neue Excel-Daten und greifen auf die vorbereiteten Daten über Power Query zu.

Wählen Sie dazu im Register Daten den Eintrag Daten abrufen / Aus Datei / Aus Arbeitsmappe und selektieren sie die gewünschte Datei:

Eine Beispieldatei liegt hier.

Ein Klick auf Importieren führt sie zum Navigation

Sie sehen im Navigator 3 verschiedenen Elemente:

• DATEN: die intelligente Tabelle im Tabellenblatt. Diese beinhaltet genau die gewünschten Daten
• ERGBNIS: die intelligente Tabelle, die das zu erwartende Ergbnis beinhaltet
• Beispieldaten: das Tabellenblatt mit den beiden intelligenten Tabellen

Selektieren sie das Element DATEN und klicken sie auf Daten transformieren.

Schritt 2: Spalte pivotieren

Wir wollen die Werte der Spalte GRUPPE als neue Spalten erhalten.

Hier klicken sie auf die Spalte GRUPPE und wählen dein Eintrag Spalte pivotieren im Register Transformieren / Beliebige Spalte:

Die Werte für die neuen Spalten (Gruppe 1, Gruppe 2 , ..) kommen aus der Spalte WERT (Wert 11, Wert 12, ..):

Wir wollen die Werte selbst übernehmen und keine (wie bei Pivottabellen meist üblich) Aggregierungsfunktion verwenden (Summe, Max, Anzahl, ..).

Klicken sie hierzu auf Erweiterte Optionen und selektieren sie den Eintrag Nicht aggregieren:

Anschließen klicken sie auf OK:

Zum Abschluss beenden wir den Power Query Editor: