Power Query | Cookbook

Arbeiten mit dem Header

Schreibweise ändern

Grossschreibung/Kleinschreibung/CamelCase

= Table.TransformColumnNames(RenameColumns, Text.Upper)
= Table.TransformColumnNames(RenameColumns, Text.Lower)
= Table.TransformColumnNames(RenameColumns, Text.Proper)

Bestimmte Zeichen entfernen (z. B. _)

= Table.TransformColumnNames(Source,each Text.Proper(Replacer.ReplaceText( _ , "_", " ")))

Aufteilen in Worte

= Table.TransformColumnNames(Source, each Text.Combine(
                    Splitter.SplitTextByCharacterTransition({"a".."z"},{"A".."Z"})(_), " "))

Als Function

(columnNames as text) =>
let 
    splitColumn = Splitter.SplitTextByCharacterTransition({"a".."z"}, {"A".."Z"})(columnNames)
in
    Text.Combine(splitColumn, " ")

Daten transformieren

Zeilen gruppenweise pivotieren

Aufgabenstellung

Werden Daten angeliefert, in denen das Gruppierungsmerkmal in den Zeilen vorhanden ist und somit mehrere Zeilen pro Datensatz vorhanden, wünscht man sich meist eine kompaktere Darstellung.

Für den Datensatz mit dem Wert “Daten 1” werden also vier Zeilen mit unterschiedlichen Werten in GRUPPE und Wert angeliefert.

Gewünscht ist aber eine kompaktere Darstellung mit den vorhandenen Gruppen als Spalten:

Die Aufgabenstellung ist somit die Umwandlung der angelieferten Daten:

Eine Beispieldatei liegt hier. Das Endergebnis liegt hier. Speichern sie beide Datein im Order C: \TMP, dann stimmt der Verweis in Query.xlsx auf die Daten Daten.xlsx.

Schritt 1: Daten vorbereiten

Im ersten Schritt erstellen wir eine neue Excel-Daten und greifen auf die vorbereiteten Daten über Power Query zu.

Wählen Sie dazu im Register Daten den Eintrag Daten abrufen / Aus Datei / Aus Arbeitsmappe und selektieren sie die gewünschte Datei:

Eine Beispieldatei liegt hier.

Ein Klick auf Importieren führt sie zum Navigation

Sie sehen im Navigator 3 verschiedenen Elemente:

  • DATEN: die intelligente Tabelle im Tabellenblatt. Diese beinhaltet genau die gewünschten Daten
  • ERGBNIS: die intelligente Tabelle, die das zu erwartende Ergbnis beinhaltet
  • Beispieldaten: das Tabellenblatt mit den beiden intelligenten Tabellen

Selektieren sie das Element DATEN und klicken sie auf Daten transformieren.

Schritt 2: Spalte pivotieren

Wir wollen die Werte der Spalte GRUPPE als neue Spalten erhalten.

Hier klicken sie auf die Spalte GRUPPE und wählen dein Eintrag Spalte pivotieren im Register Transformieren / Beliebige Spalte:

Die Werte für die neuen Spalten (Gruppe 1, Gruppe 2 , ..) kommen aus der Spalte WERT (Wert 11, Wert 12, ..):

Wir wollen die Werte selbst übernehmen und keine (wie bei Pivottabellen meist üblich) Aggregierungsfunktion verwenden (Summe, Max, Anzahl, ..).

Klicken sie hierzu auf Erweiterte Optionen und selektieren sie den Eintrag Nicht aggregieren:

Anschließen klicken sie auf OK:

Zum Abschluss beenden wir den Power Query Editor:

Power Query | Cookbook

Arbeiten mit dem Header

Schreibweise ändern

Grossschreibung/Kleinschreibung/CamelCase

= Table.TransformColumnNames(RenameColumns, Text.Upper)
= Table.TransformColumnNames(RenameColumns, Text.Lower)
= Table.TransformColumnNames(RenameColumns, Text.Proper)

Bestimmte Zeichen entfernen (z. B. _)

= Table.TransformColumnNames(Source,each Text.Proper(Replacer.ReplaceText( _ , "_", " ")))

Aufteilen in Worte

= Table.TransformColumnNames(#"Removed Other Columns", each Text.Combine(
                    Splitter.SplitTextByCharacterTransition({"a".."z"},{"A".."Z"})(_), " "))

Daten transformieren

Zeilen gruppenweise pivotieren

Aufgabenstellung

Werden Daten angeliefert, in denen das Gruppierungsmerkmal in den Zeilen vorhanden ist und somit mehrere Zeilen pro Datensatz vorhanden, wünscht man sich meist eine kompaktere Darstellung.

Für den Datensatz mit dem Wert “Daten 1” werden also vier Zeilen mit unterschiedlichen Werten in GRUPPE und Wert angeliefert.

Gewünscht ist aber eine kompaktere Darstellung mit den vorhandenen Gruppen als Spalten:

Die Aufgabenstellung ist somit die Umwandlung der angelieferten Daten:

Eine Beispieldatei liegt hier. Das Endergebnis liegt hier. Speichern sie beide Datein im Order C: \TMP, dann stimmt der Verweis in Query.xlsx auf die Daten Daten.xlsx.

Schritt 1: Daten vorbereiten

Im ersten Schritt erstellen wir eine neue Excel-Daten und greifen auf die vorbereiteten Daten über Power Query zu.

Wählen Sie dazu im Register Daten den Eintrag Daten abrufen / Aus Datei / Aus Arbeitsmappe und selektieren sie die gewünschte Datei:

Eine Beispieldatei liegt hier.

Ein Klick auf Importieren führt sie zum Navigation

Sie sehen im Navigator 3 verschiedenen Elemente:

  • DATEN: die intelligente Tabelle im Tabellenblatt. Diese beinhaltet genau die gewünschten Daten
  • ERGBNIS: die intelligente Tabelle, die das zu erwartende Ergbnis beinhaltet
  • Beispieldaten: das Tabellenblatt mit den beiden intelligenten Tabellen

Selektieren sie das Element DATEN und klicken sie auf Daten transformieren.

Schritt 2: Spalte pivotieren

Wir wollen die Werte der Spalte GRUPPE als neue Spalten erhalten.

Hier klicken sie auf die Spalte GRUPPE und wählen dein Eintrag Spalte pivotieren im Register Transformieren / Beliebige Spalte:

Die Werte für die neuen Spalten (Gruppe 1, Gruppe 2 , ..) kommen aus der Spalte WERT (Wert 11, Wert 12, ..):

Wir wollen die Werte selbst übernehmen und keine (wie bei Pivottabellen meist üblich) Aggregierungsfunktion verwenden (Summe, Max, Anzahl, ..).

Klicken sie hierzu auf Erweiterte Optionen und selektieren sie den Eintrag Nicht aggregieren:

Anschließen klicken sie auf OK:

Zum Abschluss beenden wir den Power Query Editor:

Azure Databricks | Cookbook

Reading Data

Create Table from CSV file with SQL

DROP TABLE IF EXISTS quickstart;

CREATE TABLE quickstart
USING csv
OPTIONS (path "/databricks-datasets/data.csv", header "true")

Create Table from CSV file with PySpark

%python
quickstart= spark.read.csv("/databricks-datasets/data.csv", header="true", inferSchema="true")

Analyse Data

Group and Display

%python
from pyspark.sql.functions import avg

display(quickstart.select("color","price").groupBy("color").agg(avg("price")).sort("color"))

Power Shell | Cookbook

Filesystem

List of files in a folder
(ls -r *.txt).fullname
Delete files with pattern
Get-ChildItem *.code -recurse | foreach { Remove-Item -Path $_.FullName }

Permissions

Show current policy
Get-ExecutionPolicy
Allow custom scripts to execute
Set-ExecutionPolicy -Scope CurrentUser unrestricted

PySpark | Cookbook

Websites

  • The Blaze Ecosystem (Blaze)
  • Dask: Flexible library for parallel computing in Python.
  • DataShape: Data layout language for array programming. 
  • Odo: Shapeshifting for your data
    It efficiently migrates data from the source to the target through a network of conversions.

Reading Textfiles

from pyspark.sql import SparkSession

spark = SparkSession \
    .builder \
    .appName("Demo") \
    .config("spark.demo.config.option", "demo-value") \
    .getOrCreate()

df = spark.read.csv("input.csv",header=True,sep="|");
from pyspark import SparkContext
from pyspark.sql import SQLContext
import pandas as pd

sc = SparkContext('local','example') 
sql_sc = SQLContext(sc)

p_df= pd.read_csv('file.csv')  # assuming the file contains a header
# p_df = pd.read_csv('file.csv', names = ['column 1','column 2']) # if no header
s_df = sql_sc.createDataFrame(pandas_df)
sc.textFile("file.csv") \
    .map(lambda line: line.split(",")) \
    .filter(lambda line: len(line)>1) \
    .map(lambda line: (line[0],line[1])) \
    .collect()
sc.textFile("input.csv") \
    .map(lambda line: line.split(",")) \
    .filter(lambda line: len(line)<=1) \
    .collect()
spark.read.csv(
    "input.csv", header=True, mode="DROPMALFORMED", schema=schema
)
(spark.read
    .schema(schema)
    .option("header", "true")
    .option("mode", "DROPMALFORMED")
    .csv("input.csv"))

Read CSV file with known structure

from pyspark.sql.types import StructType, StructField
from pyspark.sql.types import DoubleType, IntegerType, StringType

schema = StructType([
    StructField("A", IntegerType()),
    StructField("B", DoubleType()),
    StructField("C", StringType())
])

(sqlContext
    .read
    .format("com.databricks.spark.csv")
    .schema(schema)
    .option("header", "true")
    .option("mode", "DROPMALFORMED")
    .load("input.csv"))

Jenkins | Cookbook

Working with VS Code

Validate Jenkins File

Install VS Code Plugin Jenkins Pipeline Linter Connector

Add configuration in .vscode/settings.json

"jenkins.pipeline.linter.connector.crumbUrl": "<JENKINS_URL>/crumbIssuer/api/xml?xpath=concat(//crumbRequestField,%22:%22,//crumb)",
"jenkins.pipeline.linter.connector.user": "<USERNAME>",
"jenkins.pipeline.linter.connector.pass": "<PASSWORD>",
"jenkins.pipeline.linter.connector.url": "<JENKINS_URL>/pipeline-model-converter/validate",

Replace <USERNAME>, <PASSWORD> and <JENKINS_URL> with your values, for example

"jenkins.pipeline.linter.connector.crumbUrl": "http://localhost:8080/crumbIssuer/api/xml?xpath=concat(//crumbRequestField,%22:%22,//crumb)",
"jenkins.pipeline.linter.connector.user": "admin",
"jenkins.pipeline.linter.connector.pass": "secret",
"jenkins.pipeline.linter.connector.url": "http://localhost:8080/pipeline-model-converter/validate",

Working with Jenkins Client (CLI)

Download Client

wget localhost:8080/jnlpJars/jenkins-cli.jar

Working with Plugins

Create aPlugin

mkdir SamplePlugin
cd SamplePlugin
mvn -U archetype:generate -Dfilter="io.jenkins.archetypes:"
mvn -U archetype:generate -Dfilter="io.jenkins.archetypes:global-configuration-plugin"
[INFO] Scanning for projects...
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-metadata.xml
Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/mojo/maven-metadata.xml
Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-metadata.xml (14 kB at 32 kB/s)
Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/mojo/maven-metadata.xml (20 kB at 44 kB/s)
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-archetype-plugin/maven-metadata.xml
Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-archetype-plugin/maven-metadata.xml (918 B at 18 kB/s)
[INFO]
[INFO] ------------------< org.apache.maven:standalone-pom >-------------------
[INFO] Building Maven Stub Project (No POM) 1
[INFO] --------------------------------[ pom ]---------------------------------
[INFO]
[INFO] >>> maven-archetype-plugin:3.1.2:generate (default-cli) > generate-sources @ standalone-pom >>>
[INFO]
[INFO] <<< maven-archetype-plugin:3.1.2:generate (default-cli) < generate-sources @ standalone-pom <<<
[INFO]
[INFO]
[INFO] --- maven-archetype-plugin:3.1.2:generate (default-cli) @ standalone-pom ---
[INFO] Generating project in Interactive mode
[INFO] No archetype defined. Using maven-archetype-quickstart (org.apache.maven.archetypes:maven-archetype-quickstart:1.0)
Choose archetype:
1: remote -> io.jenkins.archetypes:global-configuration-plugin (Skeleton of a Jenkins plugin with a POM and an example piece of global configuration.)
Choose a number or apply filter (format: [groupId:]artifactId, case sensitive contains): : 1
Choose io.jenkins.archetypes:global-configuration-plugin version:
1: 1.2
2: 1.3
3: 1.4
4: 1.5
5: 1.6
Choose a number: 5:
[INFO] Using property: groupId = unused
Define value for property 'artifactId': com.examples.jenkins.plugins
Define value for property 'version' 1.0-SNAPSHOT: :
[INFO] Using property: package = io.jenkins.plugins.sample
Confirm properties configuration:
groupId: unused
artifactId: com.examples.jenkins.plugins
version: 1.0-SNAPSHOT
package: io.jenkins.plugins.sample
 Y: : y
[INFO] ----------------------------------------------------------------------------
[INFO] Using following parameters for creating project from Archetype: global-configuration-plugin:1.6
[INFO] ----------------------------------------------------------------------------
[INFO] Parameter: groupId, Value: unused
[INFO] Parameter: artifactId, Value: com.examples.jenkins.plugins
[INFO] Parameter: version, Value: 1.0-SNAPSHOT
[INFO] Parameter: package, Value: io.jenkins.plugins.sample
[INFO] Parameter: packageInPathFormat, Value: io/jenkins/plugins/sample
[INFO] Parameter: version, Value: 1.0-SNAPSHOT
[INFO] Parameter: package, Value: io.jenkins.plugins.sample
[INFO] Parameter: groupId, Value: unused
[INFO] Parameter: artifactId, Value: com.examples.jenkins.plugins
[INFO] Project created from Archetype in dir: /Users/Shared/CLOUD/Kunde.BSH/workspace/SamplePlugin_Config/com.examples.jenkins.plugins
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  45.525 s
[INFO] Finished at: 2020-03-01T17:28:27+01:00
[INFO] ------------------------------------------------------------------------

Verify Plugin

 cd com.examples.jenkins.plugins
mvn verify

Run Plugin

mvn hpi:run

Working with Groovy Scripts

Include a common groovy script in Jenkins file

1: Create a common.groovy file with function as needed

def mycommoncode() {
}

2: In the main Jenkinfile load the file and use the function as shown below

node{ 
   def common = load “common.groovy”
   common.mycommoncode()
}

Basic example of Loading Groovy scripts

File example.groovy

def example1() {
 println 'Hello from example1' 
}
def example2() {
 println 'Hello from example2'
}

The example.groovy script defines example1 and example2 functions before ending with return this. Note that return this is definitely required and one common mistake is to forget ending the Groovy script with it.Jenkinsfile

def code node('java-agent') { 
    stage('Checkout') { checkout scm } 
    stage('Load') { code = load 'example.groovy' } 
    stage('Execute') { code.example1() }
} 

code.example2()

Processing Github JSON from Groovy

In this demo, we first show how to process JSON response from Github API in Groovy.Processing JSON from Github

String username = System.getenv('GITHUB_USERNAME') 
String password = System.getenv('GITHUB_PASSWORD') 
String GITHUB_API = 'https://api.github.com/repos' String repo = 'groovy' 
String PR_ID = '2' // Pull request ID 
String url = "${GITHUB_API}/${username}/${repo}/pulls/${PR_ID}" 

println "Querying ${url}" 

def text = url.toURL().getText(requestProperties: ['Authorization': "token ${password}"]) 
def json = new JsonSlurper().parseText(text) 

def bodyText = json.body // Check if Pull Request body has certain text if ( bodyText.find('Safari') ) {
     println 'Found Safari user' }

The equivalent bash command for retrieving JSON response from Github API is as follows:Equivalent bash command

// Groovy formatted string 
String cmd = "curl -s -H \"Authorization: token ${password}\" ${url}" 
// Example String 
example = 'curl -s -H "Authorization: token XXX" https://api.github.com/repos/tdongsi/groovy/pulls/2'

Processing Github JSON from Jenkinsfile

Continuing the demo from the last section, we now put the Groovy code into a callable function in a script called “github.groovy”. Then, in our Jenkinsfile, we proceed to load the script and use the function to process JSON response from Github API.github.groovy

import groovy.json.JsonSlurper
def getPrBody(String githubUsername, String githubToken, String repo, String id) {
   String GITHUB_API = 'https://api.github.com/repos' 
   String url = "${GITHUB_API}/${githubUsername}/${repo}/pulls/${id}" 

   println "Querying ${url}"

def text = url.toURL().getText(requestProperties: ['Authorization': "token ${githubToken}"])
def json = new JsonSlurper().parseText(text) 
def bodyText = json.body return bodyText } 
return this

Jenkinsfile

def code node('java-agent') { 
    stage('Checkout') { checkout scm }
    stage('Load') { code = load 'github.groovy' } 
    stage('Execute') {

Best Practice

Django | Cookbook

First steps

The following steps are based on a summary of the Django Tutorial

Create project

$ django-admin startproject main
$ cd working_with_django
$ python manage.py migrate
$ python manage.py runserver 8080
$ python manage.py startapp app_base

Create view

Create view in app_base/views.py

from django.http import HttpResponse

def index(request):
    return HttpResponse("Hello, world. You're at the polls index.")

Add view to app_base/urls.py

from django.urls import path
from . import views

urlpatterns = [
    path('', views.index, name='index'),
]

Add urls to project main/urls.py

from django.contrib import admin
from django.urls import include, path

urlpatterns = [
    path('admin/', admin.site.urls),
    path('app_base/', include('app_base.urls')),
]

Create admin user

$ python manage.py createsuperuser
Username (leave blank to use 'user'): admin
Email address: admin@localhost
Password: 
Password (again): 
Superuser created successfully.

Create data and database

Create database model in app_base/models.py

from django.db import models

class Question(models.Model):
    question_text = models.CharField(max_length=200)
    pub_date = models.DateTimeField('date published')

class Choice(models.Model):
    question = models.ForeignKey(Question, on_delete=models.CASCADE)
    choice_text = models.CharField(max_length=200)
    votes = models.IntegerField(default=0)

Activating models in main/settings.py

INSTALLED_APPS = [
    'app_base.apps.AppBaseConfig',

    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
]
$ python manage.py makemigrations app_base
$ python manage.py sqlmigrate app_base 0001

Make app modifiable in the admin (app_base/admin.py)

from django.contrib import admin
from .models import Question

admin.site.register(Question)

Writing more views

Create views in app_base/views.py

def detail(request, question_id):
    return HttpResponse("You're looking at question %s." % question_id)

def results(request, question_id):
    response = "You're looking at the results of question %s."
    return HttpResponse(response % question_id)

def vote(request, question_id):
    return HttpResponse("You're voting on question %s." % question_id)

Add new views into app_base/urls.py

from django.urls import path
from . import views

urlpatterns = [
    path('', views.index, name='index'),

    path('<int:question_id>/', views.detail, name='detail'),
    path('<int:question_id>/results/', views.results, name='results'),
    path('<int:question_id>/vote/', views.vote, name='vote'),
]

Add template in app_base/templates/polls/index.html

{% if latest_question_list %}
    <ul>
    {% for question in latest_question_list %}
        <li><a href="/polls/{{ question.id }}/">{{ question.question_text }}</a></li>
    {% endfor %}
    </ul>
{% else %}
    <p>No polls are available.</p>
{% endif %}

Modify view in app_base/views.py

from django.shortcuts import render
...
def index(request):
    latest_question_list = Question.objects.order_by('-pub_date')[:5]
    context = {'latest_question_list': latest_question_list}
    return render(request, 'polls/index.html', context)

Raising a 404 error in app_base/views.py

from django.http import HttpResponse
from django.shortcuts import render, get_object_or_404

from .models import Question
# ...
def detail(request, question_id):
    question = get_object_or_404(Question, pk=question_id)
    return render(request, 'polls/detail.html', {'question': question})

Create template app_base/templates/polls/detail.html

<h1>{{ question.question_text }}</h1>
<ul>
{% for choice in question.choice_set.all %}
    <li>{{ choice.choice_text }}</li>
{% endfor %}
</ul>

Removing hardcoded URLs in app_base/templates/polls/index.html

<li>
   <a href="{% url 'detail' question.id %}">{{ question.question_text }}</a>
</li>

The way this works is by looking up the URL definition as specified in the app_base/urs.py

...
# the 'name' value as called by the {% url %} template tag
path('<int:question_id>/', views.detail, name='detail'),
...

Namespacing URL names in app_base/urls.py

app_name = 'app_base'

urlpatterns = [
...

Then, modify link in app_base/templates/polls/index.html

from url ‘detail’ to url ‘app_base:detail’

<li>
    <a href="{% url 'app_base:detail' question.id %}">{{ question.question_text }}</a>
</li>

Use generic views: Less code is better

Create class in app_views/views.py

class HomeView(generic.TemplateView):
    template_name = 'index.html'

Create template app_views/templates/index.html

<h1>App Views:</h1>
Welcome

Modify app_views/urls.py

urlpatterns = [
    path('', views.HomeView.as_view(), name='home'),
]

Add another app to main project

Create app

$ python manage.py startapp app_view

Modify main/urls.py

urlpatterns = [
    path('admin/',     admin.site.urls),
    path('app_base/',  include('app_base.urls')),
    path('app_views/', include('app_views.urls')),
]

Add data model in app_views/models.py

from django.db import models

class DataItem(models.Model):
    text = models.CharField(max_length=200)
    data = models.IntegerField(default=0)

    def __str__(self):
        return self.text

Register data in app_views/admin.py

from django.contrib import admin
from .models import DataItem

admin.site.register(DataItem)

Activate models

$ python manage.py makemigrations app_views
$ python manage.py sqlmigrate app_views 0001
$ python manage.py migrate app_views

Navigation / Redirection

Set root page of Django project

When accessing your Django project, the root page will normaly doesn’n show your app homepage.

To change this, you hate to modiy the url handling.

In the following sample, replace <appname> with the name of your app

Define a redirection view in your app (/<appname>/urls.py)

def redirect_to_home(request):
    return redirect('/<appname>')

Define path in the global urls.py (/main/urls.py)

from django.contrib import admin
from django.urls import include, path
from django.shortcuts import redirect

from <appname> import views

urlpatterns = [
    path('',            views.redirect_to_home, name='home'),
    path('<appname>/',  include('<appname>.urls')),
    path('admin/',      admin.site.urls)
]

Highlight current page in navigation menu

<div class="list-group">
    <a href="{% url 'app:upload:basic' %}" class="list-group-item{% if request.path == '/upload/basic/' %} active{% endif %}">
            Basic Upload
    </a>
    <a href="{% url 'app:upload:progress' %}" class="list-group-item{% if request.path == '/upload/progress/' %} active{% endif %}">
            Progress Bar Upload
    </a>
</div>

Additional reading

Tutorials

Testing

Blogs and Posts

Resolving problems

Wrong template is used

The template system is using a search approach to find the specified template file, e.g. ‘home.html’.

If you created more than one apps with the same filenames for templates, the first one will be used.

Change the template folders and add the app name, e.g.

template/
        app_base/
                home.html

Resolving error messages and erors

‘app_name’ is not a registered namespace

One reason for this error is the usage of a namespace in a link.

Back to <a href="{% url 'app_views:home' %}">Home</a>

If you want to use this way of links, you have to define the namespace/appname in your <app>/urls.py file

app_name = 'app_views'
urlpatterns = [
    path('', views.HomeView.as_view(), name='home'),
]

dependencies reference nonexistent parent node

  • Recreate database and migration files
  • Remove all migration files under */migrations/00*.py
  • Remove all pycache folders under */__pycache__ and */*/__pycache__
  • Run migration again
$ python manage.py makemigrations
$ python manage migrate

ValueError: Dependency on app with no migrations: customuser

$ python manage.py makemigrations

Project Structure

Running tasks with Makefile

PREFIX_PKG := app

default:
	grep -E ':\s+#' Makefile

clearcache:	# Clear Cache
	python3 manage.py clearcache

run:		# Run Server
	python3 manage.py runserver 8000

deploy:		# Deploy
	rm -rf dist $(PREFIX_PKG)*
	rm -rf polls.dist
	cd polls && python3 setup.py sdist
	mkdir polls.dist && mv polls/dist/* polls/$(PREFIX_PKG)* polls.dist

install_bootstrap:	# Install Bootstrap Library
	cd .. && yarn add bootstrap
	rm -rf  polls/static/bootstrap
	mkdir   polls/static/bootstrap
	cp -R ../node_modules/bootstrap/dist/* polls/static/bootstrap

install_jquery:		# Install jQuery Library
	cd .. && yarn add jquery
	rm -rf polls/static/jquery
	mkdir  polls/static/jquery
	cp ../node_modules/jquery/dist/* polls/static/jquery

install_bootstrap_from_source:	# Install Bootstrap from Source
	mkdir -p install && \
	wget https://github.com/twbs/bootstrap/releases/download/v4.1.3/bootstrap-4.1.3-dist.zip -O install/bootstrap-4.1.3-dist.zip && \
	unzip install/bootstrap-4.1.3-dist.zip -d polls/static/bootstrap/4.1.3

D3.js | Cookbook

Basic Tasks

Create Container

const svg = d3
	.select('#playground')
	.append('svg')
	.attr('width', width + margin.left + margin.right)
	.attr('height', height + margin.top + margin.bottom)
	.append('g')
	.attr('transform',
	      'translate(' + margin.left + ',' + margin.top + ')');

Axes: Create

const x = d3Scale.scaleLinear().domain([0, 100]).range([0, width]);
const y = d3Scale.scaleLinear().domain([0, 100]).range([height, 0]);

const xAxis = d3Axis.axisBottom(x).tickFormat(d3.timeFormat('%m/%d'));
const yAxis = d3Axis.axisLeft(y).ticks(10);

Axes

Format

var axisTimeFormat = d3.time.format.multi([
    [".%L",   (d) => d.getMilliseconds()],
    [":%S",   (d) => d.getSeconds()],
    ["%H:%M", (d) => d.getMinutes()],
    ["%H:%M", (d) => d.getHours()],
    ["%a %d", (d) => d.getDay() && d.getDate() != 1],
    ["%b %d", (d) => d.getDate() != 1; }],
    ["%B",    (d) => d.getMonth()],
    ["%Y",    ()  => true]
 ]);

var xAxis = d3.svg.axis()
    .scale(x)
    .orient("bottom")
    .tickFormat(axisTimeFormat);

Installation

$ npm install -save d3
                    d3-scale
                    d3-array
                    d3-shape
                    d3-interpolate
                    d3-color
                    d3-time-format
                    d3-ease
                    d3-transition
                    d3-selection
                    d3-time
                    d3-path
                    d3-dispatch
                    d3-format
                    d3-timer
                    d3-drag
                    d3-geo
                    d3-brush
                    d3-dsv
                    d3-random

ClojureScript | Cookbook

UI: Common tasks

Loading spinner

index.html

<head>
	....
	<link href="lib/bootstrap/4.4.1/css/bootstrap.min.css" rel="stylesheet" type="text/css">
	<link href="lib/css/style.css" rel="stylesheet" type="text/css">
</head>

<body>
	<div id="app">
		<div class="d-flex justify-content-center">
			<div class="loader"></div>
		</div>
	</div>
	<script src="lib/jquery/3.4.1/jquery.min.js" type="text/javascript"></script>
	<script src="lib/bootstrap/4.4.1/js/bootstrap.bundle.min.js" type="text/javascript"></script>
	....

lib/css/style.css

.loader {
	position: fixed;
	z-index: 999;
	overflow: visible;
	margin: auto;

	border: 16px solid #f3f3f3;
	border-radius: 50%;
	border-top: 16px solid blue;

	width: 120px; height: 120px;
	top: 0; left: 0; bottom: 0; right: 0;

	-webkit-animation: spin 2s linear infinite;
	/* Safari */
	animation: spin 2s linear infinite;
}

/* Transparent Overlay */
.loader:before {
	content: '';
	display: block;
	position: fixed;
	top: 0; left: 0;
	width: 100%;
	height: 100%;
}

/* Safari */
@-webkit-keyframes spin {
	0%   { -webkit-transform: rotate(0deg); }
	100% { -webkit-transform: rotate(360deg); }
}

@keyframes spin {
	0%   { transform: rotate(0deg);  }
	100% { transform: rotate(360deg);}
}

Rendering

How to render content

(r/render-component [content]
  (.querySelector js/document "#app"))
(r/render-component [content])
  (.-body js/document)
(defn ^:export run []
  (r/render [simple-example]
  (js/document.getElementById "app"))
(defn home-page []
  [:div
    [:h2 "Homepage!"]])

;; -------------------------
;; Initialize app

(defn mount-root []
  (reagent/render [home-page] (.getElementById js/document "app")))

(defn init! []
  (mount-root))

Render several html tags as one

(defn content-row
    ""
    [col1 col2 col3]
    [:<>
        [:div.grid-item col1]
        [:div.grid-item col2]
        [:div.grid-item col3]])

rxjs | Cookbook

rxjs

Informations

NGRX – Reactive State for Angular

Original Post is here. Github repo has the code snippets at choopage’s GitHub repo.

Recipes


Chaining of observable

The below snippet would return 0, 1, 2, 3…, n until it is stopped. It would be returned at every 2 sec. See reference here.

import { Observable } <strong>from 'rxjs/Rx';

let obs = Observable.<em>interval</em>(2000);
let req = obs
    .flatMap(v => { return Observable.of(v) })
    .subscribe(
        v   => console.log(v),
        err => console.error(err),
        ()  => console.log(<strong>'done'</strong>)
    );

Response transformer

The below would return undefined at every 2 sec interval.

import { Observable } from 'rxjs/Rx';

let obs = Observable.interval(2000);

obs
    .let(responseTransformer)
    .subscribe(
        v  => console.log(v),
        err=> console.error(err)
    );

responseTransformer(input: Observable<any>) {
    return input.map(v => v.data);
}

Using RxJS composition over base class extension

export class MyAPI {
    constructor(private http: Http) {
    }

    get(url: string, options: any) {
        return this.http.get(url, options).let(responseTransformer);
    }
}

responseTransformer(input: Observable<any>) {
    return input.map(v => v.data);
}

Higher Order Observable

We create higher order observable using .map.

const numObservable = Rx.Observable.interval(1000).take(4);
const higherOrderObservable = numObservable.map(x => Rx.Observable.of(1, 2));

higherOrderObservable.subscribe(<br>x => x.subscribe(y => console.log(y)));

Further use of higher order observable

usingHigherOrderObservable() {
    Observable
        .interval(1000)
        .groupBy(n => n % 2);
}

Flatten a higher order observable with RxJS switch

const numObservable = Rx.Observable.interval(1000).take(2);
const higherOrderObservable = numObservable.map(x => Rx.Observable.of(1, 2)).switch();

/* 
------+--------+---------
       \        \
        1,2      1,2
        
        switch
        
------1-2-------1-2-------

Switch map flattens the higher order observable
*/

higherOrderObservable.subscribe(x => console.log(x));

switchMap: map and flatten higher order observables

const clickObservable = Rx.Observable.fromEvent(document, 'click');

function performRequest() {
  return fetch('<a href="http://jsonplaceholder.typicode.com/users/1'" target="_blank" rel="noreferrer noopener">http://jsonplaceholder.typicode.com/users/1'</a>)
  .then(res =>; res.json());
  //this returns a Promise
}
//Observabl<Event> --> Observable<Response><br>const responseObservable = clickObservable<br>  .switchMap(click => performRequest());//switchMap can convert Promise to Observable<br>//switchMap = map .... + ... switchresponseObservable.subscribe(x => console.log(x.email));

Use groupBy in real RxJS applications

See reference here.

const busObservable = Rx.Observable.of(
  {code: 'en-us', value: '-TEST-'},
  {code: 'en-us', value: 'hello'},
  {code: 'es', value: '-TEST-'},
  {code: 'en-us', value: 'amazing'},
  {code: 'pt-br', value: '-TEST-'},
  {code: 'pt-br', value: 'olá'},
  {code: 'es', value: 'hola'},
  {code: 'es', value: 'mundo'},
  {code: 'en-us', value: 'world'},
  {code: 'pt-br', value: 'mundo'},
  {code: 'es', value: 'asombroso'},
  {code: 'pt-br', value: 'maravilhoso'}
).concatMap(x => Rx.Observable.of(x).delay(500));const all = busObservable
  .groupBy(obj => obj.code);
  .mergeMap(innerObs => innerObs.skip(1).map(obj => obj.value));//Alternatively could using filter and map
/*
const enUS = busObservable
  .filter(obj => obj.code === 'en-us')
  .map(obj => obj.value);const es = busObservable
  .filter(obj => obj.code === 'es')
  .map(obj => obj.value);const all = Rx.Observable.merge(enUS, es);
*/all.subscribe(x => console.log(x));

Using .map versus .switchMap

The below code snippet we can view the result of using .map versus .switchMap

//user.service.ts
import { Injectable } from '@angular/core';
import { Http } from '@angular/http';
import { Observable } from "rxjs";
import 'rxjs/add/operator/map';
import 'rxjs/add/operator/do';
import 'rxjs/add/operator/switchMap';
import 'rxjs/add/operator/filter';
import 'rxjs/add/operator/catch';
import 'rxjs/add/operator/debounceTime';
import 'rxjs/add/operator/distinctUntilChanged';

@Injectable()
export class UserService {

  constructor(private http: Http) { }

  getUsers(): Observable<any> {
    return this.http.get('http://jsonplaceholder.typicode.com/users')
        //.map(v => v.json());
        .switchMap(v => v.json());
  }

}
//app.component.ts
import { Component } from '@angular/core';
import { UserService } from "./user.service";

@Component({
  selector: 'app-root',
  templateUrl: './app.component.html',
  styleUrls: ['./app.component.css']
})
export class AppComponent {
  title = 'app works!';

  constructor(private userService: UserService) {

  }

  search(term: string) {
    this.userService.getUsers()
        .subscribe(v => console.log(v));

    /*
    //we can do this with .switchMap
    this.userService.getUsers()
        .subscribe(v => {if (v.email != "Sincere@april.biz") {
          console.log(v.email);
        }});
     */
  }
}

Solving the multiple Async Pipe in Angular ≥ 2.0.0 with share operator

Remember to import import "rxjs/add/operator/share"; See reference here.

squareData$: Observable<string> = Observable.range(0, 10)
        .map(x => x * x)
        .do(x => console.log(`CalculationResult: ${x}`)
        .toArray()
        .map(squares => squares.join(", "))
        .share();  // remove this line: console will log every result 3 times instead of 1

Managing Cold and Hot Observables using publish().refCount() which is similar to .share()

ngOnInit() {
    // in angular 2 and above component.ts file add these    this.coldObservable();
    this.hotObservable();
}
/*
* cold observable is like a recast of video
* */
coldObservable() {
    let incrementalObs = Observable.interval(1000).take(10).map(x => x + 1);
    incrementalObs.subscribe(val => console.log('a: ' + val));
    setTimeout(function() {
        incrementalObs.subscribe(val => console.log('      b: ' + val));
    }, 4500);
}


/*
* hot observable is like watching a live video
* */
hotObservable() {
    let incrementalObs = Observable.interval(1000).take(10).map(x => x + 1).publish().refCount(); //can also use .share()
    incrementalObs.subscribe(val => console.log('a: ' + val));
    setTimeout(function() {
        incrementalObs.subscribe(val => console.log('      b: ' + val));
    }, 4500);
}

Observables Array Operations with flatMap

Reference Rangle.io article.

getLoadList(): void {
  this.shareService
      .fetchLoad()
      .take(1)
      .filter(response => {
        if ( response.status === 200 ) {
          return true;
        } else if ( response.status === 304 ) {
          // do something more
          return false;
        } else {
          this.gotoErrorPage();
          return false;
        }
      })
      .flatMap(response => response.data.loads as Load[])
      .filter(obj => obj.content.contentGrade === 'x')
      .subscribe(
          val => console.log(val),
          err => {
            console.error(err);
          });
}

Error Handling


Error handling in RxJS

Some learning points from RxJS lesson videos. This repo is available in my GitHub repo.

import { Component, OnInit } from '@angular/core';

import { Observable } from 'rxjs/Observable';
import { Subject } from 'rxjs/Subject';
import { BehaviorSubject } from "rxjs/BehaviorSubject";

import 'rxjs/add/observable/bindCallback';
import 'rxjs/add/observable/bindNodeCallback';
import 'rxjs/add/observable/combineLatest';
import 'rxjs/add/observable/concat';
import 'rxjs/add/observable/defer';
import 'rxjs/add/observable/empty';
import 'rxjs/add/observable/forkJoin';
import 'rxjs/add/observable/from';
import 'rxjs/add/observable/fromEvent';
import 'rxjs/add/observable/fromEventPattern';
import 'rxjs/add/observable/fromPromise';
import 'rxjs/add/observable/generate';
import 'rxjs/add/observable/if';
import 'rxjs/add/observable/interval';
import 'rxjs/add/observable/merge';
import 'rxjs/add/observable/race';
import 'rxjs/add/observable/never';
import 'rxjs/add/observable/of';
import 'rxjs/add/observable/onErrorResumeNext';
import 'rxjs/add/observable/pairs';
import 'rxjs/add/observable/range';
import 'rxjs/add/observable/using';
import 'rxjs/add/observable/throw';
import 'rxjs/add/observable/timer';
import 'rxjs/add/observable/zip';
import 'rxjs/add/observable/dom/ajax';
import 'rxjs/add/observable/dom/webSocket';
import 'rxjs/add/operator/buffer';
import 'rxjs/add/operator/bufferCount';
import 'rxjs/add/operator/bufferTime';
import 'rxjs/add/operator/bufferToggle';
import 'rxjs/add/operator/bufferWhen';
import 'rxjs/add/operator/catch';
import 'rxjs/add/operator/combineAll';
import 'rxjs/add/operator/combineLatest';
import 'rxjs/add/operator/concat';
import 'rxjs/add/operator/concatAll';
import 'rxjs/add/operator/concatMap';
import 'rxjs/add/operator/concatMapTo';
import 'rxjs/add/operator/count';
import 'rxjs/add/operator/dematerialize';
import 'rxjs/add/operator/debounce';
import 'rxjs/add/operator/debounceTime';
import 'rxjs/add/operator/defaultIfEmpty';
import 'rxjs/add/operator/delay';
import 'rxjs/add/operator/delayWhen';
import 'rxjs/add/operator/distinct';
import 'rxjs/add/operator/distinctUntilChanged';
import 'rxjs/add/operator/distinctUntilKeyChanged';
import 'rxjs/add/operator/do';
import 'rxjs/add/operator/exhaust';
import 'rxjs/add/operator/exhaustMap';
import 'rxjs/add/operator/expand';
import 'rxjs/add/operator/elementAt';
import 'rxjs/add/operator/filter';
import 'rxjs/add/operator/finally';
import 'rxjs/add/operator/find';
import 'rxjs/add/operator/findIndex';
import 'rxjs/add/operator/first';
import 'rxjs/add/operator/groupBy';
import 'rxjs/add/operator/ignoreElements';
import 'rxjs/add/operator/isEmpty';
import 'rxjs/add/operator/audit';
import 'rxjs/add/operator/auditTime';
import 'rxjs/add/operator/last';
import 'rxjs/add/operator/let';
import 'rxjs/add/operator/every';
import 'rxjs/add/operator/map';
import 'rxjs/add/operator/mapTo';
import 'rxjs/add/operator/materialize';
import 'rxjs/add/operator/max';
import 'rxjs/add/operator/merge';
import 'rxjs/add/operator/mergeAll';
import 'rxjs/add/operator/mergeMap';
import 'rxjs/add/operator/mergeMapTo';
import 'rxjs/add/operator/mergeScan';
import 'rxjs/add/operator/min';
import 'rxjs/add/operator/multicast';
import 'rxjs/add/operator/observeOn';
import 'rxjs/add/operator/onErrorResumeNext';
import 'rxjs/add/operator/pairwise';
import 'rxjs/add/operator/partition';
import 'rxjs/add/operator/pluck';
import 'rxjs/add/operator/publish';
import 'rxjs/add/operator/publishBehavior';
import 'rxjs/add/operator/publishReplay';
import 'rxjs/add/operator/publishLast';
import 'rxjs/add/operator/race';
import 'rxjs/add/operator/reduce';
import 'rxjs/add/operator/repeat';
import 'rxjs/add/operator/repeatWhen';
import 'rxjs/add/operator/retry';
import 'rxjs/add/operator/retryWhen';
import 'rxjs/add/operator/sample';
import 'rxjs/add/operator/sampleTime';
import 'rxjs/add/operator/scan';
import 'rxjs/add/operator/sequenceEqual';
import 'rxjs/add/operator/share';
import 'rxjs/add/operator/single';
import 'rxjs/add/operator/skip';
import 'rxjs/add/operator/skipUntil';
import 'rxjs/add/operator/skipWhile';
import 'rxjs/add/operator/startWith';
import 'rxjs/add/operator/subscribeOn';
import 'rxjs/add/operator/switch';
import 'rxjs/add/operator/switchMap';
import 'rxjs/add/operator/switchMapTo';
import 'rxjs/add/operator/take';
import 'rxjs/add/operator/takeLast';
import 'rxjs/add/operator/takeUntil';
import 'rxjs/add/operator/takeWhile';
import 'rxjs/add/operator/throttle';
import 'rxjs/add/operator/throttleTime';
import 'rxjs/add/operator/timeInterval';
import 'rxjs/add/operator/timeout';
import 'rxjs/add/operator/timeoutWith';
import 'rxjs/add/operator/timestamp';
import 'rxjs/add/operator/toArray';
import 'rxjs/add/operator/toPromise';
import 'rxjs/add/operator/window';
import 'rxjs/add/operator/windowCount';
import 'rxjs/add/operator/windowTime';
import 'rxjs/add/operator/windowToggle';
import 'rxjs/add/operator/windowWhen';
import 'rxjs/add/operator/withLatestFrom';
import 'rxjs/add/operator/zip';
import 'rxjs/add/operator/zipAll';
@Component({
    selector: 'app-root',
    templateUrl: './app.component.html',
    styleUrls: [ './app.component.css' ]
})
export class AppComponent implements OnInit {
    title = 'app works!';
    obs = Observable.of(1, 2, 3, 4);

    ngOnInit() {
        this.howToHandleErrorV1();
        this.howToHandleErrorV2();
        this.howToUseRetry();
        this.mergeObservableAndThrowError();
        this.mergeObservableAndErrorResumeNext();
        this.mergeObservableAndErrorCatch();    
    }

    
    
    /*
     * This uses Catch for V1. This introduces Closure. It is effectively the same as V2.
     * */
    howToHandleErrorV1() {
        this.obs
            .map(x => {
                if ( x === 3 ) {
                    throw 'I hate threes'; // When it hitted error it actually unsubscribe itself at x === 3 of throw error
                }
                return x;
            })
            .catch(err => Observable.throw('Caught error here Observable.throw')) // continue go down the error path use Observable.throw
            .catch(err => Observable.of('Caught error here Observable.of')) // catch just use Observable.of
            .subscribe(
                x => console.log(x),
                err => console.error(err), // If not catch any where, the I hate threes errors will be propagated to here
                () => console.log('done completed')
            );
    }

    
    
    /*
     * There is a difference between V1 and V2. For V2 it is using onErrorResumeNext which
     * */
    howToHandleErrorV2() {
        let good = Observable.of('Caught error here Observable.of');

        this.obs
            .map(x => {
                if ( x === 3 ) {
                    throw 'I hate threes'; // When it hit error it actually unsubscribe itself at x === 3 of throw error
                }
                return x;
            })
            .onErrorResumeNext(good) // To catch just use Observable.of
            .subscribe(
                x => console.log(x),
                err => console.error(err), // If not catch any where, the I hate threes errors will be propagated to here
                () => console.log('done completed')
            );
    }

    
    
    /*
     * For this we use see it retries three times then console.error(err);
     * So retryWhen is for trying network connection websocket
     * */
    howToUseRetry() {
        this.obs
            .map(x => {
                if ( x === 3 ) {
                    throw 'I hate threes'; // When it hitted error it actually unsubscribe itself at x === 3 of throw error
                }
                return x;
            })
            .retry(3) // retry three times
            .retryWhen(err => err.delay(2000).take(3)) // similar but with 2 seconds delay and the error is not propagated.
            .retryWhen(err => err.delay(2000).take(3).concat(Observable.throw('bad'))) // this it would throw an error.
            .subscribe(
                x => console.log(x),
                err => console.error(err), // If not catch any where, the I hate threes errors will be propagated to here
                () => console.log('done completed')
            );
    }
/*
 * Using observable merge operator
 * */
mergeObservableAndThrowError() {
    let mergedObs = Observable.merge(
        this.obs, //1, 2, 3, 4
        Observable.throw('Stop Error'),
        Observable.from(this.array), //0, 1, 2, 3, 4, 5
        Observable.of(999) //999,
    );

    mergedObs.subscribe(
        val => console.log(val), //this should show 1, 2, 3, 4, Stop Error
        error => console.log(error),
        () => console.log("completed")
    );
}

/* Using observable onErrorResumeNext just like merge operator
 * */
mergeObservableAndErrorResumeNext() {
    let mergedObs = Observable.onErrorResumeNext(
        this.obs, //1, 2, 3, 4
        Observable.throw('Stop Error'),
        Observable.from(this.array), //0, 1, 2, 3, 4, 5
        Observable.of(999) //999,
    );

    mergedObs.subscribe(
        val => console.log(val), //this should show 1, 2, 3, 4, 0, 1, 2, 3, 4, 5, 999
        error => console.log(error),
        () => console.log("completed")
    );
}
/*
 * Using observable merge operator and catch
 * */
mergeObservableAndErrorCatch() {
    let mergedObs = Observable.merge(
        this.obs, //1, 2, 3, 4
        Observable.throw('Stop Error'),
        Observable.from(this.array), //0, 1, 2, 3, 4, 5
        Observable.of(999) //999,
    ).catch(e => {
        console.log(e);
        return Observable.of('catch error here');
    });

    mergedObs.subscribe(
        val => console.log(val), //this should show 1, 2, 3, 4, Stop Error, Catch Error Here
        error => console.log(error),
        () => console.log("completed")
    );
}
}

map vs flatMap in RxJS

Transform the items emitted by an Observable into Observables, then flatten the emissions from those into a single Observable. See my GitHub repo.

obs = Observable.of(1, 2, 3, 4);

ngOnInit() {
    this.usingMap();
    this.usingMapToMakeInnerObservable();
    this.usingMapAndMergeAll();
    this.usingFlatMap();
}usingMap() {
    this.obs
        .map(x => x * 2) // transform the input by multiple of 2
        .subscribe(
            x => console.log(x),
            err => console.error(err),
            () => console.log('done completed')
        );
}
usingMapToMakeInnerObservable() {
    this.obs
        .map(x => Observable.timer(500).map(() => x + 3)) // transform the input wrapping it with another observable and addition of 3
        //.map(x => Observable.timer(500).map((x) => x + 3)) // !!! REMEMBER Not the same as the immediate above
        .subscribe(
            x => console.log(x),
            err => console.error(err),
            () => console.log('done completed')
        );
}
// Map and Merge all is the same as just one FlatMap
usingMapAndMergeAll() {
    this.obs
        .map(x => Observable.timer(500).map(() => x + 3)) // transform the input wrapping it with another observable and addition of 3
        .mergeAll()
        .subscribe(
            x => console.log(x),
            err => console.error(err),
            () => console.log('done completed')
        );
}
// Flat map is the same as map then merge all
// transform the items emitted by an Observable into Observables, then flatten the emissions from those into a single Observable
usingFlatMap() {
    this.obs
        .flatMap(x => Observable.timer(500).map(() => x + 10)) // transform the input wrapping it with another observable and addition of 10
        .subscribe(
            x => console.log(x),
            err => console.error(err),
            () => console.log('done completed')
        );
}

Transforming pure Javascript array vs. Observable from array

See GitHub for source code

array = [0, 1, 2, 3, 4, 5];

ngOnInit() {
    this.setArrayToObservableThenTransform();
}/*
* This keeps creating new array. It is good that it creates new array of arr for immutability.
* But it's bad because there is clean up and resource intensive for mobile
* */
transformArray() {
    let result = this.array
        .filter(( x, i, arr ) => {
            console.log('filtering ' + x);
            console.log('is the source array ' + (arr === this.array));
            return x % 2 === 0;
        })
        .map(( x, i, arr ) => {
            console.log('mapping ' + x);
            console.log('is the source array ' + (arr === this.array));
            return x + '!';
        })
        .reduce(( r, x, i, arr ) => {
            console.log('reducing ' + x);
            return r + x;
        }, '--');

    console.log(result);
}

/*
* This is more efficient for resource management because it linearly scans and discard when not right
* */
setArrayToObservableThenTransform() {    let obsArray = Observable.from(this.array); // Use Observable.from() instead of Observable.of(). There is diff.    obsArray
        .filter(( x: any ) => {
            console.log('filtering ' + x);
            return x % 2 === 0;
        })
        .map(( x ) => {
            console.log('mapping ' + x);
            return x + '!';
        })
        .reduce(( r, x ) => {
            console.log('reducing ' + x);
            return r + x;
        }, '--')
        .subscribe(
            x => console.log(x)
        );
}

Using reduce and scan to aggregate RxJs data

See GitHub for source code

array = [0, 1, 2, 3, 4, 5];ngOnInit() {
    this.reduceArray();
    this.reduceObservableArray();
    this.reduceObservableArray_Abstract2();
    this.scanObservableArray();
}
/*
* This is the same as reduceObservableArray()
* */
reduceArray() {
    let result = this.array.reduce(
        (accumulator, currentValue) => accumulator + currentValue, 3
    ); // 3 is the init value.
    console.log('reduceArray ' + result); // output 18 => 3 + (0 ... 5)
}
/*
* This is the same as reduceArray()
* But this waits for all the arrays to finish emitting before reducing them to one single number
* See the next method to understand better
* */
reduceObservableArray() {
    let obsArray = Observable.from(this.array);
    obsArray.reduce(
        (accumulator, currentValue) => accumulator + currentValue, 3
    ).subscribe(
        val => console.log('reduceObservableArray ' + val)
    );
}
/*
* The exact same reduce function/method as of reduceObserableArray() above
* This proves that it waits for all 6 numbers to come in then reduce them
* */
reduceObservableArray_Abstract2() {
    let obsArray = Observable.interval(1000).take(6); //emits 6 times of 0, 1, 2, 3, 4, 5
    obsArray.reduce(
        (accumulator, currentValue) => accumulator + currentValue, 3
    ).subscribe(
        val => console.log('reduceObservableArray_Abstract2 ' + val)
    );
}
/*
* This is the same as the above reduceObserableArray_Abstract2()
* except this is using scan instead of reduce
* */
scanObservableArray() {
    let obsArray = Observable.interval(1000).take(6); //emits 6 times of 0, 1, 2, 3, 4, 5
    obsArray.scan(
        (accumulator, currentValue) => accumulator + currentValue, 3
    ).subscribe(
        val => console.log('scanObservableArray() ' + val)
    );
}

Create, next, and subscribe to Subject and BehaviorSubject

There is a Stack Overflow thread which discussed about the difference between Subject and BehaviorSubject. It’s worth understanding.

Also see my personal GitHub for source code.

app.component.ts

import { Subject } from 'rxjs/Subject';
import { BehaviorSubject } from "rxjs/BehaviorSubject";
// create subject
// there is no need for initial value
subject = new Subject<boolean>();

// create behaviorSubject which require initial value
// true is an initial value. if there is a subscription
// after this, it would get true value immediately
behaviorSubject = new BehaviorSubject<boolean>(true);
ngOnInit() {
    this.subject.next(false); /* Subject subscription wont get anything at this point before the subscribeSubject() */    this.subscribeSubject();
    this.subscribeBehaviorSubject();
}
/*
* Push the next val into the behavior subject
* */
nextSubject(val: boolean) {
    this.subject.next(val);
}

/*
* Any values push into the subject would not be can shown
* before this subscribeSubject() is called
* */
subscribeSubject() {
    this.subject
        //.take(1) //when we include .take(1) we will have a complete. Without this it will continue subscribing
        .subscribe(
            val => console.log(val),
            err => console.error(err),
            () => console.log('completed')
        );
}

/*
* This is the proper way to return a subject as observable
* */
getSubject(): Observable<boolean> {
    return this.subject.asObservable();
}

/*
 * Push the next val into the behavior subject
 * */
nextBehaviorSubject(val: boolean) {
    this.behaviorSubject.next(val);
}

/*
* For angular Behavior subject for a data service as a angular service often initializes
* before component and behavior subject ensures that the component consuming the
* service receives the last updated data even if there are no new
* updates since the component's subscription to this data.
* */
subscribeBehaviorSubject() {
    this.behaviorSubject
        // .first()
        .subscribe(
            val => console.log(val),
            err => console.error(err),
            () => console.log('completed')
        );
}

app.component.html

Subject:
<button (click)="nextSubject(true)">true</button>
<button (click)="nextSubject(false)">false</button>

<br>
BehaviorSubject:
<button (click)="nextBehaviorSubject(true)">true</button>
<button (click)="nextBehaviorSubject(false)">false</button>

Finally operator

usingFinallyOperator() {
    Observable
        .interval(500)
        .take(4)
        .finally(() => console.log('End of the observable, Hello World'))
        .subscribe(
            val => console.log('count taker ' + val)
        );
}

Stopping / Intercepting Observable

Imagine using Gmail where it allows you to undo email sent? We can produce similar experience with Observable

// subscription is created when an observable is being subscribed
subscription: Subscription;

// boolean variable for showing stop observable using takeWhile operator
isTrue: boolean = true;
/*
* basic interval can be used as delay too
* Imagine Gmail allows you to send and undo send within 4 seconds of sending
* Use Case: Perform an action 8 seconds later then intercept if user choose to undo the action
* */basicInterval() {
    let undoInSeconds: number = 8;
    this.subscription = Observable
            .interval(1000)
            .take(undoInSeconds)
            .takeWhile(() => this.isTrue)
            .subscribe(
                (val: number) => {
                    console.log(`${val + 1} seconds...         UNDO`);
                    ( val === (undoInSeconds - 1) ) ? console.log('Email sent / Action performed') : null;
                }
            );
}
/*
* This is to stop observable from continuing performance
* Use Case: Stop observable from running like how Gmail could undo email being sent
* */stopObservableUsingUnsubscribe() {
    if (!!this.subscription) {
        this.subscription.unsubscribe();
        console.log('subscription: Subscription is unsubscribed');
    }
}

/*
* This is also to stop observable from continuing performance
* This method is more preferable than subscribing method then unsubscribe
 * Use Case: Stop observable from running like how Gmail could undo email being sent
* */stopObservableUsingTakeWhile() {
    this.isTrue = false;
}

Perform conditional Reactive Form validation

This is my approach to performing conditional validation when using Angular. We will minimally manipulate the Observable of RxJS in this example. Let’s try by creating or using app.component.ts.

1 . Create a form that has two form controls reason and otherReason.

/* * Refer to angular official guide at https://angular.io/guide/reactive-forms on how to create reactive form with form controls * */createForm() {
    this.form = this.formBuilder.group({      reason: ['', Validators.required ],
      otherReason: [''],    });
}

2 . Create two methods addValidator and removeValidator.

/*  * For conditional form validation use  * */
private addValidator( control: AbstractControl, newValidator ){
    let existingValidators = control.validator;
    control.setValidators(Validators.compose([ existingValidators, newValidator ]));
    control.updateValueAndValidity();
}

/*  * For conditional form validation use  * */
private removeValidator( control: AbstractControl ){
    control.clearValidators();
    control.updateValueAndValidity();
}

3 . Create third method called conditionalFormValidation.

conditionalFormValidation( parentField, childField, matchValue = 'Others' )
{
    this.form
        .get(parentField)
        .valueChanges
        .forEach(( value: string ) => {
            const childFieldControl: AbstractControl = this.form.get(childField);
            if ( value === matchValue ) {
                this.addValidator(childFieldControl, Validators.required);
            } else {
                this.removeValidator(childFieldControl);
            }
        });
}

4 . Create or add to ngOnInit method

ngOnInit() {    
    this.createForm();
    this.conditionalFormValidation('reason', 'otherReason');
}

The outcome should illustrate that if we select ‘Others’ option in the dropdown list of reason, it should make otherReason form control field as required.

Drop-down selection for reason e.g. reason for absent from work
On selecting Others, the conditional form validation would make otherReason field turn into a required field

Perform manual operations after reading from Firebase database

When using AngularFire2 with Angular + Firebase, in getting a list of data from Firebase, we will get one observable instance but within that one observable is an array of N size. We can manually filter the array inside that one observable instance using arr.filter. It is different from RxJS .filter operator. Of course we can also flatten what is inside an array using .flatMap() operator. However, we’re going use JavaScript array filtering, instead of non-observable filtering, right after getting an observable object.

this.db.list(`/review`)
       .map(arr => arr.filter(item => item.rating > 3));

We can also reverse an array using JavaScript array reverse function. See reference. On a side note, using negative timestamp to reverse Firebase display is also another option.

this.db.list('/review')
       .map(arr => { return arr.reverse(); });

Besides those above, we can also use the response returned from AngularFire2 to perform “local filter/search”. This result can be valuable for autocomplete filtered list or searches. However, this approach below suffers severely in performance issue where at the magnitude of the size of the returned response from AngularFire2. E.g. if the list has N items. It has to iterate at least 1N. Perhaps an average of 2N.

getFilteredClientList(searchQuery: string): Observable<Client[]> {
    let query = searchQuery.trim().toLowerCase();

    return this.db.list(`/client`)
      .map((arr: Client[]) => {
        return arr.filter((item: Client) => {
          return item.email.toLowerCase().indexOf(query) === 0
            || item.given_name.toLowerCase().indexOf(query) === 0
            || item.family_name.toLowerCase().indexOf(query) === 0
        });
      });
  }