Azure Databricks | Cookbook

Reading Data

Create Table from CSV file with SQL

DROP TABLE IF EXISTS quickstart;

CREATE TABLE quickstart
USING csv
OPTIONS (path "/databricks-datasets/data.csv", header "true")

Create Table from CSV file with PySpark

%python
quickstart= spark.read.csv("/databricks-datasets/data.csv", header="true", inferSchema="true")

Analyse Data

Group and Display

%python
from pyspark.sql.functions import avg

display(quickstart.select("color","price").groupBy("color").agg(avg("price")).sort("color"))

Power Shell | Cookbook

Filesystem

List of files in a folder
(ls -r *.txt).fullname
Delete files with pattern
Get-ChildItem *.code -recurse | foreach { Remove-Item -Path $_.FullName }

Permissions

Show current policy
Get-ExecutionPolicy
Allow custom scripts to execute
Set-ExecutionPolicy -Scope CurrentUser unrestricted

PySpark | Cookbook

Websites

  • The Blaze Ecosystem (Blaze)
  • Dask: Flexible library for parallel computing in Python.
  • DataShape: Data layout language for array programming. 
  • Odo: Shapeshifting for your data
    It efficiently migrates data from the source to the target through a network of conversions.

Reading Textfiles

from pyspark.sql import SparkSession

spark = SparkSession \
    .builder \
    .appName("Demo") \
    .config("spark.demo.config.option", "demo-value") \
    .getOrCreate()

df = spark.read.csv("input.csv",header=True,sep="|");
from pyspark import SparkContext
from pyspark.sql import SQLContext
import pandas as pd

sc = SparkContext('local','example') 
sql_sc = SQLContext(sc)

p_df= pd.read_csv('file.csv')  # assuming the file contains a header
# p_df = pd.read_csv('file.csv', names = ['column 1','column 2']) # if no header
s_df = sql_sc.createDataFrame(pandas_df)
sc.textFile("file.csv") \
    .map(lambda line: line.split(",")) \
    .filter(lambda line: len(line)>1) \
    .map(lambda line: (line[0],line[1])) \
    .collect()
sc.textFile("input.csv") \
    .map(lambda line: line.split(",")) \
    .filter(lambda line: len(line)<=1) \
    .collect()
spark.read.csv(
    "input.csv", header=True, mode="DROPMALFORMED", schema=schema
)
(spark.read
    .schema(schema)
    .option("header", "true")
    .option("mode", "DROPMALFORMED")
    .csv("input.csv"))

Read CSV file with known structure

from pyspark.sql.types import StructType, StructField
from pyspark.sql.types import DoubleType, IntegerType, StringType

schema = StructType([
    StructField("A", IntegerType()),
    StructField("B", DoubleType()),
    StructField("C", StringType())
])

(sqlContext
    .read
    .format("com.databricks.spark.csv")
    .schema(schema)
    .option("header", "true")
    .option("mode", "DROPMALFORMED")
    .load("input.csv"))

Build an Fullstack Development Environment

In this post, we will describe how to setup a modern development environment, so that you can work with such tools as Git, GoGS, Jenkins, CI/CD, Unittesting using languages like Python, Groovy, Bash Scripting.

Introduction

Working as a software developer today poses great challenges to know-how, especially due to the large number of existing products and technologies.

It is not easy to be always up to date.

The first step to achieve this is to have your own software environment in which you can work and play with the necessary products.

This post describes how such an environment can be built for a full stack developer. Components of the environment are:

Jenkins | Build and Deploy a Groovy App

Introduction

Using Jenkins as an automation server for your development, you can automate such repeating tasks as testing and deploying your app.

Starting with a sample Groovy App (a simple calculator) with tests, you will learn how to integrate your app in Jenkins and build a pipeline, so that Jenkins runs the desired tasks every time, you change the code.

Prepare the sources

Clone the sample repository from Github.

You should clone the demo repository into you demo account, because you may change some file during this post., and you will not get write permissions for the demo repository.

Also, clone the repository to your local machine to see what our demo app looks like.

$ git clone https://github.com/jenkins-toolbox/SampleApp_GroovyCalculator
Cloning into 'SampleApp_GroovyCalculator'...
remote: Enumerating objects: 194, done.
remote: Counting objects: 100% (194/194), done.
remote: Compressing objects: 100% (138/138), done.
remote: Total 194 (delta 44), reused 137 (delta 23), pack-reused 0
Receiving objects: 100% (194/194), 93.40 KiB | 817.00 KiB/s, done.
Resolving deltas: 100% (44/44), done.

Go into the new create folder

$ cd SampleApp_GroovyCalculator/
$ ls
Jenkinsfile      README.md        bin              build.gradle     gradlew          src
Makefile         SampleCalculator build            gradle           settings.gradle

The first task, Jenkins will do in our pipeline: build your app

$ ./gradlew build

Because it’s the first time you start gradlew, the required software will be downloaded:

First: the current Gradle Version (Gradle is the Build Tool used by Groovy Projects)

Downloading https://services.gradle.org/distributions/gradle-6.2.1-bin.zip
………10%………20%………30%……….40%………50%………60%……….70%………80%………90%……….100%

Welcome to Gradle 6.2.1!

Here are the highlights of this release:
 - Dependency checksum and signature verification
 - Shareable read-only dependency cache
 - Documentation links in deprecation messages

For more details see https://docs.gradle.org/6.2.1/release-notes.html

Starting a Gradle Daemon, 2 stopped Daemons could not be reused, use --status for details

After this, your app will be tested

> Task :test

Calculator02Spec > two plus two should equal four PASSED

Calculator01Spec > add: 2 + 3 PASSED

Calculator01Spec > subtract: 4 - 3 PASSED

Calculator01Spec > multiply: 2 * 3 PASSED

BUILD SUCCESSFUL in 34s
5 actionable tasks: 5 executed

Perform the build again

No download is required. The build is much quicker.

$ ./gradlew build

BUILD SUCCESSFUL in 1s
5 actionable tasks: 5 up-to-date

Now, test our app:

./gradlew clean test

> Task :test

Calculator02Spec > two plus two should equal four PASSED

Calculator01Spec > add: 2 + 3 PASSED

Calculator01Spec > subtract: 4 - 3 PASSED

Calculator01Spec > multiply: 2 * 3 PASSED

BUILD SUCCESSFUL in 4s
5 actionable tasks: 5 executed

Create a Jenkins Pipeline

Start by clicking on the BlueOcean menu item.

Hint: Blue Ocean is not installed with the default Jenkins installation.

You have to install the corresponding Plugins.

Select Manage JenkinsManage Plugins.

Then, select the tab Available and enter in the Filter box: Blue Ocean.

Install all plugins, that will be listed.

Next: Click on the New Pipeline to create your first Pipeline

Use the Item GitHub to specify, where our code is stored

Next, use your GitHub account.

Be sure, that you cloned the demo repository

Next, we select the demo repository SampleApp_GroovyCalculator

Click on Create Pipeline and after a few seconds, the pipeline is created.

Immediately after creating the pipeline, Jenkins is starting the pipeline and all steps included.

If everything went well, you see a positive status

Now, click on the pipeline (e.g. the text master or the status icon) and you will see the pipeline with all steps and their corresponding state.

If you, want to edit the pipeline, for example to add another step, like on the pencil in the header.

Click on Cancel to leave the Pipeline editor.

Hint: If you click on Save, all changes are pushed back to the repository and Jenkins starts the Pipeline again.

Run the Pipeline

If you want to run your pipeline, click on the rerun icon for your pipeline

Jenkins | Cookbook

Working with VS Code

Validate Jenkins File

Install VS Code Plugin Jenkins Pipeline Linter Connector

Add configuration in .vscode/settings.json

"jenkins.pipeline.linter.connector.crumbUrl": "<JENKINS_URL>/crumbIssuer/api/xml?xpath=concat(//crumbRequestField,%22:%22,//crumb)",
"jenkins.pipeline.linter.connector.user": "<USERNAME>",
"jenkins.pipeline.linter.connector.pass": "<PASSWORD>",
"jenkins.pipeline.linter.connector.url": "<JENKINS_URL>/pipeline-model-converter/validate",

Replace <USERNAME>, <PASSWORD> and <JENKINS_URL> with your values, for example

"jenkins.pipeline.linter.connector.crumbUrl": "http://localhost:8080/crumbIssuer/api/xml?xpath=concat(//crumbRequestField,%22:%22,//crumb)",
"jenkins.pipeline.linter.connector.user": "admin",
"jenkins.pipeline.linter.connector.pass": "secret",
"jenkins.pipeline.linter.connector.url": "http://localhost:8080/pipeline-model-converter/validate",

Working with Jenkins Client (CLI)

Download Client

wget localhost:8080/jnlpJars/jenkins-cli.jar

Working with Plugins

Create aPlugin

mkdir SamplePlugin
cd SamplePlugin
mvn -U archetype:generate -Dfilter="io.jenkins.archetypes:"
mvn -U archetype:generate -Dfilter="io.jenkins.archetypes:global-configuration-plugin"
[INFO] Scanning for projects...
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-metadata.xml
Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/mojo/maven-metadata.xml
Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-metadata.xml (14 kB at 32 kB/s)
Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/mojo/maven-metadata.xml (20 kB at 44 kB/s)
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-archetype-plugin/maven-metadata.xml
Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-archetype-plugin/maven-metadata.xml (918 B at 18 kB/s)
[INFO]
[INFO] ------------------< org.apache.maven:standalone-pom >-------------------
[INFO] Building Maven Stub Project (No POM) 1
[INFO] --------------------------------[ pom ]---------------------------------
[INFO]
[INFO] >>> maven-archetype-plugin:3.1.2:generate (default-cli) > generate-sources @ standalone-pom >>>
[INFO]
[INFO] <<< maven-archetype-plugin:3.1.2:generate (default-cli) < generate-sources @ standalone-pom <<<
[INFO]
[INFO]
[INFO] --- maven-archetype-plugin:3.1.2:generate (default-cli) @ standalone-pom ---
[INFO] Generating project in Interactive mode
[INFO] No archetype defined. Using maven-archetype-quickstart (org.apache.maven.archetypes:maven-archetype-quickstart:1.0)
Choose archetype:
1: remote -> io.jenkins.archetypes:global-configuration-plugin (Skeleton of a Jenkins plugin with a POM and an example piece of global configuration.)
Choose a number or apply filter (format: [groupId:]artifactId, case sensitive contains): : 1
Choose io.jenkins.archetypes:global-configuration-plugin version:
1: 1.2
2: 1.3
3: 1.4
4: 1.5
5: 1.6
Choose a number: 5:
[INFO] Using property: groupId = unused
Define value for property 'artifactId': com.examples.jenkins.plugins
Define value for property 'version' 1.0-SNAPSHOT: :
[INFO] Using property: package = io.jenkins.plugins.sample
Confirm properties configuration:
groupId: unused
artifactId: com.examples.jenkins.plugins
version: 1.0-SNAPSHOT
package: io.jenkins.plugins.sample
 Y: : y
[INFO] ----------------------------------------------------------------------------
[INFO] Using following parameters for creating project from Archetype: global-configuration-plugin:1.6
[INFO] ----------------------------------------------------------------------------
[INFO] Parameter: groupId, Value: unused
[INFO] Parameter: artifactId, Value: com.examples.jenkins.plugins
[INFO] Parameter: version, Value: 1.0-SNAPSHOT
[INFO] Parameter: package, Value: io.jenkins.plugins.sample
[INFO] Parameter: packageInPathFormat, Value: io/jenkins/plugins/sample
[INFO] Parameter: version, Value: 1.0-SNAPSHOT
[INFO] Parameter: package, Value: io.jenkins.plugins.sample
[INFO] Parameter: groupId, Value: unused
[INFO] Parameter: artifactId, Value: com.examples.jenkins.plugins
[INFO] Project created from Archetype in dir: /Users/Shared/CLOUD/Kunde.BSH/workspace/SamplePlugin_Config/com.examples.jenkins.plugins
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  45.525 s
[INFO] Finished at: 2020-03-01T17:28:27+01:00
[INFO] ------------------------------------------------------------------------

Verify Plugin

 cd com.examples.jenkins.plugins
mvn verify

Run Plugin

mvn hpi:run

Working with Groovy Scripts

Include a common groovy script in Jenkins file

1: Create a common.groovy file with function as needed

def mycommoncode() {
}

2: In the main Jenkinfile load the file and use the function as shown below

node{ 
   def common = load “common.groovy”
   common.mycommoncode()
}

Basic example of Loading Groovy scripts

File example.groovy

def example1() {
 println 'Hello from example1' 
}
def example2() {
 println 'Hello from example2'
}

The example.groovy script defines example1 and example2 functions before ending with return this. Note that return this is definitely required and one common mistake is to forget ending the Groovy script with it.Jenkinsfile

def code node('java-agent') { 
    stage('Checkout') { checkout scm } 
    stage('Load') { code = load 'example.groovy' } 
    stage('Execute') { code.example1() }
} 

code.example2()

Processing Github JSON from Groovy

In this demo, we first show how to process JSON response from Github API in Groovy.Processing JSON from Github

String username = System.getenv('GITHUB_USERNAME') 
String password = System.getenv('GITHUB_PASSWORD') 
String GITHUB_API = 'https://api.github.com/repos' String repo = 'groovy' 
String PR_ID = '2' // Pull request ID 
String url = "${GITHUB_API}/${username}/${repo}/pulls/${PR_ID}" 

println "Querying ${url}" 

def text = url.toURL().getText(requestProperties: ['Authorization': "token ${password}"]) 
def json = new JsonSlurper().parseText(text) 

def bodyText = json.body // Check if Pull Request body has certain text if ( bodyText.find('Safari') ) {
     println 'Found Safari user' }

The equivalent bash command for retrieving JSON response from Github API is as follows:Equivalent bash command

// Groovy formatted string 
String cmd = "curl -s -H \"Authorization: token ${password}\" ${url}" 
// Example String 
example = 'curl -s -H "Authorization: token XXX" https://api.github.com/repos/tdongsi/groovy/pulls/2'

Processing Github JSON from Jenkinsfile

Continuing the demo from the last section, we now put the Groovy code into a callable function in a script called “github.groovy”. Then, in our Jenkinsfile, we proceed to load the script and use the function to process JSON response from Github API.github.groovy

import groovy.json.JsonSlurper
def getPrBody(String githubUsername, String githubToken, String repo, String id) {
   String GITHUB_API = 'https://api.github.com/repos' 
   String url = "${GITHUB_API}/${githubUsername}/${repo}/pulls/${id}" 

   println "Querying ${url}"

def text = url.toURL().getText(requestProperties: ['Authorization': "token ${githubToken}"])
def json = new JsonSlurper().parseText(text) 
def bodyText = json.body return bodyText } 
return this

Jenkinsfile

def code node('java-agent') { 
    stage('Checkout') { checkout scm }
    stage('Load') { code = load 'github.groovy' } 
    stage('Execute') {

Best Practice

Build a Dashboard with Django and Bootstrap: Part 3

This is Part 3 of Building a Dashboard with Django and Bootstrap:

Introduction

If you follow the first part of this blog topic, you have a running Django dashboard.

But, the content ist still static. Lets review the current state:

Prepare our Django project

Right now, the whole content of our Django project is provided by the dashboard template

dashboard/template/site/base.html

Looking at our web site, you will see the different side menu items. So, intentionally, our web site should display different pages. And each page should provide the dynamic content.

The final goal of this part is to change our web app, so that each side item navigates us to a different page. For this, we have to take care about two things:

  • Navigation: how to we get to another page in our app
  • Project Structure: where to place the required components for each page

Basics of Navigation

Navigation usually is the process of getting from one page to another by clicking on a link.

So, we need to things:

  • the source page, containing the link
  • the destination page
  • the link, pointing to the destination page

Let’s take a look into the site template with the side menu:

The corresponding code in the side template is

<div id="collapseTwo" class="collapse" aria-labelledby="headingTwo">
	<div class="bg-white py-2 collapse-inner rounded">
		<h6 class="collapse-header">Custom Components:</h6>
		<a class="collapse-item" href="buttons.html">Buttons</a>
		<a class="collapse-item" href="cards.html">Cards</a>
	</div>
</div>

This is the result:

Linking to a html page is not possible, because Django does not work with html pages. Navigation in Django works with urls (in urls.py) and views in (views.py).

We must replace the html link tag (<a href="buttons.html">) with an Django-conform code. Read here for more details and the basics.

The idea behind the navigation is

Define the needed links/buttons
Give each link a name“buttons”
Define, which view to call for this linkcomponents/buttons/views.py
Tell Django, how to insert this link in a html page<a href="{% url 'buttons' %}">Buttons</a>

With this in mind, we change our site template for the side navigation (e. g. for the components menu):

But, if you save the template and try to view the web page, you will see this error:

We missed to tell Django, what to do when the corresponding link for this name is requested. We have to tell Django to use the view defined in buttons/views.py to generate the resulting view/page.

So, change the global url mapping file dashboard/urls.py

import dashboard.apps.components.buttons.views as ButtonsViews
import dashboard.apps.components.cards.views as CardsViews

urlpatterns = [
    path('', include('dashboard.apps.urls')),

    path('buttons', 	ButtonsViews.IndexView.as_view(),   name='buttons'),
    path('cards',       CardsViews.IndexView.as_view(),     name='cards'),

    path('admin/', admin.site.urls),
]

dashboard/apps/components/buttons/views.py

from django.shortcuts import render
from django.views import generic

class IndexView(generic.TemplateView):
    template_name = 'buttons/base.html'

dashboard/apps/components/cards/views.py

from django.shortcuts import render
from django.views import generic

class IndexView(generic.TemplateView):
    template_name = 'cards/base.html'

dashboard/apps/components/cards/templates/cards/base.html

{% extends 'site/base.html' %}
{% load static %}

{% block content %}
<h1 style="text-align: center">CARDS</h1>
{% endblock content %}

dashboard/apps/components/cards/templates/buttons/base.html

{% extends 'site/base.html' %}
{% load static %}

{% block content %}
<h1 style="text-align: center">BUTTONS</h1>
{% endblock content %}

Save everything and view at the resulting page

What happens, from showing the page, clicking on the link until you see the final result:

  • Django create the side menu item Cards
  • with the corresponding link localhost:9000/cards
  • which will be the destination page, if you click on the link
  • and finally, Django creates the desired page (through view.py)

Namespaces and structure

Now, think about the names, e. g. buttons and cards in our Components.

Your project is getting bigger and you plan to add an additional page info with general information for both Components and Utilities menu.

So, you will have to additional files, for example

  • dashboard/apps/components/info/views.py
  • dashboard/apps/utilities/info/views.py

The corresponding url mapping (urls.py) could look like this:

import dashboard.apps.components.info.views as ComponentInfoViews
import dashboard.apps.utilities.info.views as UtilitiesInfoViews

urlpatterns = [
    path('', include('dashboard.apps.urls')),

    path('info', ComponentInfoViews.IndexView.as_view(), name='info'),
    path('info', UtilitiesInfoViews.IndexView.as_view(), name='info'),

Two pages with the same name (info) in different views.py!

Of course, we could choose different names and link (component_info, utilities_info), but this not a good programming style.

We will choose a more modern way of programming

  • Spread the responsibility over separate, independent modules
  • Name these modules with different names

What does this mean? We will have

  • a separate module frontend, only responsible for the start page (frontend)
  • a separate module components, responsible for all components
  • a separate module utilities, responsible for all utilities
  • a separate module pages, responsible for all pages

Resulting folder structure and file content

File dashbard/urls.py

urlpatterns = [
    path('', include('dashboard.apps.urls')),
    path('admin/',	admin.site.urls),
]

File dashbard/apps/urls.py

from django.urls import path
from django.urls.conf import include

from dashboard.apps.frontend.views import IndexView

app_name = 'app'

urlpatterns = [
    path('', IndexView.as_view(), name='index'),
    path('pages/', include('dashboard.apps.pages.urls')),
    path('components/', include('dashboard.apps.components.urls')),
    path('utilities/', include('dashboard.apps.utilities.urls')),
]

File dashbard/apps/components/urls.py

from django.urls import path

import dashboard.apps.components.buttons.views as ButtonsViews
import dashboard.apps.components.cards.views as CardsViews

app_name = 'components'

urlpatterns = [
    path('', ButtonsViews.IndexView.as_view(), name='index'),
    path('buttons/', ButtonsViews.IndexView.as_view(), name='buttons'),
    path('cards/', CardsViews.IndexView.as_view(), name='cards'),
]

File dashbard/apps/utilities/urls.py

from django.urls import path

import dashboard.apps.utilities.colors.views as ColorsViews
import dashboard.apps.utilities.borders.views as BordersViews
import dashboard.apps.utilities.animations.views as AnimationsViews
import dashboard.apps.utilities.others.views as OthersViews

app_name = 'utilities'

urlpatterns = [
    path('', BordersViews.IndexView.as_view(), name='index'),
    path('animations/', AnimationsViews.IndexView.as_view(), name='animations'),
    path('borders/', BordersViews.IndexView.as_view(), name='borders'),
    path('colors/', ColorsViews.IndexView.as_view(), name='colors'),
    path('others/', OthersViews.IndexView.as_view(), name='others'),
]

File dashbard/apps/pages/urls.py

from django.urls import path

import dashboard.apps.pages.blank.views as BlankViews
import dashboard.apps.pages.login.views as LoginViews
import dashboard.apps.pages.pagenotfound.views as PageNotFoundViews
import dashboard.apps.pages.password.views as PasswordViews
import dashboard.apps.pages.register.views as RegisterViews
import dashboard.apps.pages.charts.views as ChartsViews
import dashboard.apps.pages.tables.views as TablesViews

app_name = 'pages'

urlpatterns = [
    path('', ChartsViews.IndexView.as_view(), name='index'),
    path('blank', BlankViews.IndexView.as_view(), name='blank'),
    path('charts', ChartsViews.IndexView.as_view(), name='charts'),
    path('login', LoginViews.IndexView.as_view(), name='login'),
    path('pagenotfound', PageNotFoundViews.IndexView.as_view(), name='pagenotfound'),
    path('password', PasswordViews.IndexView.as_view(), name='password'),
    path('register', RegisterViews.IndexView.as_view(), name='register'),
    path('tables', TablesViews.IndexView.as_view(), name='tables'),
]

Let’s finally check the namespace structure:

$ find . -name urls.py
./dashboard/urls.py
./dashboard/apps/utilities/urls.py
./dashboard/apps/components/urls.py
./dashboard/apps/urls.py
./dashboard/apps/pages/urls.py

We create three levels for our namespaces:

Djane URL FileNamespace
./dashboard/urls.py
./dashboard/apps/urls.pyapp
./dashboard/apps/utilities/urls.py
./dashboard/apps/components/urls.py
./dashboard/apps/pages/urls.py
app:utilities
app:components
app:pages

These namespaces must be used in the template files, for example:

<a href="{% url 'app:components:buttons' %}" class="collapse-item" >Buttons</a>
<a href="{% url 'app:components:cards' %}" class="collapse-item" >Cards</a>
<a class="collapse-item" href="{% url 'app:utilities:colors' %}">Colors</a>
<a class="collapse-item" href="{% url 'app:utilities:borders' %}">Borders</a>
<a class="collapse-item" href="{% url 'app:utilities:animations' %}">Animations</a>
<a class="collapse-item" href="{% url 'app:utilities:others' %}">Other</a>

Install the Django Extensions for additional commands:

pip install django-extensions

Add Django Extensions to the INSTALLED_APPS

INSTALLED_APPS = [
    ...

    'django_extensions'
]

Show URLs and Namespaces (only for out apps, admin urls are removed)

python3 manage.py show_urls

Preparing required components and pages

In summary, these are the steps to create the desired folder structure:

mkdir -p dashboard/apps/components/buttons/templates/buttons
mkdir -p dashboard/apps/components/cards/templates/cards
mkdir -p dashboard/apps/pages/blank/templates/blank
mkdir -p dashboard/apps/pages/charts/templates/charts
mkdir -p dashboard/apps/pages/login/templates/login
mkdir -p dashboard/apps/pages/pagenotfound/templates/pagenotfound
mkdir -p dashboard/apps/pages/password/templates/password
mkdir -p dashboard/apps/pages/register/templates/register
mkdir -p dashboard/apps/pages/tables/templates/tables
mkdir -p dashboard/apps/utilities/animations/templates/animations
mkdir -p dashboard/apps/utilities/borders/templates/borders
mkdir -p dashboard/apps/utilities/colors/templates/colors
mkdir -p dashboard/apps/utilities/others/templates/others
python3 manage.py startapp buttons dashboard/apps/components/buttons
python3 manage.py startapp cards dashboard/apps/components/cards
python3 manage.py startapp blank dashboard/apps/pages/blank
python3 manage.py startapp charts dashboard/apps/pages/charts
python3 manage.py startapp login dashboard/apps/pages/login
python3 manage.py startapp pagenotfound dashboard/apps/pages/pagenotfound
python3 manage.py startapp password dashboard/apps/pages/password
python3 manage.py startapp register dashboard/apps/pages/register
python3 manage.py startapp tables dashboard/apps/pages/tables
python3 manage.py startapp animations dashboard/apps/utilities/animations
python3 manage.py startapp borders dashboard/apps/utilities/borders
python3 manage.py startapp colors dashboard/apps/utilities/colors
python3 manage.py startapp others dashboard/apps/utilities/others
echo "{% extends 'site/base.html' %}

{% block    content %}
{% endblock content %}"> base.html
cp base.html dashboard/apps/components/buttons/templates/buttons
cp base.html dashboard/apps/components/cards/templates/cards
cp base.html dashboard/apps/pages/blank/templates/blank
cp base.html dashboard/apps/pages/charts/templates/charts
cp base.html dashboard/apps/pages/login/templates/login
cp base.html dashboard/apps/pages/pagenotfound/templates/pagenotfound
cp base.html dashboard/apps/pages/password/templates/password
cp base.html dashboard/apps/pages/register/templates/register
cp base.html dashboard/apps/pages/tables/templates/tables
cp base.html dashboard/apps/utilities/animations/templates/animations
cp base.html dashboard/apps/utilities/borders/templates/borders
cp base.html dashboard/apps/utilities/colors/templates/colors
cp base.html dashboard/apps/utilities/others/templates/others
rm base.html

Each of the folders has the same structure, for example the buttons component. We will delete some unnecessary files

Replacing Projects with dynamic data

Replacing the static parts with dynamic content could be achieved by the following approach:

  • Identify the dynamic parts
  • Move these parts from the site template into the view template base.html of the component
  • Modify frontend view.py to generate dynamic content from data

The steps are the same for all components (all items of the side menu).

Find the

Identify dynamic parts in template

Create templates for side menu pages

For every side menu item, their is a corresponding html file in the install folder of the sb-admin-2 template:

Remember the environment variable we create in part 1 for the start of our project

DASHBOARD_ROOT=$(pwd)
cd $DASHBOARD_ROOT
find dashboard/apps install/sb-admin-2 -name *.html

Each template file base.html has a corresponding html file unter sb-admin-2. Look at the following table to find the mapping:

apps/components/buttons/templates/buttons/base.htmlsb-admin-2/buttons.html
apps/components/cards/templates/cards/base.htmlsb-admin-2/cards.html
apps/pages/blank/templates/blank/base.htmlsb-admin-2/blank.html
apps/pages/charts/templates/charts/base.htmlsb-admin-2/charts.html
apps/pages/login/templates/login/base.htmlsb-admin-2/login.html
apps/pages/pagenotfound/templates/pagenotfound/base.htmlsb-admin-2/404.html
apps/pages/password/templates/password/base.htmlsb-admin-2/forgot-password.html
apps/pages/register/templates/register/base.htmlsb-admin-2/register.html
apps/pages/register/templates/tables/base.htmlsb-admin-2/tables.html
apps/utilities/animations/templates/animations/base.htmlsb-admin-2/utilities-animation.html
apps/utilities/borders/templates/borders/base.htmlsb-admin-2/utilities-border.html
apps/utilities/colors/templates/colors/base.htmlsb-admin-2/utilities-color.html
apps/utilities/others/templates/others/base.htmlsb-admin-2/utilities-html

Each template base.html should have the following content:

{% extends 'site/base.html' %}

{% block    content %}
{% endblock content %}

And each corresponding view.py file should have the following content, only the template_name should be different (the name of the template base.html file)

from django.views import generic

class IndexView(generic.TemplateView):
    template_name = 'buttons/base.html'

So, for each template file, we have to

  • locate the corresponding html file from the install folder (see table above)
  • copy the content between these tags to the template file:
        <!-- Begin Page Content -->
        <div class="container-fluid">
        ....
        </div>
        <!-- /.container-fluid -->

Build a Dashboard with Django and Bootstrap: Part 2

This is Part 2 of Building a Dashboard with Django and Bootstrap:

Introduction

If you follow the first part of this blog topic, you have a running Django dashboard.

But, unfortunately, the content is still static. Let’s review the current state:

Perfect. We are done with the basic setup.

Still, some work to do, because our dashboard is only a static dashboard. All content is programmed in the dashboard template file dashboard/templates/site/sb-admin-2/base.html

For example, look at the cards with the earnings at the top:

To achieve a more dynamic content, we need to move the desired parts of the dashboard from the template file to the frontend view file.

We will do this by following these steps:

  • Identify the dynamic parts
  • Move these parts from the template into for frontend view template index.html
  • Modify frontend view.py to generate dynamic content from data

Identify dynamic parts

How to find the parts, which are dynamic.

One way is to ask:

  • Which parts should be on every page (unchanged) and
  • What should change on every page

You mostly get the same answers by the question:

  • What are the main components of a web page (including navigation and content)

For answer the first question, take a look at the current page and “name” the areas:

So, these “names” are also the answer for the 3. Question:

  • sidemenu
  • top bar
  • content

And maybe you find additional “names”

  • header
  • footer
  • top menu
  • left and right sidebar

Find identified parts in template

Next step is, to find the identified parts in our dashboard template

dashboard/templates/site/sb-admin-2/base.html

This is an easy step, because the developer of the SB Admin 2 template documented their template well:

Looking into the code of the template, you will find comment lines describing the content:

  • <!-- Sidebar -->
  • <!-- Topbar -->
  • <!-- Top Search -->
  • <!-- Top Navbar -->
  • <!-- Begin Page Content-->

So, it is obvious what do to next:

  • get the dynamic part (lines under)<!-- Begin Page Content-->
    the green box in the following image
  • move it to the frontend template
  • place some information in the dashboard template, that the real content should be displayed here
    the blue curly braces in the following image

This is the way, the template system of django works:

Let’s explain this with a simple example: the page title

We declare a title (which should be considered as the default title). And in the frontend page, we declare the title for this page (the frontend page).

To achieve this, we have to tell our template system the following:

Now, we do the same with the content:

Looking at our resulting page, nothing changes. This is the desired result, but how could we be sure, that we really change the structure?

Well, let’s make a test and try to include a different content in the dashboard template.

Change the lines, where we include the content into this:

{% block content_missing %}
MISSING CONTENT
{% endblock content_missing %}

Did you notice the other name of the content: content_missing?

Change the template, save the file and have a look at the result:

Change content back, so your template is working again:

{% block content %}
MISSING CONTENT
{% endblock content %}

The final step in Part 3 will be replacing all static content of the dashboard with dynamic content.

Build a Dashboard with Django and Bootstrap: Part 1

This is Part 1 of Building a Dashboard with Django and Bootstrap:

Introduction

Building a complete web app isn’t always an easy task. Designing and Implementing on both sides (backend and front-end) requires mostly a lot of knowledge. So, why don’t using tools or framework, which helps and makes our life easier. Django is one of these frameworks:

Django is a high-level Python Web framework that encourages rapid development and clean, pragmatic design. Built by experienced developers, it takes care of much of the hassle of Web development

So, let’s get started.

Create project

For subsequent steps, we will remember our starting directory

DASHBOARD_ROOT=$(pwd)
echo $DASHBOARD_ROOT
... here you will see your current folder...

First step is to create our main Django project

django-admin startproject dashboard
cd dashboard
python manage.py migrate
python manage.py runserver 8080
python manage.py startapp app_base

View current project in browser

Create first apps

mkdir -p dashboard/apps/core
python manage.py  startapp Core dashboard/apps/core

mkdir -p dashboard/apps/frontend
python manage.py createapp Frontend dashboard/apps/frontend

Add new apps to project

Modify dashboard/settings.py

INSTALLED_APPS = [
   ...

    'dashboard.apps.core',
    'dashboard.apps.frontend',
]

Modify dashboard/urls.py

from django.urls import path

import dashboard.apps.frontend.views as views

urlpatterns = [
    path('', views.IndexView.as_view(), name='index'),
    path('admin/', admin.site.urls),
]

Create view

Modify view in apps/frontend/views.py

from django.shortcuts import render
from django.views import generic


class IndexView(generic.TemplateView):
    """
    IndexView:
    """
    module = 'indexView'
    template_name = 'frontend/index.html'

Create template for frontend View

Create template file dashboard/apps/frontend/templates/frontend/index.html

<h1>
Let's get started
</h1>

View current project in browser

Current folder Structure

So far, we have the following folder structure

Prepare for administrate your project

Create admin user

python manage.py createsuperuser
Username (leave blank to use 'user'): admin
Email address: admin@localhost
Password: 
Password (again): 
Superuser created successfully.

Add Bootstrap support

Download requires files for Bootstrap, jQuery ad PopperJS

mkdir -p install/bootstrap
wget  -q https://github.com/twbs/bootstrap/archive/v4.4.1.zip -O install/bootstrap/4.4.1.zip
wget  -q https://github.com/twbs/bootstrap/releases/download/v4.4.1/bootstrap-4.4.1-dist.zip -O install/bootstrap/4.4.1-dist.zip
rm -rf   dashboard/static/vendor/bootstrap/bootstrap-4.4.1-dist
unzip -q install/bootstrap/4.4.1-dist.zip -d       dashboard/static/vendor/bootstrap/
rm -rf   dashboard/static/vendor/bootstrap/4.4.1
mv       dashboard/static/vendor/bootstrap/bootstrap-4.4.1-dist dashboard/static/vendor/bootstrap/4.4.1
mkdir -p dashboard/static/vendor/jquery/3.4.1/js
wget -q https://code.jquery.com/jquery-3.4.1.min.js      -O dashboard/static/vendor/jquery/3.4.1/js/jquery.min.js
wget -q https://code.jquery.com/jquery-3.4.1.slim.min.js -O dashboard/static/vendor/jquery/3.4.1/js/jquery.slim.min.js
make install-popperjs
mkdir -p dashboard/static/vendor/popper/2.0.6/js
wget  -q https://unpkg.com/@popperjs/core@2/dist/umd/popper.min.js          -O dashboard/static/vendor/popper/2.0.6/js/popper.min.js
wget  -q https://unpkg.com/@popperjs/core@2.0.6/dist/umd/popper.min.js.map  -O dashboard/static/vendor/popper/2.0.6/js/popper.min.js.map

Create site templates in dashboard/templates/site

Add templates path to settings

File dashboard/settings.py

TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': [
            BASE_DIR + '/dashboard/templates',
        ],

Add static path to settings

File dashboard/settings.py

STATIC_URL = '/static/'
STATICFILES_DIRS = [
    os.path.join(BASE_DIR, APP_NAME + '/static')
]

Modify frontend view template

File dashboard/apps/frontend/templates/index.html

{% extends 'site/base.html' %}
{% load static %} {% block title %}Dashboard with Django and Bootstrap{% endblock title %}
{% block nav-style %}mkt-nav{% endblock nav-style %}

{% block content %}
<div class="jumbotron">
	<div class="container"></div>
		FRONTEND
	<div class="container full"></div>
</div>
{% endblock content %}

View current status of project

Add dashboard from an existing template

For a first start, we will use this sb-admin-2 dashboard template from here:

Download required files

Bootstrap templates consist of at least 3 different types of files

  • main index.html page, responsible for collection all elements and set up your page
  • CSS files defining the style of your page
  • JavaScript files, containing needed frameworks and code

So, first start by downloading the sample template from here. Be sure, you start in our project root folder:

cd $DASHBOARD_ROOT
git clone https://github.com/BlackrockDigital/startbootstrap-sb-admin-2 install/sb-admin-2

Next, find out, what we need for our template in addition to the file index.html

cd install/sb-admin-2
grep -E "<(link|script)" index.html 
  <link href="vendor/fontawesome-free/css/all.min.css" rel="stylesheet" type="text/css">
  <link href="https://fonts.googleapis.com/css?family=Nunito:200,200i,300,300i,400,400i,600,600i,700,700i,800,800i,900,900i" rel="stylesheet">
  <link href="css/sb-admin-2.min.css" rel="stylesheet">
  <script src="vendor/jquery/jquery.min.js"></script>
  <script src="vendor/bootstrap/js/bootstrap.bundle.min.js"></script>
  <script src="vendor/jquery-easing/jquery.easing.min.js"></script>
  <script src="js/sb-admin-2.min.js"></script>
  <script src="vendor/chart.js/Chart.min.js"></script>
  <script src="js/demo/chart-area-demo.js"></script>
  <script src="js/demo/chart-pie-demo.js"></script>

That’s a lot of additional files.

To keep the file structure consistent, these files should be stored under the dash/board/static folder (same as the bootstrap files before).

To achieve this, there are two possibilities:

  • Create desired folder and copy each of the source files to the destination folder
  • Copy the complete static folder from the github repository for this post.

We use the second option to keep the focus on creating our dashboard.

So, first, clone the repository:

cd $DASHBOARD_ROOT/install
git clone https://github.com/rgoestenmeier/Dashboard_Django-Bootstrap

Then, copy the requred files

cd $DASHBOARD_ROOT
cp -R install/Dashboard_Django-Bootstrap/dashboard/static dashboard
cp -R install/Dashboard_Django-Bootstrap/dashboard/templates dashboard

Having everything needed for the dashboard template, the next step is to modify the front-end template

File dashboard/apps/frontend/templates/frontend/index.html

{% extends 'site/sb-admin-2/base.html' %}
{% load static %} {% block title %}Dashboard with Django and Bootstrap{% endblock title %}

{% block content %}
{% endblock content %}

View current project in browser

Perfect. We are done with the basic setup.

Still some work to do, because our dashboard is only a static dashboard. All content is programmed in the dashboard template file dashboard/templates/site/sb-admin-2/base.html

For example, look at the cards with the earnings at the top:

To achieve a more dynamic content, we need to move the desired parts of the dashboard from the template file to the front-end view file.

This will be described in the next step: Part 2: Prepare for dynamic content

Working with State Management and Angular

Introduction

One of the most challenging things in software development is state management.

Motivation

What is a state (or application state)? Theoretically, it is the entire memory of the application, but, typically, it is the data received via API calls, user inputs, presentation UI State, app preferences, etc.

It is the data that can differentiate two instances of the same application.

One example of application state would be a list of customers or products maintained in an application.

Problem (we’re trying to solve)

Think of an application using a list of data (products, customers, …). This list is the state that we are trying to manage.

Some API calls and user inputs could change the state ( i.e. the list ) by adding or removing items/entries. The state change should be reflected in the UI and other dependent components.

We could solve this with a global variable to hold the list and then add/remove customers from/to it and then write the code to update the UI and dependencies. But, there are many pitfalls in that design which are not the focus of this article.

Solutions

Currently there are several state management libraries for Angular apps: NGRX, NGXS or Akita.

RxJS – Reactive Extensions Library for JavaScript

RxJS is a library for reactive programming using Observables, to make it easier to compose asynchronous or callback-based code. This project is a rewrite of Reactive-Extensions/RxJS with better performance, better modularity, better debuggable call stacks, while staying mostly backwards compatible, with some breaking changes that reduce the API surface

Vorschau(öffnet in neuem Tab)

@ngrx/store

Store is RxJS powered state management for Angular applications, inspired by Redux. Store is a controlled state container designed to help write performant, consistent applications on top of Angular.

Key concepts

  • Actions describe unique events that are dispatched from components and services.
  • State changes are handled by pure functions called reducers that take the current state and the latest action to compute a new state.
  • Selectors are pure functions used to select, derive and compose pieces of state.
  • State is accessed with the Store, an observable of state and an observer of actions
NgRx State Management Lifecycle Diagram

Akita

Akita is a state management pattern, built on top of RxJS, which takes the idea of multiple data stores from Flux and the immutable updates from Redux, along with the concept of streaming data, to create the Observable Data Stores model.

Akita encourages simplicity. It saves you the hassle of creating boilerplate code and offers powerful tools with a moderate learning curve, suitable for both experienced and inexperienced developers alike.

Akita is based on object-oriented design principles instead of functional programming, so developers with OOP experience should feel right at home. Its opinionated structure provides your team with a fixed pattern that cannot be deviated from.

NGXS – State management pattern + library for Angular

NGXS is a state management pattern + library for Angular. It acts as a single source of truth for your application’s state, providing simple rules for predictable state mutations.

NGXS is modeled after the CQRS pattern popularly implemented in libraries like Redux and NgRx but reduces boilerplate by using modern TypeScript features such as classes and decorators.

Introduction (taken from here)

What is state?

State is basically everything that will define the UI that our user will be using. State could be whether a button should be visible or not, it could be the result of that button click and it could also be an Array of users that is coming from an API. State can live in different places throughout our entire application. Some state is very specific to a certain component where other state might be shared in different parts of our application. One piece of state could be a singleton instance, where a another piece of state could share the limited lifespan of a component that can be destroyed at any time.

This big variety of what state could be, how long it lives and where it comes from results in complexity that we need to manage.

What is state management?

State management is the concept of adding, updating, removing and reading pieces of state in an application. When we have deeply nested data structures and we want to update a specific part deep down in the tree, it might become complex. In that case we have state management libraries that contain a Store which helps us with state management to get rid of that complexity. A quick note, we have to be careful that these libraries don’t add complexity by overusing them.

Reactive state

Combining state management together with reactive programming can be a really nice way to develop single-page-applications. Whether our focus lies on Angular, Vue or React, combining these two principles will result in more predictable applications.

Now what has state to do with reactive programming? A piece of state can change over time, so in a way we are waiting for new state changes. That makes it asynchronous.

Let’s take this example for instance:

// false------true-----false---true...
sidebarCollapsed$ = this.state.sidebarCollapsed$

The sidebarCollapsed$ stream starts out with false, later on it becomes true and so on. This stream keeps on living. In Angular this state can be consumed with the async pipe as easy as:

<my-awesome-sidebar *ngIf="sidebarCollapsed$|async">
</my-awesome-sidebar>

The async pipe will subscribe to the sidebarCollapsed$ pass it to the component, mark it for check and will automatically unsubscribe when the component gets destroyed. Keeping state in an observer pattern is nice because we can subscribe to the changes. Oh, and did I mention it plays super nice with Angular?

We can either use a BehaviorSubject or state management frameworks that support Observables. Here are some really great ones with Observable support:

Immutability and Unidirectional data flow

Before we dive deeper in state, there are 2 important principles that we should follow when managing state. The first principle is immutability, which means that we should never mutate data directly without creating a new reference of that object. If we mutate data directly, our application becomes unpredictable and it’s really hard to trace bugs. When we work in an immutable fashion we can also take advantage of performance strategies like the ChangeDetection.OnPush from Angular or React its PureComponent.

When we use typescript we can enforce the typescript compiler to complain when we mutate data

type Foo = {
    readonly bar: string; 
    readonly baz: number; 
}

let first = {bar: 'test', baz: 1};
first.bar = 'test2'; // compilation error
first = {...first, bar: 'test2'}; // success

In the previous example we have overwritten the first instance with an entire new instance that has an updated bar property.

Arrays can be handled like this:

let arr = ['Brecht', 'Kwinten'];
arr.push('John'); // BAD: arr is mutated
arr = [...arr, 'John']; // Good, arr gets new reference

the Array prototype also has some great helper functions that we can use to enforce immutability like map() and filter() but this is not in scope for this article.

The second principle is Unidirectional data flow. In a nutshell, this means that we should never use two-way data binding on state. It is the absolute owner of that specific piece of state that is in charge of updating it (immutable of course).

Both of these principles are highly enforced by the Redux pattern.

What kind of states are there?

Router state

Often forgotten, but one of the most important pieces of state a web application can have. Putting state in the route gives us the following advantages:

  • We can use the browser navigation buttons
  • We can bookmark the state
  • We can can copy and paste the url with the state to other users
  • We don’t have to manage it, it’s always there in the route

Tip: Instead of handling modals with a userDetailModalVisible property, why not enjoy all the benefits mentioned above and bind it to a users/:userId route? Using a child router-outlet in Angular makes this a piece of cake as we can see in this snippet.

<table>
<!--contains users -->
</table>
<router-outlet>
<!-- user detail modal rendered in here -->
</router-outlet>

Component state

Every component could contain state. That state could be shared with its dumb components or could be used in the component itself. Eg: When an ItemComponent has a property selectedItems which is an array of ids, and that array is never used in other components (that aren’t children of that component), we can consider it component state. It belongs to that component, therefore the component should be responsible for it. Child components can consume that state but should never mutate it. Those components can notify their parent that is responsible for it, which could update it in an immutable way. For more information about smart and dumb components look here.

Personally, I try to avoid state management frameworks for managing component state because it’s the responsibility of that component to manage that state. There are however good reasons to use state management frameworks to manage component state:

If the state management of the component becomes a bit too complex and we don’t want to use a state management framework just yet, we could use a state reducer in the component itself.

Persisted state

Persisted state, is state that is being remembered when the user navigates between different pages. This could be whether a sidebar was collapsed or not, or when the user returns to a grid with a lot of filters and he wants them to be remembered and reapplied when he returns. Another example is a wizard with different steps, and every step needs to be persisted so the user can navigate back and forth and the last page is a result of all these steps.

Persisted state is the type of state where we typically use a state management framework for, that being said, if we don’t want to rely on an external dependency we can also manage it in a Angular service which can be a singleton that is shared throughout the entire application. If that service becomes too complex or there is a lot of state to manage, I would consider to put that state into a state management framework.

Shared state

When we are talking about shared state, we are talking about state that needs to be shared between different parts of our application. State that is being shared throughout different smart components. This means that the instance of this piece of state should live on a higher level, than the components that want to consume it.

Shared state can be managed in a state management framework like ReduxNgrxAkitaNgxs and so on, but if that state is small and simple we can also manage it manually. Let’s say that we want an Observable of an Array of countries that we need to share throughout the entire application. In Angular we could have a CountryService that fetches the countries from the API once, and then shares it throughout the entire application. For that we can use the shareReplay operator from RxJS.

export class CountryService {
    ...
    countries$ = this.httpClient.get('countries').pipe(shareReplay(1));
}

Simple right, one line of code?! For this we don’t need a state management framework, although it can also have its benefits. Some developers like to keep all their master data in a Redux store, and that’s fine. Just know that we don’t have to. I like to develop by the KISS principle (Keep ISimple Stupid) as much as possible, so I favor this approach many times. Think about the amount of lines of code we saved by this approach. Beware that every line of code we write, not only needs to be written but also maintained.

Which state needs to be managed?

Now that we know what state is, we have to ask ourselves which state needs to be managed, and where do we manage that state? In a component, singleton service or a framework (Store)?

This is the part where the strong opinions surface. I would suggest to use what works for you and your team and really think about, but here are my personal opinionated guidelines:

  • I try to avoid state management frameworks where possible. RxJS already leverages us with a lot already and I like to think KISS.
  • I try to avoid using state management frameworks to communicate with different parts in my application, I believe state is unrelated to communication.
  • When my component can handle the state and it’s not too complex, I let my component in charge of managing that state.
  • Master data like countries are exposed in a service which uses the shareReplay operator.
  • I don’t put the result of a getById API call into a store if there is no one consuming that state except for the component requesting it
  • I use a facade between my smart components and my store/services to make refactoring easier in the future.

However, there is also a popular opinion out there to put literally everything in the store which has the following advantages:

  • We can see the flow of the code in devtools
  • Consistent pattern
  • We can leverage selectors with memoization
  • Easier for realtime applications
  • Optimistic updates are easier

However, there are a few downsides as well:

  • A gigantic amount of bloat code: Bigger bundle size, more maintenance and dev time. Eg: If we would use the complete Ngrx pattern for the countries$ example we would have to write an: actionactiontypeeffect and a reducer.
  • Tightly coupled to a strong dependency that is hard to get rid of in the future
  • Generally more complex
  • The user his screen can get out of sync with the backend
  • Cache invalidation: if we add a currentUserToEdit in the store, we have to get it out when we navigate away
  • We can’t use the async pipe to cancel pending XHR requests
  • We create a distributed monolith of some sort

More to read