Web compression on Spring Boot application

Compression of JSON messages

If you look at common JSON message, the data are highly repetitive ! Each array of objects contains meta data for all object properties. If all of your messages are short you probably won’t need a compression but in typical scenario where REST services return data from database you gain a lot.

In the message of this size (220 bytes) there is no gain at all (compressed size is 250 bytes) and you lost some processor time to compress it. It is total waste of resources, so you have to limit compression above 2 kb for example.

Lets find out how much difference we make with a compression in the real world example

For the test we will get JSON message from publicly accessible site http://data.colorado.gov/resource/4ykn-tg5h.json and check the message size. The size of received message in uncompressed JSON format is  951 kb.

Now I will check in network sniffer how big this message really was : The message on the TCP level is only 170 kb ! If you look at the message payload (down right), what was really on the wire is unreadable compressed data. You can get more information here about Capsa network analyzer freeware product used in this measurement.

It means that this type of messages were compressed to less then 20% of the original size. It means at least 5 times smaller network traffic as without compression.  The compression ratio could be even higher, much higher, if your specific messages are not so full of unique data and that’s happening a lot in ERP type applications.

Enable compression in Spring boot application

Add this settings to application.properties file:

# Server compression
server.compression.enabled=true
server.compression.min-response-size=2048
server.compression.mime-types=application/json,application/xml,text/html,text/xml,text/plain

How to check if compression is enabled

Check for a specific statement about content encoding in the response header :

To inspect messages in JSON format I use Postman API development environment, free edition.

This is the link to the source code of the used project on the github.

 

Java Spring Boot project setup

You can get source code of this project from github repository “SpringBootMyApp” .

Create Spring Boot Maven project

Go to Spring Boot project generator web site and select minimal project definition with dependencies :

  • Web:  dependency for embedded tomcat server
  • jOOQ: integrated SQL query language and data model code generator
  • Flyway: database model migration tool
  • Postgresql JDBC driver

After downloading ZIP project file, unzip it to some folder and open folder in IntelliJ IDE environment.

Startup class

The project will automatically include embedded tomcat server and spring application start will configure whole application at startup.

package com.bisaga.myapp;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication(scanBasePackages = { "com.bisaga" })
public class MyappApplication {

	public static void main(String[] args) {
		SpringApplication.run(MyappApplication.class, args);
	}
}

Setup project properties

Under src/main/resources find file application.properties and add configuration for database access  and flyway configuration. Your application will start migration procedure at every application start and syncronize to the correct version automatically.

#Debug
debug=false

# Database connection
spring.datasource.driverClassName=org.postgresql.Driver
spring.datasource.url=jdbc:postgresql://localhost:5432/myappdb
spring.datasource.username=..........
spring.datasource.password=..........

# Jooq dialect
spring.jooq.sql-dialect=POSTGRES

# Flyway
flyway.locations=classpath:db/migration

Flyway migration files

You need a folder where your SQL migration files will reside:

src/main/resources/db/migration

Add first migration file, for example “create currency table” will be saved in file V1_0_0__currency.sql:

/*
 * Currency
 */
CREATE TABLE currency (
	code 						VARCHAR(60) not null,
	abbreviation 				varchar(60),
	description 				text,
	row_id 						serial primary key,					-- row identifier
	created_at 					timestamptz not null default now(),
	created_by 					varchar(60) not null,
	modified_at 				timestamptz,
	modified_by 				varchar(60)
);
CREATE INDEX currency_created_at ON currency (created_at) ;
CREATE INDEX currency_modified_at ON currency (modified_at) ;

-- DEMO DATA
INSERT INTO public.currency (code, abbreviation, description, created_by)
VALUES('EUR', 'Eur', 'Euro', 'admin');

INSERT INTO public.currency (code, abbreviation, description, created_by)
VALUES('USD', 'Usd', 'US Dollar', 'admin');

Be extra careful with names , first part (V1_0_0) is version of migration file, second part of the name is a description, separated with two underline characters.

The scripts must be correct SQL! Validate it first in database environment. If script is not valid, migration procedure will break and you will probably have hard time to figure out what was wrong.

Now you run your application and your database must be prepared according to migration script changes automatically. Flyway start automatically with your application.

More information on this video tutorial for flyway and spring boot..

Maven plugin settings

If you want more control around code generation and migration life cycle, you add flyway and jooq plugins:

Add this properties to the properties section in pom.xml file:

        <!-- Database -->
        <db.url>jdbc:postgresql://localhost:5432/myappdb</db.url>
        <db.username>........</db.username>
        <db.password>........</db.password>
        <!-- FlyWayDB -->
        <org.flywaydb.schema>public</org.flywaydb.schema>
        <org.flywaydb.location>classpath:db/migration</org.flywaydb.location>
        <org.flywaydb.version>4.2.0</org.flywaydb.version>
        <!-- jOOQ -->
        <org.jooq.codegen.namespace>com.bisaga.myapp.database.model</org.jooq.codegen.namespace>
        <org.jooq.input_schema>public</org.jooq.input_schema>
        <org.jooq.target_dir>src/main/java/</org.jooq.target_dir>
        <org.jooq.version>3.9.2</org.jooq.version>
  • Database connection settings in pom.xml will be used by jooq code generator and flyway migration maven plugins.

Flyway and Jooq dependency and version

You probably already have it from generated pom.xml, add it if you don’t have it.

        <dependency>
            <version>${org.flywaydb.version}</version>
            <groupId>org.flywaydb</groupId>
            <artifactId>flyway-core</artifactId>
        </dependency>
        <dependency>
            <groupId>org.jooq</groupId>
            <version>${org.jooq.version}</version>
            <artifactId>jooq</artifactId>
        </dependency>

Version was not required by spring boot but I put version in because I encounter differences between plugin version and spring boot version of the libraries.

Now you need to define plugin definitions as :

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>

            <plugin>
                <groupId>org.flywaydb</groupId>
                <version>${org.flywaydb.version}</version>
                <artifactId>flyway-maven-plugin</artifactId>
                <!-- Note that we're executing the Flyway plugin in the "generate-sources" phase -->
                <executions>
                    <execution>
                        <phase>generate-sources</phase>
                        <goals>
                            <goal>migrate</goal>
                        </goals>
                    </execution>
                </executions>
                <!-- Note that we need to prefix the db/migration path with filesystem:
                    to prevent Flyway from looking for our migration scripts only on the classpath -->
                <configuration>
                    <url>${db.url}</url>
                    <user>${db.username}</user>
                    <password>${db.password}</password>
                    <locations>
                        <location>${org.flywaydb.location}</location>
                    </locations>
                    <schemas>${org.flywaydb.schema}</schemas>
                </configuration>
            </plugin>

            <plugin>
                <!-- Use org.jooq for the Open Source Edition org.jooq.pro for commercial
                    editions, org.jooq.pro-java-6 for commercial editions with Java 6 support,
                    org.jooq.trial for the free trial edition Note: Only the Open Source Edition
                    is hosted on Maven Central. Import the others manually from your distribution -->
                <groupId>org.jooq</groupId>
                <version>${org.jooq.version}</version>
                <artifactId>jooq-codegen-maven</artifactId>
                <!-- The jOOQ code generation plugin is also executed in the generate-sources phase,
                     prior to compilation -->
                <executions>
                    <execution>
                        <phase>generate-sources</phase>
                        <goals>
                            <goal>generate</goal>
                        </goals>
                    </execution>
                </executions>

                <!-- This is a minimal working configuration. See the manual's section about the code generator
                     for more details -->
                <configuration>
                    <jdbc>
                        <url>${db.url}</url>
                        <user>${db.username}</user>
                        <password>${db.password}</password>
                    </jdbc>
                    <generator>
                        <database>
                            <includes>.*</includes>
                            <inputSchema>${org.jooq.input_schema}</inputSchema>
                        </database>
                        <target>
                            <packageName>${org.jooq.codegen.namespace}</packageName>
                            <directory>${org.jooq.target_dir}</directory>
                        </target>
                    </generator>
                </configuration>
            </plugin>
        </plugins>
    </build>

Flyway and Jooq maven commands

After you add plugin definitions to maven pom.xml file you get new lifecycle commands :

When flyway commands are used directly they search for SQL files in the target “classes” folder and not in the source tree (“src”). When project compiled the files, target folder is synchronized with the current version of code in the source tree.

After compiling project you could run “flyway:migrate” command for example. You can always check files in the target folder and delete it if you are not sure you have the latest version of the files.

Jooq code generation

Jooq generate data model code from your database. The code is added to the project in the namespace as you defined in plugin definition.  It’s wise to split plugin definition to properties and plugin definition part.

<!-- jOOQ -->
<org.jooq.input_schema>public</org.jooq.input_schema>
<org.jooq.codegen.namespace>com.bisaga.myapp.database.model</org.jooq.codegen.namespace>
<org.jooq.target_dir>src/main/java/</org.jooq.target_dir>

Every time you compile project, database model code is regenerated. Check here for more information about jooq.

You can check how database model code is regenerated simply by deleting a line from one of generated files and recompile the project again.

More about jooq & spring boot can be found here.

Create JSON service

Now I need to test the environment if everything is set in place as should be. I will create simple currency service as sample application and test basic CRUD operations.

Create simple data container class

To be able to send JSON payload back and forth from client to server we create a simple transport class:

package com.bisaga.myapp;

/**
 * Created by igorb on 17. 06. 2017.
 */
public class CurrencyDto {
    private String code;
    private String abbreviation;
    private String description;
    private Integer rowId;

    public String getCode() {
        return code;
    }

    public void setCode(String code) {
        this.code = code;
    }

    public String getAbbreviation() {
        return abbreviation;
    }

    public void setAbbreviation(String abbreviation) {
        this.abbreviation = abbreviation;
    }

    public String getDescription() {
        return description;
    }

    public void setDescription(String description) {
        this.description = description;
    }

    public Integer getRowId() {
        return rowId;
    }

    public void setRowId(Integer rowId) {
        this.rowId = rowId;
    }
}

Service class and database interaction with Jooq

Now we create service class with which we will interact with database :

/*
*  MIT License
*  Copyright (c) 2017 Igor Babic
*/
package com.bisaga.myapp;

import static com.bisaga.myapp.database.model.tables.Currency.CURRENCY;

import com.bisaga.myapp.database.model.tables.records.CurrencyRecord;
import org.jooq.DSLContext;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;

import java.util.List;


@Service
@Transactional
class CurrencyService {
    private static final Logger LOG = LoggerFactory.getLogger(CurrencyService.class);

    @Autowired
    private DSLContext db;

    /**
     * List all currencies in the currency table
     * @return Return list of currency records
     */
    List<CurrencyDto> getAll() {
        List<CurrencyDto> currList;
        currList = db.select().from(CURRENCY)
                .fetch().into(CurrencyDto.class);

        return currList;
    }

    /**
     * Search for a currency record by currency code
     * @param code  currency code as parameter
     * @return Return currency record
     */
    CurrencyDto getByCode(String code) {
        List<CurrencyDto> currList;
        currList = db.select().from(CURRENCY)
                .where(CURRENCY.CODE.eq(code))
                .fetch().into(CurrencyDto.class);
        
        return currList.get(0);
    }

    /**
     * Delete currency record by row identifier
     * @param rowId rowId row identifier
     * @return Return number of deleted records
     */
    Integer delete(Integer rowId) {
        return db.delete(CURRENCY).where(CURRENCY.ROW_ID.eq(rowId)).execute();
    }

    /**
     * Save currency execute insert or update command if currency already exist.
     * @param currency instance of CurrencyDto class as parameter
     */
    CurrencyDto saveCurrency(CurrencyDto currency) {
        if (currency.getRowId() == null) {
            return this.insert(currency);
        } else {
            return this.update(currency);
        }
    }

    /**
     * Insert new currency record to database
     * @param currency Currency record as parameter
     * @return Return currency record filled with database defaults after insert
     */
    private CurrencyDto  insert(CurrencyDto currency)  {
        CurrencyRecord record = db.newRecord(CURRENCY, currency);
        record.setCreatedBy("admin");
        record.insert();                // insert to database
        record.into(currency);			// return changes made by database
        return currency;
    }

    /**
     * Update currency record in database
     * @param currency currency record as parameter
     * @return Return currency record filled with changes from database update
     */
    private CurrencyDto update(CurrencyDto currency) {
        CurrencyRecord record = db.newRecord(CURRENCY, currency);
        record.update();                // update to database
        record.into(currency);			// return changes made by database
        return currency;
    }

}

REST controller with router mapping

On the end we need a controller class to expose access points for REST service:

/*
*  MIT License
*  Copyright (c) 2017 Igor Babic
*/
package com.bisaga.myapp;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.HttpStatus;
import org.springframework.web.bind.annotation.*;

import java.net.Authenticator;
import java.util.List;

@RestController
@RequestMapping(value = "/api/currency", produces = "application/json")
public class CurrencyController {

    @Autowired private CurrencyService service;

    @RequestMapping(method=RequestMethod.GET)
    @ResponseStatus(HttpStatus.FOUND)
    public List<CurrencyDto> findAll() {
        return service.getAll();
    }

    @RequestMapping(value="/{code}", method=RequestMethod.GET)
    @ResponseStatus(HttpStatus.FOUND)
    public CurrencyDto findByCode(@PathVariable("code") String code) {
        return service.getByCode(code);
    }

    @RequestMapping(method=RequestMethod.POST)
    @ResponseStatus(HttpStatus.OK)
    public CurrencyDto saveCurrency(@RequestBody CurrencyDto currency) {
        return service.saveCurrency(currency);
    }

    @RequestMapping(method=RequestMethod.DELETE)
    @ResponseStatus(HttpStatus.OK)
    public Integer deleteCurrency(@RequestParam(value = "rowid", required = true) Integer rowId ) {
        return service.delete(rowId);
    }


}

Don’t forget to instantiate services with DI (dependency injection) with @Autowire annotation. This simplify development a lot !

Interactive testing

The service should in this point work as expected, just run application and navigate to “localhost:8080/api/currency”.

You can search for a specific currency with added path variable appended to Url address:

Postman application for Http API testing

To inspect service in more details on the client side, you can use Postman application.

Look at example ofadding a currency with a POST message:

In the headar I add “content-type” variable with value “application/json” and in the payload a json message with a new currency json structure. On the right side I received “200-OK” and a currency record with a record and new row identifier registered in the database.

 

Eclipse scout – development environment

Eclipse scout

Java development framework (HTML5) for rapid development of line of business (LOB) apps.

To download it , go to eclipse packages download page and select Eclipse for Scout Developers package.

2017-01-17-21_54_17-eclipse_scoutUnblock ZIP file

Before you unzip it somewhere, try to unblock zip file first, you will avoid some difficulties later on. Right click on zip file and under Security section check “Unblock”.

2017-01-17-22_18_25-eclipse-scout-neon-2-win32-x86_641-properties

Eclipse workspaces

Each scout project consist of a at least few sub-projects.  It will become very crowded very soon if you do not decided to work with multiple workspaces. I open new workspace for each business app, later I will automate cross workspace configurations with Oomph.

Eclipse – git integration

Read this great tutorial for git integration.

For default repository folder define eclipse variable: ${workspace_loc}/git, it will create separate repository for each workspace.

Scout hello world application for version 6.1.

If you create new Scout Hello World sample application and try to run it, at least in current moment in time (Eclipse.Neon.2, with default scout SDK version 6.1.0.M4), application doesn’t work.

   <org.eclipse.scout.rt.version>6.1.0.M4</org.eclipse.scout.rt.version>

After run server and client “dev” project in eclipse, open http://localhost:8082/ and application doesn’t show main application page, just show something without any style and no javascript support.

Looks like generated hello world wizard doesn’t generate latest  project styles (less styles) with proper javascript support as required with 6.1. version of scout SDK. Probably because the version of scout SDK and eclipse tooling (Eclipse version Neon.2) are not in sync.  I didn’t won’t to change to unstable eclipse version just because of that and maybe that won’t solve the problem.

Anyway, I repair hello world example based on “Contacts” demo application and publish corrected empty hello world sample on  github project: scout_empty.

To run “empty” example, open “empty.all.app.dev” project, select “[webapp] dev server+ ui.launch” file  and with right click “Run as” launch configuration file.

Open http://localhost:8082/ in browser and you will get something like this:

2017-01-17-23_27_34-empty-example

Scout application architecture and documentation

To understand scout application architecture download scout documentation in PDF or read Beginner guide and Technical guide.

 

Key binding in Eclipse and Slovenian keyboard

AltGr+B => doesn’t enter “{” character in java editor, it is “Skip all breakpoints” binding. (Ctrl+Alt+B).

AltGr+G => doesn’t enter “]” character , it is “Find Text in Workspace” binding. (Ctrl+Alt+G).

 

Netbeans and Polymer

Polymer HTML code completion for Netbeans 8.2

Thanks to this wonderful extension I can get rid of annoying errors in HTML editor when I edit polymer components in html files.

Because I use maven project type and netbeans , there is no “.nbproject” folder. In that case the “customs.json” file must be present in src/main/webapp folder.

2016-12-10-11_08_11

Recreate new version of customs.json

Because not all attributes are recognized by customs.json I will recreate it as explained in project documentation.

After downloading master zip file, you need to create “dist” folder under src folder and then run index.js file.

igorb@Pavilion ~/PolymerForNB-master/src
$ mkdir dist

igorb@Pavilion ~/PolymerForNB-master/src
$ node index.js

New copy of customs.json will be created in dist folder, just copy it to the webapp folder and that’s it.

Merge changes back to project file

Don’t forget if you already used customization and added something, you need to merge old version with new one or you will lose your changes…

Well, it’s not so simple, looks like after you add few changes to customs.json with HTML editor, the file (customs.json) changed very  dramatically. Looks like netbeans  reorganize whole structure or something.

 

 

 

 

Maven – installation

How to install Maven on Windows 10

  1. Download Apache Maven ZIP file and unzip it to some folder (example: C:\Programs\apache-maven-3.3.9 ).
  2. Add maven folder to environment variables:
  • MAVEN_HOME = C:\Programs\apache-maven-3.3.9
  • M2_HOME = %MAVEN_HOME%
  • add to path = %MAVEN_HOME%\bin

Test:

$ mvn –version

$ mvn --version
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T17:41:47+01:00)
Maven home: C:\Programs\apache-maven-3.3.9
Java version: 1.8.0_101, vendor: Oracle Corporation
Java home: C:\Program Files\Java\jdk1.8.0_101\jre
Default locale: en_US, platform encoding: Cp1252
OS name: "windows 10", version: "10.0", arch: "amd64", family: "dos"