ProTips PostGres 1: Looking through the Postgres Window

Content posted here with the permission of the author, who is currently employed at Josh Software. Original post available here.

Problem – 

We encountered a situation where we had to select only one record for each values of a column (lets say ‘user_id’) ordered by a different column (lets say ‘price’) in a certain table (lets say ‘purchases’). Doing this active-record way was very inefficient. By active-record way, we had to:

  1. Fetch all the user_ids from purchases table
  2. Loop through all the user_ids, for each user_id
    1. Select once record ordered by price

This approach lead us to query the database number of user times + one query for selecting all the user_ids (N + 1 query problem).

Solution – 

We struggled to find a better solution in active-record for this problem and window functions of postgresql came to the rescue.

To understand postgres window functions, we must understand GROUP BY clause. We use GROUP BY clause to apply aggregate functions on a subset of rows of a table.

For example you have a table called “purchases” with the schema as shown below.

  CREATE TABLE purchases (
    purchase_id serial PRIMARY KEY,
    product_id INT NOT NULL,
    price DECIMAL (11, 2),
    user_id INT NOT NULL
  );

Lets insert data into this table.

 INSERT INTO purchases (product_id, price, user_id)
 VALUES
 (1, 10, 1),
 (2, 20, 1),
 (3, 30, 1),
 (4, 40, 2),
 (1, 10, 2),
 (5, 50, 2),
 (6, 30, 2)

 

To calculate average of all the purchase prices, we will use AVG aggregate function.

  SELECT AVG (price) FROM purchases;

This returned the average of prices for both users with id 1 and 2.

  avg
 --------
  27.1428571428571429

 

Now, if you want to find AVG (A postgres aggregate function) of the price of items that a specific user has purchased. You would use GROUP BY clause on user_id. Thus the following statement will return average for each user_id.

  SELECT AVG (price) FROM purchases GROUP BY user_id;

Above query return the following

 user_id          avg
 --------      --------
    1          20.0000000000000000
    2          32.5000000000000000

As you can see,

The AVG aggregate function has reduced the number of rows returned by the sql query.

The GROUP BY clause groups the rows of the table into subset of rows based on a column(s).

Similarly, a window function also operates on a subset of rows of a table but it does not reduce the number of rows. It returns the actual records of the table in the output with an extra column “avg” if you have applied AVG function to the prices.

In the query below the AVG function works as a window function that operates on a set of rows specified by the OVER (PARTITION BY) clause.

For example, below query will return the product_id, user_id, price and average from “purchases” table for each user group.

  SELECT
  product_id,
  user_id,
  price,
  AVG (price) OVER ( PARTITION BY user_id )
  FROM
  purchases

The output

  product_id    user_id   price   avg
 ------------ ---------- ------- -----
      1           1       10.00   20.0000000000000000
      2           1       20.00   20.0000000000000000
      3           1       30.00   20.0000000000000000
      4           2       40.00   32.5000000000000000
      1           2       10.00   32.5000000000000000
      5           2       50.00   32.5000000000000000
      6           2       30.00   32.5000000000000000

You can also apply other operations with PARTITION BY clause. Like if you want the results to be returned in an order, you can use ORDER BY clause. You can also use built in window functions like row_number(), rank(), etc.

These built-in window functions adds a number to each row based on their order. The row_number() function assigns a serial number to each of the rows returned. So if you want a limited number of records for each user_id, you can use it.

The query below will return only one record for each user_id, decreasingly ordered by their purchase price.

  SELECT * FROM
  (
    SELECT
    product_id,
    user_id,
    price,
    row_number() OVER ( PARTITION BY user_id ORDER BY price DESC )
    FROM
    purchases
  ) tmp
  WHERE tmp.row_number <= 1

In the above query, I have ordered each partition by decreasing price and then added limit on the number of rows for each partition by using the row_number() function. Below is the output.

  product_id   user_id   price   row_number
------------- --------- ------- ------------
     3           1       30.00       1
     5           2       50.00       1

Postgres provides many built-in window functions. Some of them are explained below.

RANK() function

The RANK() function assigns ranking within an ordered partition.  If the values of the two rows are the same, the  RANK() function assigns the same rank, with the next ranking(s) skipped.

  SELECT
  product_id,
  user_id,
  price,
  RANK () OVER ( PARTITION BY user_id ORDER BY price )
  FROM purchases

The output

 product_id    user_id    price   rank
 ------------ ---------- ------- -----
    1            1        10.00    1
    1            1        10.00    1
    2            1        20.00    3
    3            1        30.00    4
    1            2        10.00    1
    6            2        30.00    2
    4            2        40.00    3
    5            2        50.00    4

 

DENSE RANK()

The DENSE_RANK() function assigns the ranking within an ordered partition, but the ranks are consecutive. In other words, the same ranks are assigned to multiple rows and no ranks are skipped.

  SELECT
  product_id,
  user_id,
  price,
  DENSE_RANK () OVER ( PARTITION BY user_id ORDER BY price )
  FROM purchases

The output

  product_id   user_id    price   rank
 ------------ ---------- -------  -----
      1         1        10.00     1
      1         1        10.00     1
      2         1        20.00     2
      3         1        30.00     3
      1         2        10.00     1
      6         2        30.00     2
      4         2        40.00     3
      5         2        50.00     4

There are many other window functions that are listed below.

  1. FIRST_VALUE()
  2. LAST_VALUE()
  3. NTH_VALUE()
  4. PERCENT_RANK()
  5. LAG()
  6. LEAD()

 

In the next blog I will share performance analysis of using postgres window functions vs active-record way for same task.

 

Advertisements
Posted in General | Leave a comment

Securing your web applications

Content posted here with the permission of the author, who is currently employed at Josh Software. Original post available here.

While developing web apps, how frequently do you think about securing it?

Just wait for a moment and think!

You do? Really? That’s Awesome!

This blog will act like a checklist for you. Go through it once, to figure out just in case you have missed anything OR maybe I might have missed something that you have taken care of. Either way, please be kind enough to drop a comment.

You don’t? No worries, let’s do it together. Let’s understand:

  • What are the various types of attacks?
  • How do they work?
  • How to secure your app from such attacks?

Alright, without any further ado, let’s get started.

Cross-Site Request Forgery (CSRF)

It’s a kind of attack that forces an end user to execute unwanted actions on a web application in which they’re currently authenticated. These attacks target state-changing request, not theft of data.

csrf explanation

Consider a scenario wherein a genuine admin user logs in to your web app (http://www.genuine-domain.com). When login form is submitted, server validates the credentials and sends a cookie in response. Now, for each subsequent request to your domain, browser will automatically send along this cookie. Server will verify the existence and contents of the cookie before considering the request authenticated.

All sounds good? Yeah? Think again!

Your web app is vulnerable to CSRF attacks right now. How?

Let’s say the admin user is authorized to delete some other user’s account. Also assume that URL for such endpoint is accessible over GET request. Now, open a new tab, type this URL and press enter. Voila! you just deleted a user.

Now think, how difficult is it for a malicious user to trick your customer into clicking such a URL? Not very difficult? Right?

Just by following certain basic rules, you can protect your app from such attacks:

  1. Show a confirmation prompt before every destructive action
  2. Use GET verb for read-only requests. Use POST/PUT/PATCH/DELETE for write operations.

How do non-get verbs help? Did you notice that only URLs exposed over GET verb can be opened from browser’s URL bar (and hence the links). By using non-get verbs, you are making the hackers job a little more difficult.

The hacker may still trick your customer into submitting a form, which would trigger a POST/PATCH/PUT/DELETE request.

We can step-up our security by including a one-time disposable secure token in every form rendered by the server. Now, if a form is submitted without a token OR with an invalid token, the server would simply reject it. Only our server and genuine client (browser tab) knows about this token. Additionally, verify the Origin and Referrer header whenever possible. Please note that Origin header may not always be present (specially in GET requests). Similarly, Referrer header is not guaranteed to be present in every request.

Cookie Stealing/Hijacking

The attacker can also modify the contents of the cookie OR just copy it. To mitigate this risk, your web app needs to ensure the following while setting the cookie:

  1. User “session” cookies – they are deleted when browser/client is shut down.
  2. Turn on the “secure” flag on the cookie – only accessible over HTTPS.
  3. Turn on the “HttpOnly” flag on the cookie – inaccessible using JavaScript API’s.

Network Sniffing

An attacker can always intercept your network requests and get access to sensitive information. To secure your data during transit, we need to encrypt it. This can be done by using HTTPS certificates.

However, using HTTPS alone is not sufficient. HTTPS encrypts data using some ciphers. You encryption is only as strong as your ciphers. Older version of HTTPS are known to have vulnerabilities. Therefore, I urge you to use HTTP Strict Transport Security (HSTS).

CORS (Cross-Origin Resource Sharing)

Similar to  HTML requests, we also need to protect our web app from AJAX based security threats. AJAX calls can be triggered either from your domain OR from a different domain. Luckily, most modern browsers block cross-domain AJAX calls by default. But, what if a genuine user wants to share resources across the domains?

Secure inter-domain AJAX based communication is handled by using CORS (Cross Origin Request Sharing). Here is an excellent blog on implementing CORS.

NOTE: CORS restrictions are only applicable on AJAX requests (i.e. not applicable on HTML requests).

Clickjacking

Clickjacking (User Interface redress attack OR UI redress attack OR UI redressing) is a malicious technique of tricking a Web user into clicking on something different from what the user perceives they are clicking on, thus potentially revealing confidential information or taking control of their computer while clicking on seemingly harmless web pages.

clickjacking explanation

In order to secure your web app from clickjacking attacks, the server needs to set X-Frame-Options: DENY. This means, your web app cannot be opened inside an iframe.

If you want to allow rendering of your app inside an iframe from your domain only, you can set:

X-Frame-Options: SAMEORIGIN
OR
X-Frame-Options: ALLOW-FROM <whitelisted-domain>

SQL Injection

A SQL injection attack consists of insertion or “injection” of a SQL query via the input data from the client to the application. A successful SQL injection exploit can read sensitive data from the database, modify database data (Insert/Update/Delete), execute administration operations on the database (such as shutdown the DBMS), recover the content of a given file present on the DBMS file system and in some cases issue commands to the operating system.

Bind variables are the best way to prevent SQL injection. When using bind parameters you do not write the actual values but instead insert placeholders into the SQL statement. That way the statements do not change when executing them with different values. The question mark (?) is the only placeholder character that the SQL standard defines.

Cross-Site Scripting (XSS)

Cross-Site Scripting, commonly known as XSS, is a vulnerability that is often found in web apps. XSS allows attackers to inject client-side scripts into public facing web pages and, in many cases, can be used by attackers to work their way past access controls.

This is done by tricking a browser so that it accepts data from an untrusted source, and this typically happens when attackers use familiar code (such as JavaScript, for example) as developers don’t scrub out these characters.

xss explanation

To protect your web app from XSS attacks, you need to:

  1. Never trust user input. i.e. Always escape/sanitize input before inserting into:
    1. HTML element’s content. e.g. innerHTML, $(‘#some-id’).html(), etc.
    2. Values of common attributes. e.g. class, data-*, etc.
    3. HTML or JS comments
    4. Style property values
    5. URL parameters
  2. Whenever possible use an Auto-Escaping Template System
  3. Avoid eval() or such dynamic evaluation methods
  4. Implement CSP

CSP makes it possible for web app developers to specify the domains that the browser should consider to be valid sources of executable scripts. A CSP compatible browser will then only execute scripts loaded in source files received from those whitelisted domains, ignoring all other script (including inline scripts and event-handling HTML attributes).

e.g. The following header instructs the CSP compatible browser to only allow scripts from site’s own origin. This excludes the sub-domains.

Content-Security-Policy: default-src ‘self’

Event after implementing all the above mentioned security measures, your web app is still not 100% secure. That’s when I say: Security is a myth.

Despite that, I recommend following ALL the above mentioned precautions to protect your web app. That’s because something is better than nothing and this is certainly more than something ;). If nothing else, at least this will discourage attackers from choosing you as a target.

Finally, a sincere appeal to all:

Implement all these security measures together. Don’t pick n choose.

Posted in General | Leave a comment

Ionic 3 : A Step Forward

Content posted here with the permission of the author, who is currently employed at Josh Software. Original post available here.

Most of the people have already switched from Ionic 1 to Ionic 2. You are maybe one of them and feel more comfortable and settled with Ionic 2. But now the latest version of Ionic i.e Ionic 3 has been released. The change from Ionic 2 to Ionic 3 is nothing as compared to that of Ionic 1 to Ionic 2.

Ionic 3 release is not the framework update. It’s just a major update in Ionic 2. It supports angular 4 and intern follows semantic versioning system of angular( The 3 dots system.).That is MAJOR.MINOR.PATCH. The major update implies that there will be some breaking changes and in case of minor/ patch level update, some API will be changed.

Ionic has come up with many features and improvements over Ionic 2. Basic 3 features I implemented in my app and it helped me a lot. My app is now working much faster. In this blog I am sharing details of these features and steps to implement them:

1. Ionic native plugin support:

The plugin support system has been changed in Ionic native 3.x. These changes are:

  • A native package needs to be installed for every plugin that is added to a project
  • The application root module requires that the native package for each installed plugin is imported and added to the provider’s array

Easy steps to add a new plugin to Ionic 3 project:

1. Install Cordova plugin with the help of ionic CLI command:

ionic cordova plugin add &lt;plugin-name&gt;

2. Install ionic native package for the installed plugin by npm:

npm install --save @ionic-native/&lt;plugin-name&gt;

Note that, we need to just add plugin name after @ionic-native/ path. This command will make entry of same package in package.json.

3. Import installed native package to project.

For that, we need to import it into app.module.ts Open root module of project – /src/app/app.module.ts. Add import statement like, (exmaple: camera plugin)

 import { Camera } from '@ionic-native/camera'; 

Now add entry in providers,

  @NgModule({
  ...
  providers: [
    Camera
  ]
  ...
  })

4. Now wherever you required this plugin, just import it. Fairly simple, right?

Ionic native 3 providing us a Cordova wrapper for more than 130 plugins. Instead of using plugin directly, ionic allows us to import respective plugin wrapper in root module once and use it whenever and wherever we want. This reduces app bundle size and make ionic app run faster

2. Lazy loading:

Loading something as and when required. For example, In a web page if we have 50 images, then for better loading time we would not prefer to load all images at once. As user scroll down, images should load. The Same concept is used in Ionic 3.

Instead of loading everything in a front, it will load the component when needed. This would help in starting ionic app. In big applications, when lots of things need to be loaded in the start, this feature will be helpful.

To implement lazy loading we need to follow some steps. Lets see how we can implement lazy loading in Ionic 3

1. Remove all import statement of a page

Lets take example for HomePage. We need to remove all references of HomePage from all other pages, providers, and modules. Start with src/app/app.module.ts. In this file we will have two arrays: entryComponents and declarations. All the components that we want to load in app, we specify them here. Now we want to load HomePage only when it is requested.

So, first of all remove all the reference of HomePage from app.module.ts includinng import statement.

@NgModule({
  declarations: [
    MyApp,
    HomePage
  ],
  ...
  entryComponents: [
    MyApp,
    HomePage
  ],
  ...
})

2. Create ngModule for HomePage

Create new file src/pages/home/home.module.ts. Add declaraion, import and export statement for HomePage.

import { NgModule } from '@angular/core';
import { IonicPageModule } from 'ionic-angular';
import { HomePage } from './home';

@NgModule({
declarations: [
    HomePage,
],
imports: [
    IonicPageModule.forChild(HomePage),
],
exports: [
    HomePage
]
})
export class HomePageModule {}

3. Add decorator in HomePage

Import IonicPage from ionic-angular and add @IonicPage() decorator to home.ts page.

import { Component } from '@angular/core';
import { NavController } from 'ionic-angular';
import { IonicPage } from 'ionic-angular';

@IonicPage()
@Component({
  selector: 'page-home',
  templateUrl: 'home.html'
})
export class HomePage {
  constructor(public navCtrl: NavController) {}
}

4. Remove all references of HomePage

Remove all the import statements to HomePage from all pages, provides, root files. Now its time to use HomePage module. We can use it by simply specifying HomePage in a string. For example, If you are using it like this:

rootPage:any = HomePage

Now, it should be replaced by,

rootPage:any = 'HomePage'

These are the simple steps to make our app run faster by implementing lazy loading. We just need to create separate ngModule for each component and decorate it with IonicPage.

3. Plugin mocking and browser development:

Till Ionic 2 we were not able to test any plugin in the browser of the ionic app. Ionic 3 provides full support for plugin mocking. This means that we can easily use and test more than 130 plugins in a browser. This makes ionic developer more comfortable to build entire app in browser without any help of device or emulator.

To use this feature we need to create mocks for our plugin to use and make them return proper data.

Native plugin mock is just a copy of actual plugin. It allows user to test the plugin functionality without using actual plugin in browser itself. Lets take exmaple of camera plugin. We will implement getPictures() method and return test data. Lets write camera plugin mock.

1. Create mock folder in root folder

mkdir src/mocks
cd mocks

2. Create camera-mock.ts and overwrite getPictures() method like following:

export class CameraMock {
  getPicture(params) {
    return new Promise((resolve, reject) =&gt; {
      resolve("BASE_64_IMAGE_DATA");
    });
  }
}

3. Import camera-mock.ts in app.module.ts and add it to providers array too

import { CameraMock } from "../mocks/camera-mock";
@NgModule({
...
providers: [
...
CameraMock
]
...
})

4. Now plugin mock is ready to use in any component. Just import it and use it.

I recommend you to use Ionic 3 native plugin support as well as load the components as and when required i.e. lazy loading, Surely you will get performance refinement. Also, there is proper documentation available for steps to migrate from Ionic 2 to Ionic 3. You can refer this.

Posted in General | Leave a comment

Record and export Audio files in browser using Recorder.js and upload it on cloud.

Content posted here with the permission of the author, who is currently employed at Josh Software. Original post available here.

In this article, Let’s see how audio recording is possible without any shim or plugin, but only by using the Recorder.js library in the browser and how to upload recorded file on s3 (Or any other cloud storage). Before going to do audio recording first we will see which technology we need and why.

There are resources available for handing these individually but not together. I used references from below links. It is advisable to read below links before proceeding.

Technology Used:

How does the recording process works?

To record audio in the browser, you will need obviously access to the microphone with the getUserMedia API and the AudioContext API, so be sure that your browser supports these APIs

  • Web Audio API

The Web Audio API is a JavaScript API for processing and synthesizing audio in web applications.The API supports loading audio file data in multiple formats, such as WAV, MP3, AAC, OGG and others. Browser’s support for different audio format varies(Check Here).

  • AudioContext API

The AudioContext is used for creating, processing and decoding audio. You need to create an instance of AudioContext class before you can do anything because all are processed / decoded, controlled for audio generation by this API.
How AudioContext declare?:

window.AudioContext = window.AudioContext or window.webkitAudioContext
  • getUserMedia API

Before going to do recording we should know whether our web application can access user’s camera or microphone. So getUserMedia is an API that gives a web page access to a user’s camera and microphone via JavaScript. getUserMedia() has been supported since Chrome 21, Opera 18, and Firefox 17.

Make sure your application is secured with HTTPS protocol to use “getUserMedia API”. It does not work on insecure origins. Starting with Chrome 47, getUserMedia() requests are only allowed from secure origins: HTTPS or localhost.
How to check getUserMedia

if navigator.getUserMedia
  # do something cool
else
  # fallback code
  • Recorderjs

Recorderjs is a JavaScript library that records audio input (Web Audio API AudioNode object) and encodes to audio file image (Blob object). It supports three encoding formats. By using Recorder.js, the recording process is very simple. All that you need to do is to request the microphone in browser, once the user allows the access to the microphone, a stream object is received in the success callback of the initialization.This stream needs to be handled with the createMediaStreamSource method from an instance of the AudioContext API.All the variables should be stored globally or at least accessible in the scope for the rest of functions.

To understand above explanation, added code here:

startUserMedia = (stream) ->
  input = window.audio_context.createMediaStreamSource(stream)
  console.log 'Media stream created.'
  window.recorder = new Recorder(input, 'audio/mp3')
  window.recorder.record()
  console.log 'Recorder initialised.'
  return

The Recorder.js class expects the handled stream as first argument namely the input. With this instance of Recorder.js you can trigger the record method that starts to recording the received Audio and it will run indefinitely until the stop method from the same instance of recorder is triggered.

Export Recorded Audio

Export the Audio Blob from the recorder audio using the exportWAV method of the recorder instance. The exportWAV method can export the audio in wav and mp3 format when the second parameter is specified with the correct mimetype (‘audio/wav’, ‘audio/mp3’). Don’t forget to use the clear method of the recorder to start with a new one later. With the generated blob you will be able to upload it as an audio file to the server, in the same browser or just to play it in the same document.
Added Code here:

createDownloadLink = ->
  recorder and recorder.exportWAV((blob) ->
    url = URL.createObjectURL(blob)
    li = document.createElement('li')
    au = document.createElement('audio')
    hf = document.createElement('a')

    au.controls = true
    au.src = url
    hf.href = url
    hf.download = 'audio.mp3'
    hf.innerHTML = hf.download
    li.appendChild au
    li.appendChild hf
    playRecord.appendChild li

How to record Audio with Recorder.js

To make the explanation as simple as possible, we’ll just write a haml document that implements all the previously described process. It consists of 2 buttons namely Start and Play that triggers the recording process with Recorder.js. When the user stops the recording, it will append an Audio Track to an UL list with a downloadable audio file with like “audio.mp3”:

HAML code:
Create 2 buttons as “Start Recording” and “Stop Recording”

!!!
%html
  %head
    %meta{:content =&gt; "text/html; charset=utf-8", "http-equiv" =&gt; "Content-Type"}
    %title Live input record and playback
    :css
      ul {
          list-style: none;
      }

      #playRecord audio {
          display: block;
          margin-bottom: 10px;
      }
  %body
    %h1 Recorder.js export example
    %button#start-btn Start recording
    %button#stop-btn{:disabled =&gt; "disabled"} Stop recording
    %h2 Stored Recordings
    %ul#playRecord

Click on Start button will start recording and clicking on Stop button will stop recording and export it.

$('#start-btn').on 'click', -&gt;
  if confirm "Once you start recording, you can't pause it."
    initAudioRecording()

$('#stop-btn').on 'click', -&gt;
  stopRecording(this)

stopRecording = (button) -&gt;
  if window.recorder != undefined
    window.recorder  window.recorder.stop()
    createRecording()
    return

createRecording = -&gt;
  recorder and recorder.exportWAV((blob)
    data = new FormData();
    data.append("audio", blob);
    console.log(data)
    $.ajax
      url: 'Your Controller path'
      type: 'POST'
      data: data
      contentType: false
      processData: false
      success: -&gt;
        console.log "Successfully uploaded recording."
  )
  return

startUserMedia = (stream) -&gt;
  input = window.audio_context.createMediaStreamSource(stream)
  # console.log 'Media stream created.'
  window.recorder = new Recorder(input, 'audio/mp3')
  window.recorder.record()
  # console.log 'Recorder initialised.'
  return

initAudioRecording = -&gt;
  try
    window.AudioContext = window.AudioContext or window.webkitAudioContext
    navigator.getUserMedia = navigator.getUserMedia or navigator.webkitGetUserMedia or navigator.mozGetUserMedia
    window.URL = window.URL or window.webkitURL
    window.audio_context = new AudioContext
    console.log 'navigator.getUserMedia ' + (if navigator.getUserMedia then 'available.' else 'not present!')
  catch e
    alert 'No web audio support in this browser!'
  navigator.getUserMedia { audio: true }, startUserMedia, (e) -&gt;
    console.log 'No live audio input: ' + e
    return
  return

Browser Support

  • Make sure you are using a recent version of browser.
  • Make sure your application is secured with HTTPS protocol to use “Web Audio API”. It does not work on insecure origins. As of Chrome 47, the getUserMedia API cannot be called from insecure origins.
  • There is no issue while implementing Web Audio API in development environment i.e. http://localhost:3000
  • You can test your browser compatibility test.Here

How to upload recorded file on s3

Till now we have seen how to get audio blob object, but the major challenge is how to upload this audio blob on s3(or on cloud).
See below coffee script code:

createRecording = ->
  recorder and recorder.exportWAV((blob) ->
    data = new FormData();
    data.append("audio", blob);
    $.ajax
      url: 'Your controller path to upload recording'
      type: 'POST'
      data: data
      contentType: false
      processData: false
      success: ->
        if confirm "Successfully uploaded recording."
          window.location.href = 'Redirect path after successful ajax call.'

To make it very simple to understand I will explain you above code:
I have created a method as “createRecording” which calls exportWAV() method from Recorder.js, so we get a blob object.
Our aim is to upload this blob object on s3, for that I have created FormData() object and append audio blob to FormData() object.
Send this FormData() object to server by ajax call with “POST” method.

NOTE:
As we may be sending large amount of data to server (in terms of MB), we should have proper nginx entry for client_max_body_size.We should change nginx configuration in file /etc/nginx/nginx.conf, to expected max size of our recorded file.

client_max_body_size 50M;

Let’s see server side implementation:

def your_controller_method
    @model_object.audio = params[:audio]
    @model_object.save
    render nothing: true
end

Once file object reaches to your server upload same to S3 (or any other cloud storage) using regular file upload method.

For uploading file we have used carrierwave gem(click here)

Posted in General | Leave a comment

Use power of database to speedup your application!

Learn with fun

In this blog post, I am going to tell you some queries/ tricks to speed up your application by using the power of database. I am using postgres and activerecord rails so all queries are related to these, but you can relate it with your database.

Recently I got some performance issue when our user base increased many folds. We did some optmization and would like to share the tips:

Note: As I am using ActiveRecord most of the examples have ActiveRecord syntax with corresponding SQL statement

Select only required fields from database – Not all

Mostly developers miss to specify the required columns and instead select all fields from database which leads to performance degradation.

Let us say my User Table has 100 users

Above query took 15.4 ms as it is selecting all columns from the table. But, the query below took only 3.3ms.

Here we are not…

View original post 744 more words

Posted in General | Leave a comment

How to upload a large CSV efficiently using rails!

Think, Click and Succeed

Active-record is an abstraction layer that facilitates creation, deletion and use of ORM objects whose data requires persistent storage to a database. This keeps us away from having to think too much about SQL level queries and makes it very easy for us to work with data. It gives us super easy interface that helps us to do “almost” anything that we can do with bare SQL statements. Apart from the basic CRUD operations, active-record lets us do more complicated database stuff like pick a group of records based on a criteria, order them, join tables, perform mathematical operations, etc.

Active-record pattern is liked by most because of the above mentioned reasons. But using active-record solely may not help when your application scales.

For example, active-record does not have a support to bulk insert into the database. Ofcourse we can use gems to do that, but I personally…

View original post 661 more words

Posted in General | Leave a comment

Loading…widgets in react-redux application

I got an interesting requirement for my React app to display multiple widgets with the following conditions:

  1. Number of widgets to displayed – UNKNOWN
  2. Content to be displayed – UNKNOWN
  3. API endpoints – UNKNOWN
  4. Only known thing was the placeholder in the widgets where data needs to be displayed
  5. Need to display loader for each API call

Fortunately, an API was available to get details such as the number of widgets and their endpoints.

As per the requirement to display loader for each API call, the first screen was:

main-widget Initial Loader screen

When the first API returns the response containing details regarding widgets such as the number of widgets and their endpoints, the above widget should turn into the following screen if there are 4 widgets data returned by initial API call:

widget-loaders
screen 2

And each of the widgets will call different APIs to get the data it needs…

View original post 524 more words

Posted in General | Leave a comment