IS EVERYTHING REALLY PASSED BY VALUE IN GO?

Content posted here with the permission of the author Anuj Verma, who is currently employed at Josh Software. Original post available here.

As a Go programmer or while learning Go you might have heard that everything in Go is passed by value.

Also in the official FAQ it says that:

As in all languages in the C family, everything in Go is passed by value. That is, a function always gets a copy of the thing being passed, as if there were an assignment statement assigning the value to the parameter. For instance, passing an int value to a function makes a copy of the int, and passing a pointer value makes a copy of the pointer, but not the data it points to.

What does that mean ?

It means that if you pass a variable to a function, the function always gets a copy of it. Remember always. So the caller and callee have two independent variables with the same value. Hence, If the callee modifies the parameter variable, the effect is not visible to the caller.

Lets prove it via an example

package main

import "fmt"

type person struct {
    Name string
}

func main() {
    p := person{Name: "Smith"}
    fmt.Println("Value of name before calling updateName() is: ", p.Name)
    updateName(p)
    fmt.Println("Value of name after calling updateName() is: ", p.Name)
}

func updateName(p person) {
    p.Name = "John"
}

Output:

Value of name before calling updateName() is:  Smith
Value of name after calling updateName() is:  Smith

As you can clearly see that the value of name even after calling updateName function is unchanged.

Lets try the same with Go slices

I tried the same on Go slices and it demonstrated some pretty interesting or you can say surprising behaviour. Lets quickly have a look at the code below:

package main

import (
    "fmt"
)

func main() {
    greetings := []string{"Hi", "Welcome", "Hola"}
    updateGreetings(greetings)
    fmt.Println(greetings)
    
}

func updateGreetings(greetings []string) {
    greetings[0] = "नमस्ते"
}

Output:

[नमस्ते Welcome Hola]

As you can see the value of first element of slice greeting is changed. Now this is completely opposite of what we have seen for Go struct.

Why slices are behaving differently ?

We are seeing this change in behaviour because the way Go slices are implemented internally. Lets take a minute to understand how Go slices are implemented. Have a look at the diagram below:

So when we make a slice of string, Go internally is creating two separate data structures.

The first is what we refer to as the slice. The slice is a data structure that has 3 elements inside it:

  1. Pointer to array: is a pointer over to the underlying array that represents the actual list of items.
  2. Capacity: is how many elements it can contain at present
  3. Length: is the number of elements referred to by the slice

The second is the actual array that represents the actual list of items. So lets have a look at what happens in memory when we declare a slice.

As you can see the slice at address 0002 is pointing to array stored at address 0003. Lets have a look at what happens when we pass the greetings slice to function updateGreetings.

As you can see, Go is still behaving as a pass by value language, as it is making of a copy of the slice data structure at address 0005. Now here is the very important thing, even though the slice data structure is copied, it is still pointing at the original array in memory at address 0003.

When we modify the slice inside the function, we are modifying the same array that both copies of slice pointing to. So in Go slices are what referred as reference types. 

Are there any more reference types in Go ?

So slices are not the only data structure that behave in this fashion, there are also other types which behave exactly the same way. In the below diagram, I have segregated the value types and reference types in Go.

The point to note here is that while passing on reference types, we do not need to pass address of the type. Go will handle it and any change in the variable will be reflected in the caller function. When we are passing value types like int, bool etcand we expect that the changes in value should be reflected, we must use pointers.

Reference:

  1. https://www.udemy.com/go-the-complete-developers-guide/
  2. https://blog.golang.org/go-slices-usage-and-internals

Conclusion

This is one Go gotcha which can lead to many many issues when we start Go programming. Just keep in mind the diagram of value and reference types. Hope this post helps you to avoid getting into issues in your program. Thanks for reading. Please like and share the post so that it can reach to other valuable readers too.

 

Advertisements
Posted in General | Leave a comment

Does Sports Technology impact the Healthcare Sector?

Wearable Tech & Preventative Healthcare_LinkedIn

In the ever-evolving technological landscape, emerging disruptive technologies like machine learning, deep learning and artificial intelligence have empowered industries like healthcare and sports significantly. Both industries are interconnected. Both cannot function without human involvement, and they are also dependent on informative data from each other due to the increasing demand of predictive analytics. Predictive analytics applications use metrics that can be measured and analyzed to predict the likely behavior of individuals, machinery or other entities. Today’s tech-savvy audience is constantly on the look-out for technology that is efficient, quick and time-saving.

The rising data revolution with digital transformation

Data analytics is becoming increasingly popular within healthcare, sports and life sciences professionals.  All these industries are also currently embracing innovations such as wearable-based technologies, faster computing and smaller form factor or devices.

“Paradoxically, the evolution of machine learning, which aims to raise the threshold of intelligent analysis beyond that of the human brain, can teach us more about what it means to be human.”

Today our smart-phones can not only be used as biometric devices but also can be used as a platform from which to deliver tailored algorithm analysis that can optimize personal metrics in real time.

The need for real-time data analysis is more real than ever. Analytics based reports and surveys are becoming increasingly popular with researchers because this helps them monitor trends in real time and directly impacts innovation in products. The entry of chatbots in these industries is an example of a perfect solution that helps in bringing all the initial data together, with absolute accuracy and less time consumption. This data can be collected as per different parameters, such as age, gender, location, medical history, diet, fitness regime and what not, and in turn to be used by, let’s say insurance companies when they chart out plans for a premium. Pretty amazing, right?

 

Emergence of Telemedicine 

Telemedicine is gaining popularity within the masses, but at the same time, it can safely be assumed that telemedicine is not going to replace visits to the doctor completely. Extending healthcare accessed within the home will lower healthcare costs. The departure from the treatment of illnesses to the renewed convergence on prevention is a symbol of the new 2020 healthcare patient. Health care is about more patient outcomes and less about elaborate fee structures. Technologies, primarily involving chatbots, have paved their way into the healthcare industry, allowing automation of services and leading to increased productivity with optimum results.

Precision medicine

Treating individuals by using therapies specific to them with the help of pools of data collected through smartphone apps and mobile biometrics is the backend of precision medicine. This provides patients the information about their health while simultaneously analyzing data. Misuse this data and personal information is prevented by adhering to Health standards like HIPPA. What precision medicine does is this: instead of viewing patients as end users of healthcare services, it engages more with them like partners, a role that is key to accelerate such initiatives further. It integrates patient-generated health data from different devices to better understand the disease and how it can not only be a physical, but mental burden for them. Using data analytics to better patient care is a sure shot method of moving towards efficient, sustainable models of care that are driven by data and technology and are mutually beneficial for healthcare professionals and patients alike.

There are certain cases where innovative techniques are being used to gather information, for example information from parents about infants in the Intensive Care Nursery (or NICU). Parents refrain from filling out a survey every day because it can be quite stressful and repetitive. Instead, UCSF and Benioff’s Children hospital now use an intelligent chatbot to communicate with parents. This has reduced stress for the parents as it’s now personal and now they feel they are talking to someone. The chatbot also converses intelligently to gather the baby’s symptoms for the doctor’s diagnosis later on. It also educates the parents with videos and web links so that parents can learn more about the medical condition of their baby and be more aware!

Data analytics for healthcare: Learning from sports technologies!

There is an incredible wealth of available data in sports but capturing and making use of that data in a way that will lead to better outcomes for the team remains a major problem. It has also been observed that many sports organizations find traditional data science methods to be out of their league.

Let’s take the example of NBA. In the last decade, the NBA has undergone a data science revolution that has entirely changed the game. They have used data to optimize performance in real time and built strategies to increase the chances of teams winning. Basketball is an incredibly difficult game to study, simply because it’s quick and difficult to keep track of in comparison to cricket and baseball. But NBA didn’t give up. Sophisticated tracking systems that kept their eyes on every player, machine learning and cartography helped them analyse which players were helping their teams win. Right from rebounds to three-pointers to assists, every move was analysed. Almost every NBA team had a data analyst on board to make sure this was taking place. And why did all of this happen? Because they had a senior leadership team that was invested more in the future than the present.

In healthcare, once a treatment or chain of thought becomes popular, it is hard to dislodge. It is hard to disrupt. But disruption must prevail. The medical fraternity needs to learn from the above example and take that leap of faith. Smart leaders need to be educated about the gold mine data analytics can prove to be. Somewhere, healthcare is still stuck in the data collection phase. There’s so much raw data collection happening, and private data sets, health surveys, billing records, medical sensors- everything is involved. But not all of it is being shared freely across organizations, hence, we are losing out on many insights that can be obtained. While systems are being modernized and the need for expert data scientists is now more real than ever, there are still not enough of these people on board. The result is a huge missed opportunity to deploy data in a meaningful way. It’s still not too late though, and the earlier this is recognized, the closer we will be to unleashing the true power that technology holds in the future for healthcare!

Posted in Artificial Intelligence, General, Healthcare | Tagged , , , , , , | Leave a comment

AI-led Chatbots – A boon to Healthcare Industry

The Dawn of Bots_Infographic LinkedIn

The demand of healthcare services is higher than ever, and this industry is deeply sensitive and complex in nature. In the recent years, innovations in AI-led technologies have tried to maximize productivity and help in saving time and effort of healthcare professionals. Technologies, primarily involving chatbots, have paved their way into the healthcare industry, allowing automation of services and leading to increased productivity with optimum results.

Chatbots: First level support for doctors

Chatbots are a great tool for first level support for doctors. For e.g. gathering initial symptoms, taking surveys and helping doctors take informed decisions of the patient’s condition. While the healthcare industry is sensitive to patient privacy and involves the risk of incomplete or incorrect diagnosis, technologies like chatbots can ensure that all information is gathered accurately and well in time. This will indeed save a doctor’s time and help them provide patients with a better-informed diagnosis. However, it is important that Chatbots should be used only for first level support and not for generating a diagnosis or treatment because nothing can replace a doctor’s analysis, their expertise and the required human intervention.

Optimum use of Chatbots in Health insurance

Popular names like Siri, Alexa, Cortana and Google Assistant are all chatbots that have seeped into the field of Healthcare to enhance its functioning. Healthcare insurance demands immense amount of data gathering and analysis with systematic mapping of patients, records, medical history etc. The traditional methods of health insurance lack the efficiency that AI-based Chatbots bring to the table. A decision tree based Chatbot can help health insurance companies gather data from its customers or potential customers using interesting, innovative surveys, and friendly chats with people that enable them to give out more data in a recurring manner. In a world that is guided by disruptive technology, these developments in the healthcare field and especially for services like health insurance are nothing less than a revolution. With the evolution of Chatbots, it would be very convenient for health tech companies in data management and patients’ assistance through optimum utilization of Chatbots.

Usage of Chatbots is scalable and agile

It’s always convenient to chat with someone than filling out a form instead. Usage of smart phones, languages and texting has become so common, and therefore, Chatbots have become an ideal AI-led tool is used to gather data from the audience without being intrusive in nature. The received data is priceless for many healthtech companies as they not only get past medical history but also current health statistics. Also, it helps in a company’s brand as it helps them keep track of customers better and thus chart out strategies for proper customer retention management.

Huge business potential for Chatbots in the future

There is a huge business potential for Chatbots in the healthcare industry, as it there is massive change incoming with digital transformation. To be completely tech-driven for real-time results is the need of the hour. Currently, the growing implementation of Chatbots in the healthcare sector is not a development that is temporary. With a demand of frequent data analysis reports, like research studies and surveys to name a few, there are some initial steps for product creation in that also need to be monitored. Chatbots ’entry in this industry is a perfect solution which helps in receiving all the initial data with absolute accuracy and less time consumption.

According to an insights study, it is estimated that the global Chatbot market will reach $1.23 billion by year 2025. The staggering numbers are a sign of the developments that shall shape the future of the healthcare industry, allowing both, the doctor and the patient to have a fluent interpretation and implementation of things.

These evident advancements have paved way for an enticing future for technological developments like Chatbots to make a turn for the better in the healthcare Industry. By 2019, up to 40 percent of large businesses are likely to integrate virtual assistants like Microsoft Cortana, Apple’s Siri, Amazon Alexa, or Google Assistant into their day-to-day workflows. With these established figures that are witnessing constant growth, healthcare, in terms of technology and efficiency, has the potential to redefine the field at a pace that once could not be imagined.

 

 

Posted in Artificial Intelligence, General, Mobile Development, Search, Search Engine Optimization | Leave a comment

9 Awesome Tips for Go Developer

Content posted here with the permission of the author Anuj Verma, who is currently employed at Josh Software. Original post available here.

I have just started learning Go and found it to be a very interesting language. It bridges the gap between rapid development and performance by offering high performance like C, C++ along with rapid development like Ruby, Python.

Through this blog, I wanted to share some behaviours of Go that i found tricky along with some style guidelines.

Un-exported fields in struct can be a mystery

Yes it was a mystery for me when I started. My use case was simple, I was having an object of Person struct and wanted to marshal it using encoding/json package.

package main

import (
    "encoding/json"
    "fmt"
)

type Person struct {
    name string
    age  int
}

func main() {
    p := Person{name: "Anuj Verma", age: 25}
    b, err := json.Marshal(p)
    if err != nil {
        fmt.Printf("Error in marshalling: %v", err)
    }
    fmt.Println(string(b))
}

Output:

{}

Oh things worked fine without any error. But wait, why is the response empty ? I thought it must be some typo. I checked and checked and checked…
I had no idea why things were not working. Then I asked to every developer’s god(Google). You will not believe me but this was the first time  I understood the real importance of exported and un-exported identifiers in Go.
Since encoding/json is a package outside main and the fields inside our struct name and age are un-exported (i.e begins with a small case), therefore encoding/json package does not have access to Person struct fields and it cannot marshal it.

So to solve this problem, I renamed fields of Person struct to Name, Age and it worked like a charm. Check here

json.Decode vs json.Unmarshal ?

I was once writing an application which makes HTTP call to Github api. The api response was in JSON format.
So to receive the response and use it I created a Go struct (GithubResponse) matching the format of API response. The next step was to deserialise it. After looking up from the internet I came up with two possible ways to do it.

var response GithubResponse
err = json.NewDecoder(req.Body).Decode(&response)
var response GithubResponse
bodyBytes, _ := ioutil.ReadAll(req.Body)
err := json.Unmarshal(bodyBytes, &response)

Both will exactly do the same thing and will de-serialise a JSON payload to our go struct.  So I was confused about which one to use ? After some research I was surprised to know that using json.Decode  to de-serialise a JSON response is not a recommended way. It is not recommended because it is designed explicitly for JSON streams.

I have heard JSON, what is JSON stream ?
Example JSON:

{
  "total_count": 3,
  "items": [
    {
        "language": "ruby"
    },
    {
        "language": "go"
    }
  ]
}

Example JSON stream:

{"language": "ruby"}
{"language": "go"}
{"language": "c"}
{"language": "java"}

So JSON streams are just JSON objects concatenated. So if you have a use case where you are streaming structured data live from an API, then you should go for json.Decode. As it has the ability to de-serialise an input stream.
If you are working with single JSON object at a time (like our example json shown above), go for json.Unmarshal.

var declaration vs :=

So this one is just a cosmetic suggestion, Remember when declaring a variable which does not needs an initial value prefer:

var list []string

over

list := []string{}

There’s no difference between them, except that the former may be used at package level (i.e. outside a function), but the latter may not. But still if you are inside a function where you have the choice to use both, It is a recommended style to use the former one.

Rule of thumb is to avoid using shorthand syntax if you are not initialising a variable.

Imports using blank identifier

In one of my application we are using postgres database. I am using “lib/pq” which is a go postgres driver for database. I was going through the documentation here and I saw this:

import (
    "database/sql"

    _ "github.com/lib/pq"
)

func main() {
    connStr := "user=pqgotest dbname=pqgotest sslmode=verify-full"
    db, err := sql.Open("postgres", connStr)
    if err != nil {
        log.Fatal(err)
    }
}

Is this correct ? Why are we using an underscore in front of a package import. Checking on the internet I found that it is an anonymous import. It will import the package, but not give you access to the exported entities.

So the next question is very obvious:
If I do not have access to package entities, why we are importing it?

You remember when I said Go is an interesting language. In Go we can define an init() function in each source file, which allows us to setup things before the program executes. So sometimes we need to import a package so that its init() function gets called, without using the package directly in code.

Now lets understand why in code snippet above github.com/lib/pq is imported as a blank identifier. Package database/sql has a function

func Register(name string, driver driver.Driver)

which needs to be called to register a driver for database. If you have a look at this line from lib/pq library, things become more clearer. So lib/pq is calling the Register function to register an appropriate database driver even before our main function executes.

So even we are not using lib/pq directly from our code, but we need it to register driver postgres  before calling sql.Open().

Naked Returns

In Go return values can be named. When we name a return value, they are treated as variables defined at top of the function.

func Insert(list []string) (err error) {
    // Do Stuff here
    return
}

This creates a function-local variable by name err, and if you just call return with no parameters, it returns the local variable err.

Rule of thumb is that we should use naked return if the function is short (handful of lines). They can harm readability in longer functions.

Use shorter variable names in limited scope

In most of the languages you might have observed that it is advised to use descriptive variable names. For example use index instead of i. But in Go it is advised to use shorter variable names for variables with limited scopes.

For a method receiver, one or two letters is sufficient. Common variables such as loop indices and readers can be a single letter (ir). More unusual things and global variables need more descriptive names.

Rule of thumb is:

The further from its declaration that a name is used, the more descriptive the name must be.

Examples:

Good Style

// Global Variable: Use descriptive name as it can be used anywhere in file
var shapesMap map[string]interface{}

// Method
// c for receiver is fine because it has limited scope
// r for radius is also fine
func(c circle) Area(r float64) float64 {
  return math.Pi * r * r
}

 

Explicitly ignore a json field

If you want to ignore a field of struct while serialising/de-serialising a json, you can use json:"-". Have a look at an example below:

type Person struct {
    ID      int    `json:"-"`
    Name    string `json:"name"`
    Age     int    `json:"age"`
    Address string `json:"address"`
}

In above struct ID field will be ignored while serialising/de-serialising.

Backquotes to the rescue

The back quotes are used to create raw string literals which can contain any type of character. So if you want to create a multi line string in Go, you can use back quotes. This will help you to save the effort for using escape characters inside string.

For example, suppose you want to define a string containing a JSON body:

{"name": "anuj verma", "age": 25}

See the below two ways:

b := "{\"name\": \"anuj verma\", \"age\": 25}"// Bad Style
b := `{"name": "anuj verma", "age": 25}`// Good Style

Comparing strings can be tricky

If in your code you need to compare a string with empty string, before comparison do not forget to trim spaces.
resultString == “”, may produce incorrect results as resultString can contain extra spaces(”    “)

strings.TrimSpace(resultString) == "" // good style

Conclusion

What am I missing here? Let me know in the comments and I’ll add it in. If you enjoyed this post, I’d be very grateful if you’d help it spread by sharing. Thank you.

Posted in General | Leave a comment

What I learned from my first ever software development internship

Content posted here with the permission of the author Viraj Chavan, who is currently employed at Josh Software. Original post available here.

I was a student at an engineering college in India. After 3 and a half years years of learning computer science academically, I now had a chance to test my knowledge in the real world through an internship.

In this article, I’ll be sharing my internship experience at Josh Software, Pune with the hope that it is helpful to other IT and computer engineering students that are looking for internships.

Like most of my colleagues at the college, I had a very limited view about software development in general and didn’t know what to expect from an internship.

Lucky for me, I was assigned a live project, which was based on Ruby on Rails, something that I had already developed an interest for.

After I had learned PHP and MySQL in the 2nd year of my studies, I built a basic web app, and all that it did was some CRUD (Create, Read, Update, Destroy) operations. I remember talking with a friend who had similar skills to mine, and said “Even we can build Facebook now that we know PHP and MySQL!”

How ridiculously simple things seemed at that time. Now I understand how complex building/maintaining a software can be.

So here’s what I learned from my Internship while working on a live project.

 

General lessons

Scale Makes a huge difference

  • How many users are going to use the software?
  • How much data will be processed?
  • What are the expected response times for a function?

These are questions that we, as college students, hardly think about. Our college projects were usually short-sighted. In real-world projects though, the above questions fundamentally affect decisions about hardware, technologies/tools to be used, system architecture, algorithms, and so on.

Working with a large codebase

Back in college, we used to work on projects that had like 15 – 20 files or so. Built in under a week, the whole project could be understood in a few hours.

Now the project I’m working on has hundreds of files spread across dozens of folders. It can take months to understand the whole project, and hours to debug a bug that’s spread across multiple files. And the first time you look at the whole project directory, you don’t know where to start understanding the code.

Writing maintainable code

Knowing that the code you write will be read, understood, and improved/changed by someone else (or even yourself) in the future makes you write code that’s maintainable.

In college, all I focused on was getting the expected functionality to be complete, and never considered whether the code I wrote was maintainable.

This resulted in scrambled pieces of code that somehow worked at the time. But two days later even I wouldn’t understand why I had written a certain piece of code that way. And changing some part of the code almost always broke other parts. 😆

Code Maintainability is easier to recognise by its absence, like when something you thought should take an hour ends up taking a week.

Using a version control system – properly

When I first started building small software, all the files existed on my own development machine, and maybe they were backed up to Google Drive as regular files.

Then I got to know about GitHub, but I merely used it as a safe storage place for my code. I used the GitHub desktop app to commit all changes on just the master branch. I even hesitated using it through the command line.

Now not a day goes by that I don’t use Git. It’s such a great tool for collaboratively writing code, distributed development, branching out for new features, pull requests, and so on.

Here’s a little article on why version control systems are awesome!

The importance of using a Test Driven Development approach

During my internship, I was assigned to work on a new feature that was to be added to the main project .

I wrote the code and tested if it was working the way it was supposed to. It worked perfectly, or so I thought. I deployed the feature to the production confidently, and moved on to work on something else.

After a few hours, Rollbar, a real time error reporting tool burst with a number of errors in our code deployed to production. I checked the errors and they seemed unrelated to anything I had ever worked on.

After some debugging, all of those errors traced back to a single method. A method that was called in numerous places, and in which I had modified just a single line, and hadn’t checked where else it was used.

Now this could’ve been avoided if the code that used that method had test cases written for it, and if I had checked if all the test cases ran successfully before deploying the code. That made me realize the importance of test driven development.

Here’s an article to understand why writing test cases is important.

Things specific to Ruby on Rails/ Web Development

The MVC Architecture

Back in my college days, when I developed applications in PHP, I had no clue what Model, View, and Controller were. Any project was so complexly scrambled that I couldn’t find in which file a piece of important logic was written. The HTML embedded PHP scripts at odd places and I had placed all the files in just one folder.

Then I learned about the Rails framework, and got accustomed with the MVC architecture.

Model-View-Controller (MVC) is an architectural pattern that separates an application into three main logical components – Model, View, and Controller. Each of these components are built to handle specific development aspects of an application (source)

MVC really simplifies things and is an important part of many major frameworks.

Dealing with Databases

In the last 6 months, I haven’t written a single direct SQL database query. Yet I deal with databases everyday, even doing some complex operations. This is thanks to the ORM (Object Relational Mapper) that Ruby On Rails uses.

ORMs convert object-oriented programming language such as Ruby into database lingo in which to perform operations. Which makes data access more portable and abstracted from the required database queries that are necessary when manipulating data.

Thanks to ORM, it’s much much easier to query the database. This gives a big advantage to beginners, who can start writing applications without even knowing SQL.

Writing/Using REST APIs (Application Programming Interfaces)

APIs make it easier for one application to talk to another.

APIs make some other applications’s functionalities easily accessible to our application. For example, I once developed a Road Trip Planner application that used the Google Maps API to show various places on a map that a user could visit on a particular route.

APIs can also be used to separate the front-end and the back-end completely. For example, we can write the back-end as an API-only Rails application that can be used by a web site, an Android/iOS application, or even some third party applications.

Using ElasticSearch for searching

Although I don’t know much about ElasticSearch so far, but I’ve learned that it’s a NOSQL, distributed full text database. It acts as a distributed search engine that is incredibly easy to scale and returns results at lightning speed.

Why would we need it for searching? Because having millions of records in a regular database can make it really complex to make efficient searches.
With Elasticsearch, we can index the documents needed to be searched and it can perform queries across all those millions of documents and return accurate results in a fraction of a second.

Elasticsearch has a Restful API, which makes it really easy to query the searches and get the results.

Here’s a tutorial that helped me, and here are some use cases of Elasticsearch.

Using asynchronous/background tasks

Sometimes the user will perform an action on our application that takes a considerable amount of time to complete. We don’t want the user to sit there waiting for this action to complete, so we send it off to a background worker.

Here’s a link that explains it better.

In Ruby On Rails, I came across Sidekiq, which makes it easy to handle background tasks efficiently.


Thanks for reading! If you found this article helpful, give me some claps. 👏

There’s still a long way to go!

Check out my Github profile here.

Posted in General | Leave a comment

Rails: Conserve your database by “audit”ing its space in right way !!!

Content posted here with the permission of the author Ganesh Sagare, who is currently employed at Josh Software. Original post available here.

In most of the Rails applications, we track the important data for auditing. Most of the time, database table in which these audit records are stored, is under same database of our application.

Keeping this table in same database is helpful until it doesn’t grows tremendously. We use this table most of the time for analysis & sometimes for recovering the data. As this size increases, it will start showing impacts in space consumption, like, increase in database size and backup size and also time taken for database backup.

There are multiple reasons for size of table to increase like

  • tracking lots of columns from different tables
  • tracking more actions happening on data

So to optimize our database & backup storage usage and to increase speed in backup process, we just thought what if we store this history/audit records in another database and we found its very easy to do so.

First lets see advantages of this.

  • Avoid rapidly growing database size.
  • Reduced database backup size.
  • Speed up in backup process.
  • Data isolation.

Now lets see how to store audit records to second database.

1. Update your Gemfile

We used gem audited to keep track of our data. So you can add below entry to your Gemfile.

gem “audited”, “~> 4.7”

2. Create configuration for second database.

We can configure our application to connect to second database using YAML file similar to our database.yml code.

# config/audited.yml

development:
encoding: utf8
adapter: postgresql
database: audit_development
port: 5432

production:
encoding: utf8
adapter: postgresql
database: audit_production
port: 5432

The purpose of this configuration file is to have a nice clean place to store our database connection configuration options.

Note: Assuming database has already been created, and running on default postgres port i.e 5432.

3. Connect to the second database.

Using ActiveRecord::Base.establish_connection method, we can connect to second database. Using our YAML configurations let’s create connection to our second database.

Also let Audited::Audit table (i.e table which stores audit records) to read & write data to second database.

# config/initializers/audited.rb

AUDIT_DB = YAML.load_file(
File.join(Rails.root, "config", "audited.yml")
)[Rails.env.to_s]

# Configure Audited to read/write to second database
Audited::Audit.class_eval do
establish_connection AUDIT_DB
end

4. Create “audits” table in second database

Audited gem uses audits table to store model related changes. You can generate migration for audits table using below command,

rails generate audited:install

For more information refer to gem documentation.

# db/migrate/20180629113852_install_audited.rb

class InstallAudited < ActiveRecord::Migration[5.2]
def self.up
create_table :audits, :force => true do |t|
t.column :auditable_id, :integer
t.column :auditable_type, :string
t.column :associated_id, :integer
t.column :associated_type, :string
t.column :user_id, :integer
t.column :user_type, :string
t.column :username, :string
t.column :action, :string
t.column :audited_changes, :text
t.column :version, :integer, :default => 0
t.column :comment, :string
t.column :remote_address, :string
t.column :request_uuid, :string
t.column :created_at, :datetime
end

add_index :audits, [:auditable_type, :auditable_id], 
:name => 'auditable_index'
add_index :audits, [:associated_type, :associated_id], 
:name => 'associated_index'
add_index :audits, [:user_id, :user_type], :name => 'user_index'
add_index :audits, :request_uuid
add_index :audits, :created_at
end

def self.down
drop_table :audits
end
end

Wait, we don’t simply want to run this migration, because this will update schema in our Rails application, instead we want this migration to be executed on our second database.

Hence we need to update generated migration so it should connect with second database as below. (Note changes highlighted.)

# db/migrate/20180629113852_install_audited.rb

class InstallAudited < ActiveRecord::Migration[5.2]
def self.up
 Audited::Audit.connection.create_table :audits, 
:force => true do |t|
t.column :auditable_id, :integer
t.column :auditable_type, :string
t.column :associated_id, :integer
t.column :associated_type, :string
t.column :user_id, :integer
t.column :user_type, :string
t.column :username, :string
t.column :action, :string
t.column :audited_changes, :text
t.column :version, :integer, :default => 0
t.column :comment, :string
t.column :remote_address, :string
t.column :request_uuid, :string
t.column :created_at, :datetime
end

 Audited::Audit.connection.add_index :audits,
[:auditable_type, :auditable_id], :name => 'auditable_index'
 Audited::Audit.connection.add_index :audits,
[:associated_type, :associated_id],:name => 'associated_index'
 Audited::Audit.connection.add_index :audits,
[:user_id, :user_type], :name => 'user_index'
 Audited::Audit.connection.add_index :audits, :request_uuid
 Audited::Audit.connection.add_index :audits, :created_at
end

def self.down
 Audited::Audit.connection.drop_table :audits
end
end

And then execute migration in order to create our table in second database.

rake db:migrate

That’s it, now all your audit records will be stored in second database.

Happy auditing !! 😃

Posted in General | Leave a comment

Preventing Machine Downtime by Predicting it Beforehand

For the past few months, I have been observing the growth of the manufacturing sector in India, and how the contribution of the manufacturing sector to the India’s gross domestic product (GDP) will increase from the current levels of ~16% to 25% by 2022.

One of the major concerns and challenges of having a seamless manufacturing output is to prevent & avoid unfavorable machine performance. With the assumption that machines will degrade over time, manufacturing companies, prior to advanced technology intervention, aimed at focusing on preventive and reactive maintenance of the health of their machines, but the use of deep learning technology is leading towards a new age term method to safeguarding the health of machines, coined in the industry as predictive maintenance.

Predictive maintenance technology approaches can help the manufacturing sector to find the optimal inflection point between costs and machine failures. But, predictive maintenance is not as simple as a plug ‘n’ play solution as the requirement of machine learning requires layers of historic data to be collected over time.

Consider the life-cycle of a CNC machine. Today, most CNC manufacturers define the maintenance cycles based on the type of work the CNC machine does for their customer. It is based on their individual experience and judgement. However, if we were to get not just real-time data on the display but also store and analyze the historical data and use of the CNC machine, deep learning algorithms could find out the pattern of use and predict the maintenance and life of the CNC machine.

False positives would occur, i.e. a situation where the algorithm may predict the maintenance incorrectly based on the parameters it has to play with. With some human intervention, this pattern is corrected, learnt, and applied on the following data set to improve the result. So, the algorithm can learn from its mistake and give more relevant and accurate results over time.

Using cloud based scalable technologies, we could reduce the infrastructure requirements at each premise and even customize the maintenance cycle for each CNC machine based on the customer’s usage patterns. This will not only reduce the cost of maintenance but also improve the efficiency – a win win for both the CNC machine manufacturer and their customer!

Deep Neural Networks are used in this approach to learn from sequences from data. Unscheduled machine downtime can be damaging for any business. Preemptive identification of these issues can help enhance quality of production and significantly improve supply chain processes. The advantages of using predictive maintenance strategies can enhance overall operational efficiency.

Predictive Maintenance strategy is built on the fundamental methodology of Internet of Things (IoT). IoT will not be functional without data and machine learning. This approach is not only about gathering data, but also creating an ecosystem to predict and make decisions as a response to the sequences of data collected. Predictive maintenance will be a larger opportunity as global economies progress, and IT solutions providers need to look at this opportunity to further innovate to help manufacturing companies disrupt their industries.

Posted in General | Leave a comment