Rails command line options I hardly ever use.

A good rake refresher!

Learning shall never stop..!!!

This is the blog post about few things in Rails command line that I hardly ever use.

rake notes

Left a comment like #FIXME or #TODO on a method, but never actually end up changing anything or fixing anything about the method. If this sounds familiar, then rake notes helps you avoid just that. It will find all the files with comments beginning with OPTIMIZE, FIXME, OPTIMIZE.

Screen Shot 2016-05-29 at 12.49.47 pm

Also, we can add search for specific annotation, for ex: searching for just FIXME comments would be as simple as firing rake notes:fixme.

Also it lets us search for any custom annotation that we might be using in our code, rake notes:custom ANNOTATION=ADDTESTCASE

Read more about rake notes on the guides.

rails console –sandbox

If you wish to test some code, but you also want the data to be reverted back to original after you have tested the…

View original post 442 more words

Posted in General | Leave a comment

Using Golang with Api.ai Artificial Intelligence service

Part 2 — In this article, we will try out their /tts service.

In part 1 of this series we explored their /query service. Here, we will explore their/tts service which takes text and returns the generated speech (as an audio WAV file).

I have created a folder “apiai” which will hold the Go source code “apiaifile.go”.

C:\go_projects\go\src\github.com\SatishTalim\apiai

We shall be running our program at the command prompt in the folder “apiai” as follows:

go run apiaifile.go

apiaifile.go

Here’s the complete source code:

package main

import (
        "fmt"
        "io"
        "log"
        "net/http"
        "os"
)

func main() {
        url := fmt.Sprintf("https://api.api.ai/v1/tts?v=20150910&text=The+Go+programming+language+rocks!")

        // Create file to store the .wav file
        output, err := os.Create("d.wav")
        if err != nil {
                log.Fatal("Create: ", err)
                return
        }
        defer output.Close()

        // Build the request
        req, err := http.NewRequest("GET", url, nil)
        if err != nil {
                log.Fatal("NewRequest: ", err)
                return
        }        

        // Replace Your_Client_Token with your actual token
        req.Header.Add("Authorization", "Bearer Your_Client_Token")
        req.Header.Add("Accept-language", "en-US" )

        // For control over HTTP client headers,
        // redirect policy, and other settings,
        // create a Client
        // A Client is an HTTP client
        client := &http.Client{}

        // Send the request via a client
        // Do sends an HTTP request and
        // returns an HTTP response
        resp, err := client.Do(req)
        if err != nil {
                log.Fatal("Do: ", err)
                return
        }
        
        // Callers should close resp.Body
        // when done reading from it
        // Defer the closing of the body
        defer resp.Body.Close()       
        
        if _, err := io.Copy(output, resp.Body); err != nil {
                log.Fatal("Copy: ", err)
                return
        }

        fmt.Println("File d.wav downloaded.")
}

https://api.api.ai/v1/tts?v=20150910&text=The+Go+programming+language+rocks!

The above URL processes the phrase “The Go programming language rocks!” and generates an audio file from it.

func Create(name string) (*File, error)

“Create” creates the named file, truncating it if it already exists.

func NewRequest(method, urlStr string, body io.Reader) (*Request, error)

“NewRequest” returns a new “Request” given a method, URL, and an optional body. “NewRequest” returns a “Request” suitable for use with “Client.Do”.

func (h Header) Add(key, value string)

“Add” adds the key, value pair to the header. It appends to any existing values associated with key.

// Replace Your_Client_Token by your actual Client access token 
req.Header.Add(“Authorization”, “Bearer Your_Client_Token”)
req.Header.Add("Accept-language", "en-US" )

A single header “Accept-language” is used to identify TTS language. For each API request, include the above HTTP headers.

client := &http.Client{}

A “Client” is an HTTP client.

resp, err := client.Do(req)

“Do” sends an HTTP request and returns an HTTP response. When “err” is nil, “resp” always contains a non-nil “resp.Body”.

defer resp.Body.Close()

Callers should close “resp.Body” when done reading from it. Use “Defer” for closing the body. “resp.Body” is of type “io.Reader”.

func Copy(dst Writer, src Reader) (written int64, err error)

“Copy” copies from src to dst until either EOF is reached on src or an error occurs. It returns the number of bytes copied and the first error encountered while copying, if any.

Here’s the file “d.wav” that got downloaded using our program.

That’s it!

Posted in Go, Tutorials | Tagged , , | Leave a comment

Using Golang with Api.ai Artificial Intelligence service

Part 1 — In this article, we will try out their /query service.

Api.ai is a service which provides a nice combination of both voice recognition and machine learning for developers. We shall use their free tier.

Api.ai has “Domains”. Domains are a whole collection of knowledge and data structures from Api.ai that are ready for use in every Api.ai agent (apps are called “agents” in Api.ai).

Api.ai has SDKs for most languages except Golang. Nevertheless, we shall use Go to write a program called “apiai.go” which will query the weather for a particular place and display the result.

I have created a folder “apiai” which will hold the Go source code “apiai.go”.

C:\go_projects\go\src\github.com\SatishTalim\apiai

We shall be running our program at the command prompt in the folder “apiai” as follows:

go run apiai.go

apiai.go

As mentioned in their docs, a sample query response is:

{
  "id": "cfcbd337-6b66-4393-a6a3-74fc5487cedb",
  "timestamp": "2016-02-16T00:30:13.529Z",
  "result": {
    "source": "agent",
    "resolvedQuery": "hi my name is Sam",
    "action": "greetings",
    "actionIncomplete": false,
    "parameters": {
      "name": "Sam"
    },
    "contexts": [
      {
        "name": "user_name",
        "parameters": {
          "name": "Sam"
        },
        "lifespan": 5
      },
      {
        "name": "greetings",
        "parameters": {
          "name": "Sam"
        },
        "lifespan": 5
      }
    ],
    "metadata": {
      "intentId": "c251ef97-0c43-404d-bf75-98e806f942be",
      "intentName": "Greetings"
    },
    "fulfillment": {
      "speech": "Hi Sam! How can I help you?"
    }
  },
  "status": {
    "code": 200,
    "errorType": "success"
  }
}

JSON-to-Go is an excellent tool that instantly converts JSON into a Go type definition. Using it, I get:

type QResponse struct {
        ID string `json:"id"`
        Timestamp time.Time `json:"timestamp"`
        Result struct {
                Source string `json:"source"`
                ResolvedQuery string `json:"resolvedQuery"`
                Action string `json:"action"`
                ActionIncomplete bool `json:"actionIncomplete"`
                Parameters struct {
                        Name string `json:"name"`
                } `json:"parameters"`
                Contexts []struct {
                        Name string `json:"name"`
                        Parameters struct {
                                Name string `json:"name"`
                        } `json:"parameters"`
                        Lifespan int `json:"lifespan"`
                } `json:"contexts"`
                Metadata struct {
                        IntentID string `json:"intentId"`
                        IntentName string `json:"intentName"`
                } `json:"metadata"`
                Fulfillment struct {
                        Speech string `json:"speech"`
                } `json:"fulfillment"`
        } `json:"result"`
        Status struct {
                Code int `json:"code"`
                ErrorType string `json:"errorType"`
        } `json:"status"`
}

Here’s the complete “apiai.go” program:

package main

import (
        "encoding/json"
        "fmt"
        "log"
        "net/http"
        "time"
)

type QResponse struct {
        ID string `json:"id"`
        Timestamp time.Time `json:"timestamp"`
        Result struct {
                Source string `json:"source"`
                ResolvedQuery string `json:"resolvedQuery"`
                Action string `json:"action"`
                ActionIncomplete bool `json:"actionIncomplete"`
                Parameters struct {
                        Name string `json:"name"`
                } `json:"parameters"`
                Contexts []struct {
                        Name string `json:"name"`
                        Parameters struct {
                                Name string `json:"name"`
                        } `json:"parameters"`
                        Lifespan int `json:"lifespan"`
                } `json:"contexts"`
                Metadata struct {
                        IntentID string `json:"intentId"`
                        IntentName string `json:"intentName"`
                } `json:"metadata"`
                Fulfillment struct {
                        Speech string `json:"speech"`
                } `json:"fulfillment"`
        } `json:"result"`
        Status struct {
                Code int `json:"code"`
                ErrorType string `json:"errorType"`
        } `json:"status"`
}

func main() {
        url := fmt.Sprintf("https://api.api.ai/v1/query?v=20150910&query=weather&lang=en&latitude=35.925&longitude=-86.8688889&sessionId=1234567890")

        // Build the request
        req, err := http.NewRequest("GET", url, nil)
        if err != nil {
                log.Fatal("NewRequest: ", err)
                return
        }        
        // Replace 9ea93023b7274cfbb392b289658cff0b by your Client access token
        req.Header.Add("Authorization", "Bearer 9ea93023b7274cfbb392b289658cff0b")

        // For control over HTTP client headers,
        // redirect policy, and other settings,
        // create a Client
        // A Client is an HTTP client
        client := &http.Client{}

        // Send the request via a client
        // Do sends an HTTP request and
        // returns an HTTP response
        resp, err := client.Do(req)
        if err != nil {
                log.Fatal("Do: ", err)
                return
        }
        
        // Callers should close resp.Body
        // when done reading from it
        // Defer the closing of the body
        defer resp.Body.Close()       
        
        // Fill the record with the data from the JSON
        var record QResponse   
        
        // Use json.Decode for reading streams of JSON data
        if err := json.NewDecoder(resp.Body).Decode(&record); err != nil {
                log.Println(err)
        }

        fmt.Println("Status = ", record.Status.Code)
        fmt.Println("Response = ", record.Result.Fulfillment.Speech)
}

https://docs.api.ai/docs/reference allow you to submit queries and get text-to-speech results.

All the URLs have the following base:

https://api.api.ai/v1/

The query endpoint is used to process natural language, either in the form of text or a sound file. The query requests return structured data in JSON format with an action and parameters for that action. Get /query — Takes natural language text and information as query parameters and returns information as JSON.

We are going to find out the weather at Franklin, Tennessee, USA.

Query parameters

Clipboard01

https://api.api.ai/v1/query?v=20150910&query=weather&lang=en&latitude=35.925&longitude=-86.8688889&sessionId=1234567890

With the parameters in the above table, our URL is as shown above.

fmt.Sprintf

“Sprintf” formats and returns a string without printing it anywhere.

func NewRequest(method, urlStr string, body io.Reader) (*Request, error)

“NewRequest” returns a new “Request” given a method, URL, and an optional body. “NewRequest” returns a “Request” suitable for use with “Client.Do”.

func (h Header) Add(key, value string)

“Add” adds the key, value pair to the header. It appends to any existing values associated with key.

// Replace 9ea93023b7274cfbb392b289658cff0b by your Client access token 
req.Header.Add(“Authorization”, “Bearer 9ea93023b7274cfbb392b289658cff0b”)

For each API request, include the above HTTP header.

client := &http.Client{}

A “Client” is an HTTP client.

resp, err := client.Do(req)

“Do” sends an HTTP request and returns an HTTP response. When “err” is nil, “resp” always contains a non-nil “resp.Body”. Callers should close “resp.Body” when done reading from it. Use “Defer” for closing the body. “resp.Body” is of type “io.Reader”.

defer resp.Body.Close()

Next, “NewDecoder” returns a new decoder that reads from “io.Reader”. A “Decoder” reads and decodes JSON objects from an input stream.

func NewDecoder(r io.Reader) *Decoder

“Decode” reads the next JSON-encoded value from its input and stores it in the value pointed to by v.

func (dec *Decoder) Decode(v interface{}) error

Finally, we extract the information from our populated “QResponse” struct variable “record”.

Here’s a sample output of the program:

Response = It’s currently 63 degrees and cloudy in Franklin, Tennessee. It will be intermittent clouds and around 62 degrees this afternoon. Expect a thunderstorms evening, with temperatures around 66 degrees.

That’s it!

Posted in Go, Tutorials | Tagged , , | 1 Comment

Use Go and Clarifai API to Recognize NSFW Images

The ‘Not Safe For Work’ (NSFW) model analyzes images and videos and returns probability scores on the likelihood that the image or video contains pornography. For those of you who don’t know what NSFW is, we’re talking about Not Safe For Work. R-rated content. Stuff that would probably get you fired if you were looking at it at work.

I was reading “How To Use Clarifai To Protect Your Eyes From Seeing Something They Can’t Unsee” and thought of writing a Go program instead of Python, as used in that blog post.

To write our Go program, we shall build upon what we learned in my previous article “Using Go with an image and video recognition API from Clarifai”.

Models

When images or videos are run through the tag endpoint, they are tagged using a model. A model is a trained classifier that can recognize what’s inside an image or video according to what it ‘knows’. Different models are trained to ‘know’ different things. Running an image or video through different models can produce drastically different results.

If you’d like to get tags for an image or video using a different model, you can do so by passing in a model parameter. If you omit this parameter, the API will use the default model for your application.

NSFW

The ‘Not Safe For Work’ model analyzes images and videos and returns probability scores on the likelihood that the image or video contains pornography.

The response for NSFW returns probabilities for nsfw (Not Safe For Work) and sfw (Safe For Work) that sum to 1.0. Generally, if the nsfw probability is less than 0.15, it is most likely Safe For Work. If the nsfw probability is greater than 0.85, it is most likely Not Safe For Work.

We shall be using the Model: nsfw-v1.0 and the following image available at the url — https://samples.clarifai.com/nsfw.jpg:

nsfw

Here’s the complete code for our Go program “mynsfw.go”.

package main

import (
        "encoding/json"
        "fmt"
        "net/http"
        "net/url"
        "strings"
)

const (
        clientID     = "Your ClientID"
        clientSecret = "Your Client Secret"
)

type TokenResp struct {
        AccessToken string `json:"access_token"`
        ExpiresIn   int    `json:"expires_in"`
        Scope       string `json:"scope"`
        TokenType   string `json:"token_type"`
}

type TagResp struct {
        StatusCode string `json:"status_code"`
        StatusMsg string `json:"status_msg"`
        Meta struct {
                Tag struct {
                        Timestamp float64 `json:"timestamp"`
                        Model string `json:"model"`
                        Config string `json:"config"`
                } `json:"tag"`
        } `json:"meta"`
        Results []struct {
                Docid uint64 `json:"docid"`
                URL string `json:"url"`
                StatusCode string `json:"status_code"`
                StatusMsg string `json:"status_msg"`
                LocalID string `json:"local_id"`
                Result struct {
                        Tag struct {
                                ConceptIds []string `json:"concept_ids"`
                                Classes []string `json:"classes"`
                                Probs []float64 `json:"probs"`
                        } `json:"tag"`
                } `json:"result"`
                DocidStr string `json:"docid_str"`
        } `json:"results"`
}

type NSFWResp struct {
        StatusCode string `json:"status_code"`
        StatusMsg string `json:"status_msg"`
        Meta struct {
                Tag struct {
                        Timestamp float64 `json:"timestamp"`
                        Model string `json:"model"`
                        Config string `json:"config"`
                } `json:"tag"`
        } `json:"meta"`
        Results []struct {
                Docid float64 `json:"docid"`
                URL string `json:"url"`
                StatusCode string `json:"status_code"`
                StatusMsg string `json:"status_msg"`
                LocalID string `json:"local_id"`
                Result struct {
                        Tag struct {
                                Classes []string `json:"classes"`
                                Probs []float64 `json:"probs"`
                        } `json:"tag"`
                } `json:"result"`
                DocidStr string `json:"docid_str"`
        } `json:"results"`
}

func main() {
        accessToken, err := requestAccessToken()
        if err != nil {
                fmt.Println(err)
        }
        
        nsfw, err := getNSFW(accessToken)
        if err != nil {
                fmt.Println(err)
        }

        if nsfw[0] > nsfw[1] {
            fmt.Println("Safe for work, you can trust Bob again (for now)!")
        } else {
            fmt.Println("Not safe for work. Classic Bob.")
        }
}

func requestAccessToken() (string, error) {
        form := url.Values{}
        form.Set("grant_type", "client_credentials")
        form.Set("client_id", clientID)
        form.Set("client_secret", clientSecret)
        formData := strings.NewReader(form.Encode())

        url := fmt.Sprintf("https://api.clarifai.com/v1/token/")

        req, err := http.NewRequest("POST", url, formData)
        if err != nil {
                return "", err
        }

        req.Header.Set("Content-Type", "application/x-www-form-urlencoded")

        httpClient := &http.Client{}

        resp, err := httpClient.Do(req)
        if err != nil {
                return "", err
        }

        defer resp.Body.Close()

        var record TokenResp

        if err := json.NewDecoder(resp.Body).Decode(&record); err != nil {
                return "", err
        }

        return record.AccessToken, nil
}

func getNSFW(token string) ([]float64, error) {
        // Analyze the image at https://samples.clarifai.com/nsfw.jpg
        url := fmt.Sprintf("https://api.clarifai.com/v1/tag/?model=nsfw-v1.0&url=https://samples.clarifai.com/nsfw.jpg")

        req, err := http.NewRequest("GET", url, nil)
        if err != nil {
                return nil, err
        }

        req.Header.Set("Content-Type", "application/json")
        req.Header.Set("Authorization", "Bearer "+token)

        httpClient := &http.Client{}

        resp, err := httpClient.Do(req)
        if err != nil {
                return nil, err
        }

        defer resp.Body.Close()
        
        var record NSFWResp

        if err := json.NewDecoder(resp.Body).Decode(&record); err != nil {
                return nil, err
        }
        
        return record.Results[0].Result.Tag.Probs, nil
}

The main thing to observe in the above program is the url:

url := fmt.Sprintf(“https://api.clarifai.com/v1/tag/?model=nsfw-v1.0&url=https://samples.clarifai.com/nsfw.jpg")

in the function:

func getNSFW(token string) ([]float64, error) {

and the new structure:

type NSFWResp struct {
        StatusCode string `json:”status_code”`
        StatusMsg string `json:”status_msg”`
        Meta struct {
                Tag struct {
                        Timestamp float64 `json:”timestamp”`
                        Model string `json:”model”`
                        Config string `json:”config”`
                } `json:”tag”`
        } `json:”meta”`
        Results []struct {
                Docid float64 `json:”docid”`
                URL string `json:”url”`
                StatusCode string `json:”status_code”`
                StatusMsg string `json:”status_msg”`
                LocalID string `json:”local_id”`
                Result struct {
                        Tag struct {
                                Classes []string `json:”classes”`
                                Probs []float64 `json:”probs”`
                        } `json:”tag”`
                } `json:”result”`
                DocidStr string `json:”docid_str”`
        } `json:”results”`
}

When you run the program:

go run mynsfw.go

The output you get is:

Safe for work, you can trust Bob again (for now)!

That’s it!

Posted in Go, Tutorials | Tagged , , | Leave a comment

seldom used but powerful aggregation operators

Siva talks about the MongoDB Aggregation operators and how to use them to get the maximum benefits. Good code samples help us understand complex use-cases!

rails learning

Most programmers who use MongoDB may have heard about its aggregation framework but only a few have used it to its full potential. Recently, in one of our projects, we got a complex requirement where the user needed to generate a report of sales done for any day, month or year. Using just MongoDB queries would have been very slow compared to the aggregation framework. So I decided to take up the challenge and learn the aggregation operators and make my code awesome! Here are the operators that helped me:

1. $and

$and is one of the MongoDB aggregation conditional operators where you can specify multiple conditions. Any document which satisfies all these conditions would get forwarded to next stage in the pipeline which is nothing but $group stage. Here is an example

The above example returns the $agreement_amount if it is today’s booking.

2. $add and $subtract

Similar to…

View original post 200 more words

Posted in General | Leave a comment

CodeCuriosity – Beginnings of the curious coder

Sometimes what starts as an experiment becomes something much bigger. That’s exactly what happened with CodeCuriosity.

For the past few years, we have always encouraged people at Josh to do open-source contributions. This helps everyone:

    * It helps you build your public profile and showcase your strengths.
    * You learn new techniques and coding practices from the community that improves your daily work.
    * Your contributing back and it makes you happy!

How does this benefit the company, you ask? When we have people who have a prolific history of open-source contributions, it’s always easy to convince our potential customers that they are working with the right team!

Some team members were ardent and committed contributors but we wanted EVERYONE to get ‘addicted’ to open-source contributions. For the past few years, we had made a lot of attempts to entice, motivate and even force the team to increase their open source contributions:

Open source Fridays was started with the intent to get everyone together to contribute to some open source work. Invariably, it ended up having a few dedicated “idea owners” building something they were passionate about. After a year, we stopped it because I hated the idea of “forcing” people to do something even if it’s for their own good. It’s like wearing seat-belts😉

3rd Saturdays was an initiative, where every 3rd saturday of the month was declared a working day but dedicated only for open-source activities. You could write blog posts, commit code, answer questions on StackOverflow, teach or attend in-house workshops etc. This worked for a few months but then people started giving very innovative and genuine excuses to skip these mandatory Saturdays.

Last year around Sept-2015, after some brain-storming, we realised that we needed 3 key ingredients

Open-source contributions should be fun, competitive and have continuity.

So, we started the CodeCuriosity exercise. Here is a screenshot of the first email I sent to the team.
Screen Shot 2016-04-25 at 12.54.10 pm
The key factors were:

  • Teams of 3 people
  • You have to contribute to other popular repositories. Not your own “test app”.
  • Everything is scored – commits and activities.
  • Prizes are given to the winning team.

To ensure that we are able to monitor everything, we built a platform that allowed us to monitor commits, activities and score everything. For the first few months, we started scoring manually (at least 3 scores were required to get a full average score). However, that quickly became a great problem to solve – we were drowned in open-source activities and could not do any scoring manually! (an algorithm followed).

The first month, we had everyone trying to get used to this, so it was skewed.

Screen Shot 2015-11-24 at 12.03.19 pm

The second month, everyone caught on and caused a better competition! EUREKA!

Screen Shot 2015-11-27 at 3.22.45 pm

We also realised over a few months that we may have succeeded in “Gamifying Open Source contributions”. We then decided to open this out to the community and allow us to help not just people in our company but everywhere.

CodeCuriosity – The push you need to contribute to open-source.

    * Points for every commit and activity you make.
    * Contributions count only when made to a “valuable repository” i.e. a repos that has more than 25 stars.
    * Points redemption for digital goods.
    * Groups level the playing field. So, you now compete only among your peers.
    * Automated scoring that can be over-ridden – a learning algorithm that can help make scoring more accurate.
    * Royalty Bonus for all your past efforts and contributions to open-source.

Above all – CodeCuriosity is open-source and contributions are welcome at http://github.com/joshsoftware/code-curiosity

Talks given about Code Curiosity:

Sethupathi Asokan – RubyConf India 2016

Gautam Rege – Collision Conf 2016 (video awaited)

Posted in General | Tagged | 2 Comments

Software Engineering Vs Programming

[Original post by Anil Maurya is here – posted here with his permission]

When I was growing up, Software Engineer and Programmer seems same term to me.

Now after working a couple of years as a Software Engineer I think I can distinguish between the two terms.

Programming is a small part of Software Engineering.

Software Engineering include

Understanding Requirement
Designing Software
Organising code
Testing Software
Collaborating with other software engineers

Writing good code is essential skill for being a Good software engineer but you can not neglect others skills as they are equally important for building good product.

I read something similar in Software Engineering subject during my graduation, but none of the above make sense than. I think reading about software engineering is not the great way to learn about it instead build a lots of software and you will get a grasp of software engineering.

What text books do not tell you about Software Engineering is :

Understanding Requirement:

Understanding requirement is easy but always keep in mind that its going to change in future.
Question everything, until it make sense to you.

Designing Software & Organising Code:

People don’t pay much attention about organising code when writing software but its not healthy for software health. When Software grows old, non-organised software is difficult to take care off.

Keep It Simple

To check if you are doing it right, think with perspective of a person who does not know anything about your software. Now if you can still navigate through the software without anyone’s help than you are doing it right else you need to step back and make your software simple enough to be understood by all.

Testing Software:

Nobody can teach you the importance of testing until you realise it and it may be too late when you realise its importance, therefore its important to learn from the mistakes of others.

Like all other engineering department , software engineering is also about building product. Civil engineer design bridges, buildings and then act upon and make it happen. Similarly we design software and write code to make it happen. But the main difference is the change in requirement we get every now and than. People take it for granted that changes in software is not a big deal.

If your software is not automated tested than your software will surely break in future with all the inevitable change in requirements coming.

Collaborating with other software engineers:

Text books don’t tell you that you will be working with other people on same software. Collaborating with others is essential and you can not run away from it. Collaboration helps deliver better product if people working together are passionate about the product.

Its always better to have different opinions and healthy discussion on possible solutions.

To all Software Engineer out there, lets collaborate and make something awesome. cheers:)

Posted in General | Leave a comment