Gmail API(via IMAP) In Ruby on Rails – A piece of cake

Gautam Rege:

Adding Gmail IMAP directly into a Rails application is quite easy. As Pramod explains in this post.

Originally posted on Deep In Rails :

We often need to read user email messages from Gmail, In this post I will be taking you through how to access user inbox messages from Gmail using gmail_xoauth gem and Ruby Net::IMAP Net library in your Rails application.

I have the sample example file on gist , I would suggest refer this example side by side with this post.

gmail_xoauth allows you to authenticate Gmail IMAP and STMP via OAuth, using the standard Ruby Net libraries. Gmail Platform provides XOAUTH2 mechanism a OAuth 2.0 protocol used for IMAP AUTHENTICATE and SMTP AUTH commands.

Ruby implements Internet Message Access Protocol (IMAP) client functionality in Net::IMAP Net libraries.

Before we start here are some prerequisite,

1) You need to Authorize user via Google using omniauth-google-oauth2 with omniauth configuration specified in sample omniauth_config.rb

2) While Authorize you must define following scopes.

1

3) If you are accessing user email in offline mode then you need…

View original 395 more words

Posted in General | Leave a comment

When I travelled halfway across the world to attend RubyConf 14 San Diego

It would be folly to even try and sum up all my experiences in one blog post. Hence this part is a short recap of my impressions of RubyConf San Diego.

DSC_0033After a long journey of ~12000kms, little sleep, lots of coffee, and several moments of uncertainty ; I finally arrived at San Diego Convention Center (warm, sunny, seaside host to ComicCon!)

DSC_0023

 

Yukihiro Matsumoto a.k.a Matz, the creator of Ruby, was going to kick-start the conference, and when he started to speak, all that uncertainty vanished, and pure excitement took its place. Matz is an absolute celebrity in the Ruby community, as one could guess from the number of people asking for his autograph (me included!). So when he started talking about his future plans for Ruby 3.0, the whole hall was hooting and cheering. His three main ideas for Ruby 3.0 revolved around JIT compilation, concurrency, and static typing. Now, I admit, quite some of it was beyond my understanding, but when he talked about ‘static typing’ in Ruby, it was huge. Mainly because, it affects the very core of Ruby. “Will Ruby still be Ruby if it were statically typed?” Even though Matz had raised some existential questions, he had made it clear, how only ~30% of his ideas actually made it into production code. He also stressed on the idea of ‘soft-typing’ through which he aims to combine best of both dynamically and statically typed languages. One of the papers he referenced to regarding Soft typing, can be found here.

Over the next three days, I attended many talks, some useful, some inspiring, and some amusing; and picking favourites won’t be just! So, here are some highlights in no particular order.

I love DIY projects. And luckily there were a few talks revolving around using Ruby and Raspberry pi to build some awesome stuff! Christopher sexton‘s talk about using beacons to locate presence and identify a person approaching a room opens up a plethora of applications for beacons. His use case for the technology, to play intro-music for your boss or colleagues at office was really amusing though. Jonan Scheffler‘s talk explored home security software, and home-made motion detection software.

Testing is integral to ruby, and two talks in particular were quite informative and useful. The first one by Sam Phippen was about ‘Spies’ in Rspec. Spies allow us to test if a method has been called, using ‘have_received’ which can be used effectively to improve tests. The second talk by Brock Wilcox was about a gem called pry-timetravel that allows you to go back to checkpoints set during debugging. This way you could rewrite history! (use different sequences of commands ‘next’, ‘step’, etc while debugging, without running the code again and again..) The gem itself is promising and uber cool, but comes with some disclaimers; namely memory exhaustion and partial time-travel (you can only go back to check-points).

Amy Wibowo‘s talk about how her team used Ruby to hack an old sewing machine was a classic underdog success story; and we all love those right? They fed the machine 8-bit digital prints through a hacked floppy drive to knit into sweaters. This talk reminded everyone how Ruby’s simplicity was its greatest strength, and how it can encourage anyone to code and bring forth their creativity to the community.

Sandi Metz‘s keynote made quite an impact as she weaved through examples from our history where major technological advancements wiped out many jobs off the planet. Her key message; to live in the present, focus on things greater than life, adapt and evolve.

Talking up a classic topic, Algorithms, Richard Schneeman‘s  talk explained in simple and clear language, the ‘Levenshtein’ algorithm for calculating distances. The information theory algorithm used by various auto-complete and auto-correct softwares is an efficient alternative to Hamming distances. He also talked about a cool gem called ‘did_you_mean‘ which is an excellent example of distance algorithms. You can find more information about this talk here.

Lastly, Jim Gay gave some great uncomplicated advice to write better object oriented code. He emphasized on simple indicators such as strings of ‘if-else’ and ‘case-when’ that call for need to re-factor. He veered away from the big words and explained each concept through implementation. To quote him “Tell, don’t ask”. His slides can be found right here.

Making new friends is always interesting – Laila Zaki.

DSC_0034
RubyConf 14 was one of the most exciting and happening technical events I had ever attended. Be it the excellent talks, beautiful venue or an amazing audience, they had it all. Kudos to Ruby Central who did a superb job at organizing the event, and to the sponsors for the awesome freebies!

There was an awesome crazy lightning talk about keeping Ruby weird.

DSC_0042

Not everyone is fortunate enough to have their company send them to attend a conference anywhere in the world. Lucky for me, I happen to work at Josh Software. And  I have to thank them for the opportunity to attend RubyConf 14 at San Diego; which was pretty awesome! I am sure it will have a lasting impact on my career.

Hoping to get lucky next year too!

Image | Posted on by | Tagged , , , , , , , , , | Leave a comment

How to create nested form using AngularJS

In a Rails application, when we need nested forms, we frequently use the ‘nested_form’ gem. However, we cannot use that when we are making an AngularJS + Rails app. To make nested forms using AngularJS, we need to write some specific code and in this blog I do just that. I have some sample code base with a working demo and hope it helps you out.

In this example, we have a User model and an Address model. A User has many addresses. Naturally, when we are adding or updating a user, we want to manage multiple addresses for that user in the same form.

The sample code is hosted on my Github repos and I also have a demo hosted on Heroku.

Adding the UserAddCtr Action


myApp.controller("UserAddCtr", ['$scope', '$resource', 'Users', '$location', function($scope, $resource, Users, $location) {
  $scope.user = {addresses: [{street1: '', street2: '', city: '', state: '', country: '', zipcode: '' }]}
  $scope.save = function () {
    if ($scope.userForm.$valid){
       Users.create({user: $scope.user}, function(){
       $location.path('/');
    }, function(error){
       console.log(error)
    });
  }
 }

 $scope.addAddress = function(){
   $scope.user.addresses.push({street1: '', street2: '', city: '', state: '', country: '', zipcode: '' })
 }

 $scope.removeAddress = function(index, user){
   var address = user.addresses[index];
   if(address.id){
     address._destroy = true;
   }else{
     user.addresses.splice(index, 1);
   }
 };

}]);

In the above code:

  • “$scope.user” has been initialized with an addresses array to ensure that the addresses are nested inside the scope of a user.
  • “$scope.addAddress” function is used to add a new address in the form. This is done simply by push a blank address $scope.user.addresses.
  • “$scope.removeAddress” function is used to remove an address from user form. Here we need to decide whether to mark for deletion from the database or simply remove from the form. If the address.id is present, we need to set “address._destroy = true” to mark it for deletion. (This is similar to what we do when using the nested_form gem for deleting nested attributes). If the address.id is not present, this was newly added in the form, so we simply remove it from the array using “user.addresses.splice(index, 1)”.

Now, lets see the html we have for the user form. For brevity, I am showing only the address template code here. You can check the complete user form html code here.


<div ng-repeat="address in user.addresses">
 <div ng-hide="address._destroy">
 <div class="form-group">
 <label class="control-label col-md-2">Address 1 </label>
 <div class="col-md-3">
 <input type="text" class="form-control" ng-model="address.street1" placeholder="Address 1"/>
 </div>
 <label class="control-label col-md-2">Address 2 </label>
 <div class="col-md-3">
 <input type="text" class="form-control" ng-model="address.street2" placeholder="Address 2"/>
 </div>
 <div class="col-md-2">
 <a ng-click="removeAddress($index, user)" class="btn btn-xs btn-danger">X</a>
 </div>
 </div>
 <div class="form-group">
 <label class="control-label col-md-2">City </label>
 <div class="col-md-3">
 <input type="text" class="form-control" ng-model="address.city" placeholder="Pune"/>
 </div>
 <label class="control-label col-md-2">State </label>
 <div class="col-md-3">
 <input type="text" class="form-control" ng-model="address.state" placeholder="MH"/>
 </div>
 </div>
 <div class="form-group">
 <label class="control-label col-md-2">Country</label>
 <div class="col-md-3">
 <input type="text" class="form-control" ng-model="address.country" placeholder="India"/>
 </div>
 <label class="control-label col-md-2">Zipcode </label>
 <div class="col-md-3">
 <input type="text" class="form-control" ng-model="address.zipcode" placeholder="12345"/>
 </div>
 </div>
<span style="line-height: 1.5;"><hr/>
</span><span style="line-height: 1.5;"></div></span>
</div>
<div class="form-group">
 <div class="col-md-offset-12">
 <a ng-click="addAddress()" class="btn btn-success btn-xs">+ Add address </a>
 </div>
</div>

In this html view, we can seen “ng-repeat” to iterate over user.addresses.

‘<div ng-hide=”address._destroy”>’ hides the deleted address in the from if  _destroy=true.

‘<a ng-click=”addAddress()”>+ Add address </a>’ adds a new address.

‘<a ng-click=”removeAddress($index, user)”>X</a>’ removes the address from user form.

Hope this helps. Please send me feedback if you have a complicated case in a nested form. I will add it to my example.

Posted in Javascript, Ruby on Rails | Tagged , | Leave a comment

Contributing to Open-Source Projects

Gautam Rege:

A “straight from the heart” post by Rishi about contributing to open-source projects. This post is more about why you should contribute, how people help out and how satisfying it is.
Here is a quote from the post that sums it all up – “The only regret I have is, why did I start so late …”.

Originally posted on Learning shall never stop..!!!:

Today I would like to share what I have learned from the very little open-source contributions I have made so far. But before I do that, I would like to write a little bit about myself. I am ruby on rails developer at JoshSoftware for the past 3 years. Before that I was studying Electronics in Pune Institute of Computer Technology, Pune. Yes.. you read that right. I was an electronics student in computers college. The reason I am laying out the details is, I do not want you to think that I am some big shot who has been working for a decade. I am just a guy who likes to write programs and do web development.

So, here is the list of hurdles I came across until I made my first open-source contribution:

1. Which projects should I contribute to?
Thats a valid question. There are tons of…

View original 740 more words

Posted in General | Leave a comment

Obrigado RubyConf Portugal

There are many dayIMG_6026s of the year where nothing memorable happens, and then there are some days which leave a lasting impact on your life. I can vouch for the fact that if you ask people who attended this conference, at least 90% (including me) would say that the 2 days at the Ruby Conf Portugal fall into the latter. Whether it was the talks, the food, the vinho(wine) or the venue itself, everything was wonderful.

IMG_6029

You hell know that the conference is going to be fun when the master of ceremonies appear as caesar. Hail Jeremy !! You did a wonderful job of managing the whole energy in the auditorium and your wonderful introduction to Caleb’s talk still lingers in memory. And the sssshhhsh sshhh too :)wrong-conf

First a warm hug to all of the speakers. You were awesome. And I know I cannot contain this awesomeness in a line or two but I will still try ..

Katrina Owen – Practise simple problems to get better at the craft which will then help you in the bigger problems at hand.
Alex Coles – Rails way has not changed in the last 10 years when it comes to Front end development. And considering the growth of some of the Javascript frameworks, he suggested a mix of 2 applications where API development and JS client framework would be the way to go about designing a web app in today’s day and age.
Piotr Szotkowski – Do read about enumerable.rb and you will be a better programmer.

Carlos Souza – Some pointers to keep in mind when you are developing Web API’s.
Chris Kelly – Some of the content just went tangential to my head but I remember that a string of 23 character is better than a string of length 24. Don’t ask me why :)
Gautam Rege – “Ruby is for developers while go is for programmers”. I <3 Ruby and I know he loves it too :)IMG_6031
Erik Ober – Keeping a track of object space is important and just making those little changes in ruby code and benchmarking them can lead to sizeable improvement in performance. Do have a look at his slides for some tips and tricks. And also if you want to see some beautiful illustration.DSC_0403

That was the end of first day. But then it was not the end. There was the traditional Portugese performance in the lobby and then the Ruby Karaoke session took the fun to another level.IMG_6046

And yes even after all the wine and the fun last night, I did make it to the next day on time and I had breakfast too :)

Steve Klabnik – He introduced us to a new language rust and then gave a demo of a c extension which used a rust service. Pretty cool eh. Also I came to know that he is these days involved with the active_model_serializer gem which plans to tackle some of the pain points of the JBuilder experience.

PJ Hagerty – Mozart effect works in programming too .. ie. listening to a particular type of music does result in cognitive boost.
Piotr Solnica – You may be a clean coder(very particular about code smells) or a cowboy coder(you sometimes cut corners just to deliver more features) and both the styles are pretty acceptable as long as you have the confidence to change something or refactor if the need be. I sure need to work on my testing skills to increase my programming confidence.
Luca Guidi- It is important to reinvent the wheel and the Lotus web framework is trying to do that. Do use it if possible and provide feedback. The community can only get better with such new stuff coming up
Danish Khan – This talk did remove some of my misconceptions about sales. Awesome slides.
Caleb Thompson – He gave an awesome presentation of how scenic gem can be used to do full text searching across more than 1 table. Yes elastic search and other 3rd party search tools are there but then I would love it if there was something in rails out of the box for these advanced searches. The problem with the gem right now is that it is only for Postgres database.
Terence Lee – A great closing speech where he spoke about the ways in which you can contribute to Ruby and that is not limited to writing C code and sending in patch requests. You can open tickets for features/bugs at their ticketing website and not on twitter :) Share data about application performance which can be used to improve ruby. More or less participate in the discussion and take Ruby to the next level(3.0)

DSC_0421

Obrigado to the organizers, the volunteers and the sponsors for making the RubyConf Portugal a big success. And a thank you to Josh software for making this trip happen. That’s it from Braga. Hopefully I will there again next year.

Image | Posted on by | Leave a comment

Raspar – Build a html parser in 5 minutes

Raspar is a HTML parsing library that parses HTML pages and converts HTML to ruby object by defining a map of ‘css’ or ‘xpath’ selectors. This gem can also manage parsers for multiple websites.

The sample output looks something like this

{ product: [ 
    <Raspar::Result:0x007ffc91e4d640 @attrs => { :name=>"Test1", :price=>"10"}, 
    @domain => "example.com", @name => :product> 
    # ... 
    # ... 
    ] 
}

Why Raspar?

For almost every website that we parse, we need to customise the code to parse and convert data into our defined format. While doing this, we potentially face some of the following problems in parsing html.

  • HTML page may contain multiple items or single item with the same CSS selectors.
  • We need to collect different types of data from single page. For example, products, offers, comments etc.
  • For the single website, the HTML structure could be difference on various page. For example, on one page the product name could have a CSS selector as.product-name while on another page in the same website, it may have the CSS selector as .pname.
  • Sometimes we want to collect particular attributes as an array. For example,  in the product section, we may want the various product features that are defined in the li tag as an array.
  • Some attributes are common for all pages, for example, the product comparison page has the same name and description but other attributes would differ from page to page.

Raspar helps to solve all these problems!

Usage

You can define a parser as shown below. In this example, we are parsing a currency exchange rate website and fetching the country, currency and it’s code.

class CurrencyCodeParser
  include Raspar
  domain 'www.exchange-rate.com'

  collection :currency_code, 'table.currency-codes tr' do
    attr :country, 'td.country'
    attr :currency, 'td.currency'
    attr :code, 'td.code'
  end
end

We first include the Raspar module and register the domain that is going to be parsed. Then we have to plan the parsing strategy. In this page, there are currency codes for each country. So using the collection method, we can collect all the values based on the currency code. We can also define multiple collections – for example, in a page containing products and brands, we can define two collections, one for products and another for brands.

NOTE: You can set multiple css selectors too in order of priority. In the example below, if .country element is not set, then the .nation will be checked and returned.

attr :country, '.country, .nation'

collection: This takes two arguments and a block of code: the collection name, the html selector and the block in which the attributes are defined. In the example above, ‘table.currency-codes tr’ is a selector that contains all there attributes country, currency and the code. So, the parser collects all ‘table.currency-codes tr’ elements and makes a result object using the selectors defined for attribute.

attr: This takes two mandatory arguments: the name and html selector. It can take an optional third options argument that help in formatting or getting a particular property of the html element. Potential options are :prop, :eval. If the options are not defined, then the attr returns the text value of that html element.

:prop: This will return the value of the mentioned property of the selected element. In the case below, we want the src property of img tag.

attr :image, '.lg_photo img', prop: 'src'

:eval: This evaluates the HTML element and processes it. This can be a Proc or a method name (i.e. a symbol). Remember, the method or Proc defined must take 2 agreements: the method name and the element. For example,

attr :address, '.address', eval: Proc.new{|text, ele| text.split(':').last}

or

attr :address, '.address', eval: :parse_address

def parse_address(text, ele)
  text.split(':').last
end

If we need the attribute as an array, we can simply do the following:

attr :specifications, '.specs li', as: :array

NOTE: If attr is defined outside any collection block, it is considered a common attribute and will be included in all collections!

The parsing logic

Here is an example of the parsing a particular page in the domain we have specified. In the example below, Raspar will automatically load the parser depending on the domain, in our case the CurrencyCodeParse. We don’t need to specify this in our code. The advantage of this is that we can customise or add new parser at will as long as we specify the right domain!

url = 'http://www.exchange-rate.com/currency-list.html'

// Using RestClient get html page
html = RestClient.get(url).to_str

Raspar.parse(url, html).each {|c| p c; }

This will get us the following result:

{
  currency_code: [
    #<Raspar::Result:0x007ffc91e4d640
     @attrs={:country=>"USA", :currency=>"USD", :code =>"$"}>,
    #<Raspar::Result:0x007ffc91e57be0
     @attrs={:country=>"Japan", :currency=>"¥; ", :code =>"JPY"}>,
   ...
   ...
 ]
}

Alternate ways to create parsers

There are other ways to define a parser.

By passing a block

Here we don’t need to define a separate class; just an anonymous parser!

Raspar.add('www.exchange-rate.com') do
  collection :currency_code, 'table.currency-codes tr' do
    attr :country, 'td.country'
    attr :currency, 'td.currency'
    attr :code, 'td.code'
  end
end

By Passing a hash

This can be helpful if we have pre-defined selector map configured in the code or saved in our database or even if we want to add map dynamically i.e JSON file of web service etc.

domain = 'http://www.leguide.com'
selector_map = {
  collections: {
    product: {
      select: '.offers_list li',
      attrs: {
        image: { select: 'img', prop: 'src'},
        price: { select: '.price .euro.gopt', eval: :parse_price}
      }
    }
  }
}

In the selector map above, we have defined a :parse_price method. Here is how we can add it to Raspar. We can also define more data processing helpers in the ParserHelpermodule as shown below.

module ParserHelper
  def parse_price(val, ele)
    val.gsub(/,/, '.').to_f
  end
end

Raspar.add(domain, selector_map, ParserHelper)

This gem is available on ruby gems and on github: Raspar
You can check out various examples too.

Go forth and parse!

Posted in Ruby | Tagged , | Leave a comment

Precision number parsing in spreadsheet using ruby.

This blog post is not about how to parse spreadsheets using ruby. If you are looking for that, you are not gonna find that here. This blog post is about a problem I faced while parsing decimal numbers from a spreadsheet, long story short precision related problems. Here are the details (the longer version):

Recently I had to parse an spreadsheet. The cells in the spreadsheet could have either of strings or numbers (float or otherwise). And since this application had to do with lot of calculations, even a specification being off by a single digit could lead to wrong set of results. So my point being that it was vitally important that I parse the data as it has been entered. Seems like a regular parsing situation… I had the same thought, but it turns out excel does a lot of things in the background.

FYI: I am using ‘roo’ gem for parsing the file. And these files are written in MS Excel.

Try doing this in irb:

$ irb

irb(main):001:0> .09 + 0.0016
=> .0916000000000001
irb(main):002:0> .09 + 0.0016 == .0916
=> false

First of all this is the correct behaviour. And second of all the reason for this behaviour is because ruby uses IEEE754 doubles. And so does Excel. So if the cell has data whose value is .0916 and its format is set not set as ‘TEXT’, but something else for example: Number, General, etc, then what excel returns is .09160000000001. So even if user has entered a value like this, it could come up with a different precision altogether.

So, the question is how did I solve this problem ? Well, I did not. Not because I could not solve it, but because I was not fully satisfied with the solutions present right now and I am still trying to figure it out. But anyways I thought those solution could solve somebody else’s problem. So here are some details on those solutions:

. You could ask all your users to use the format of the data they are entering as ‘TEXT’. So excel or any other software that I am not aware of right now, will not change the values, and you would get what user has entered as it is.

. roo gem which I am using to parse the spread sheet, it has methods to find out the format of the data it is parsing. So you can just check in your code, if the format is ‘:float’, then just round the number. Here is how you could do it:

s = Roo::Excel.new('myspreadsheet.com')
s.cell_type(4,2)

But again, as I said earlier, these solutions are not right or wrong, but these are trade-offs that you have to think of in terms of whats best for your application.
If you want to follow up the issue, here are some links to look at: roo gem github, same issue in a different gem and a related issue.

Posted in Ruby | Tagged , , | 1 Comment