Tuesday, July 29, 2014

Rainbow Stream - Twitter the geek way

How a geek uses Twitter

Do you call yourself a geek ? How are you using Twitter ? 
Did you see something like this ?













This application is called "Rainbow Stream". I wrote it firstly for myself, after trying various tools out there but can not satisfy with any of them. Rainbow Stream is smart, very nicely displayed and even can draw images directly on your terminal.

Homepage : rainbowstream.org

Github Poject: github.com/DTVD/rainbowstream

How to install

Just Python and Pip in one line!


sudo pip install rainbowstream
# Python 3 users: sudo pip3 install rainbowstream

But I will recommend to use virtualenv instead



virtualenv venv
# Python 3 users : use -p to specify your Python 3 localtion as below
# virtualenv -p /usr/bin/python3 venv
source venv/bin/activate
pip install rainbowstream

More information and troubleshooting can be found at Github' repo.




Usage


You can start the app right now with 

rainbowstream


And the rainbow with the ability to display image directly on terminal can be called with 


rainbowstream -iot
# Or rainbowstream --image-on-term

In the first time you will be asked for authorization of Rainbow Stream app at Twitter. Simply click "Authorize app" button ...





to get the PIN ...

                                         

                                      and paste back to terminal.


You can find out that this app supports a massive list of commands (see Read The Docs latest document). Play with it a while and you'll like it for sure :)

Conclusion

Written by geek and for geeks, Rainbow Stream aims to be the perfect Twitter workflow on your favorite terminal. If you found any bugs or have any feature requests, you can create a new issue on Github or ping me at @dtvd88.

Monday, April 14, 2014

Inversion Of Control



The inspiration 

I had tried Laravel 4 for rebuilding a part of my company's production site. Laravel has really impressed me in many ways, and the most is its flexibility with IoC container and the implement of Facades Design Pattern. I soon wonder if I can bring all that power to other legacy source code which was written in the early era of pure MVC in PHP.

Using a framework for a fresh new project is just about reading the docs and following some tutorials, but using it in a heavily structured system is not that easy. Laravel IoC is also deeply integrated in it's components, so get a core package that only bring up the IoC container seem difficult.

For a sake of self-learning and for my own convenience, I wrote a library and published it on Packagist. You are welcome to browse source on Github and checkout it's usage in the next section.





Quick setup

Since the package was published in Packagist, you can use Composer to install it. Wonder how ? Check out Composer installation guide or follow this simple steps:
$ curl -sS https://getcomposer.org/installer | php
$ mv composer.phar /usr/local/bin/composer 
# Make sure that /usr/local/bin included in your $PATH
Then create a composer.json file in project's root folder with the following lines:
    "require": {
        "orakaro/ioc": "dev-master"
    }
and install:
$ composer install
You are ready to start now. Let go through an example.




Usage

This package provide a very simple-to-use but still powerful of Inversion Of Control (IoC) container. Image that you have a Library class like below
<?php
class Library{
  public function introduce($ISBN)
  {
    $book = Book::getTitle($ISBN);
    return 'check out '. $book;
  }
}
Note that there is another Book class elsewhere with a static method called getTitle which take an argument named ISBN. Calling another class's static method is not bad itself, but will make unit testing after become impossible. We will avoid that pain with a simple usage of IoC container.

Let's create a file IoCRegister.php in same directory level with composer.json
<?php
require_once __DIR__ . '/vendor/autoload.php';/*Shoule be correct path to composer autoload!*/
use orakaro\IoC\core\IoC;

class IoCRegister{
  public static function registerAll()
  {
    /* IoC register for Book::getTitle */
    IoC::register('Book_getTitle', function($ISBN){
        return Book::getTitle($ISBN);
    });
  }
}
IoCRegister::registerAll();

class IoCBook{
  /* IoC resolver for Book::getTitle */
  public static function getTitle($ISBN)
  {
    $registedClosure = IoC::resolve('Book_getTitle');
    return $registedClosure($ISBN);
  }
}
What is happening here ? Let me explain a bit about this piece of code.

Firstly we included '/vendor/autoload.php' which is automatically generated by composer autoloader, allow us to use the IoC container library at the next line 'orakaro\IoC\core\IoC'.

The IoCRegister class generates an alias 'Book_getTitle' for a closure which returns the real method Book::getTItle($ISBN). The IoCBook class have a getTItle method which "resolve" the 'Book_getTitle' alias and actually call above closure. 

At the end, original Library class can be rewrite like
<?php
require_once __DIR__ . '/vendor/autoload.php';
require_once __DIR__ . 'IoCRegister.php';
class Library{
  public function introduce($ISBN)
  {
    $book = IoCBook::getTitle($ISBN);
    return 'check out '. $book;
  }
}


How IoC container help Unit tetsing

In PHPUnit it's never easy to deal with static methods. Although PHPUnit has the capability of mocking static methods where caller and callee are in same class, static methods from separated class are just death to testability. This means that Book's static method getTitle can not be mocked and we can not test our original Library class.

With above example, Library class use the IoC container IoCBook::getTitle($ISBN), so inject a mock object is quite simple:
<?php
require_once __DIR__ . '/vendor/autoload.php';
require_once __DIR__ . 'Library.php';
use orakaro\IoC\core\IoC;

class LibraryTest extends PHPUnit_Framework_TestCase{
  /**
   * Bootstrap for each test case
   */
  public function setUp()
  {
    /* Redefined your desirable mock */
    IoC::register('Book_getTitle', function($ISBN){
      return 'mocked title';
    });
  }

  /**
     * Ending for each test case
   */
  public function tearDown()
  {
  }

  /**
   * test introduce method
   */
  public function testIntroduce()
  {
    $l = new Library;
    $this->assertSame('check out mocked title', $l->introduce('1234567890'));
  }


Conclusion

In this post I just introduced some very basic usage of IoC container in development and unit testing. We can keep the changes to legacy source is minimum by grouping all IoC class in a separated namespace, and better to have a generator to do all the rest of re-defining classes. The next post will cover about these features and ends up with a more complete solution.



Saturday, March 29, 2014

Problem with pycrypto when install Fabric (Linux/MacOS)



Fabric in CentOS 5

Fabric is a Python library allow me to write deployment task in Python like Ruby did with Capistrano. Despite Capistrano seems a bit difficult, Fabric is straight forward. 


My virtual machine runs on CentOS 5 when libGMP is installed with version 4. When I tried to install Fabric, pip threw a warning look like:
PowmInsecureWarning: Not using mpz_powm_sec. You should rebuild using libgmp >= 5 to avoid timing attack vulnerability.
The problem is, Fabric has a dependency called Pycrypto and it seems not happy with libGMP version 4. So I had to reinstall libGMP manually and link pycrypto with this version.



Install libGMP manually

Follow steps below

mkdir /home/local
cd
wget ftp://ftp.gmplib.org/pub/gmp/gmp-5.1.1.tar.bz2
tar jxf gmp-5.1.1.tar.bz2
cd gmp-5.1.1
./configure --prefix=/home/local
make
make check
make install

Now we have gmp-5.1.1 installed in /home/local. Next step is install Python and build pycrypto with this GMP version



Custom Install Python 2.7 and Pycrypto.

CentOS is shipped with Python version 2.4 which cause so many syntax error. First we need to install Python2.7 as well as pip2.7 manually ...

cd /home
mkdir localpython
wget https://www.python.org/ftp/python/2.7.6/Python-2.7.6.tgz --no-check-certificate
tar -xvzf Python-2.7.6.tgz
cd Python-2.7.6
./configure --prefix=/home/localpython --enable-shared
make
make install
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/localpython/lib
export PATH=$PATH:/home/localpython/bin
cd /home
wget https://raw.github.com/pypa/pip/master/contrib/get-pip.py --no-check
python2.7 get-pip.py

... and use pip to install virtualenv in which we will build a custom pycrypto base on libGMP version 5 at previous section.

pip install virtualenv
cd /home
virtualenv venv
source venv/bin/activate

If virtualenv was installed correctly, we should be in a bash with "(venv)" at beginning. Next step is about download Pycrypto, modify it's setup.py and install it before install fabric

cd venv/
pip list | grep pycrypto
pip uninstall pycrypto
pip install --download ./ pycrypto
tar zxvf pycrypto-2.6.1.tar.gz 
cd pycrypto-2.6.1
vim setup.py
# Edit line 373 as below
@@ -373,7 +373,8 @@
       'ext_modules': plat_ext + [
             # _fastmath (uses GNU mp library)
             Extension("Crypto.PublicKey._fastmath",
-                      include_dirs=['src/','/usr/include/'],
+                      include_dirs=['src/','/home/local/include/','/usr/include/'],
+                      library_dirs=['/home/local/lib'],
                       libraries=['gmp'],
                       sources=["src/_fastmath.c"]),
cd ..
pip install ./pycrypto-2.6.1
pip install fabric
fab --list

The warning should be clear now :-)



MacOSX Maverick's note

Maverick is already shipped with Python version 2.7.5, but fabric installation will fall with Xcode5.1 -mno-fused-madd error. The workaround is installing fabric with following command.

ARCHFLAGS=-Wno-error=unused-command-line-argument-hard-error-in-future pip install --upgrade reportlab fabric





Wednesday, March 26, 2014

Sublime Text under a vimmer's eye


Sublime Text - the editor you will fall in love with

There are already too many people writing about Sublime Text from these days, about  it's ton of amazing features or well-known perfect workflows. Sublime Text is really impressive and made a great impact on me, but I will not re-introduce these cool features or cover any workflow in this post. 

I just want to share my customized Sublime and viewpoint from a vimmer nerd. 


For your information, the theme is a customized Larapaste theme, available in My Gists


From Vim to Sublime Text

I still remember my first day of Vimmer's life. That was a shock to know that 15-years-old editor like Vim has so many fans, and all of them love the way of using 'i' to start editing, which made me feel nothing but disgust at that time. 

But when I learned the power of modal editing, I soon became vim faithful, started to want to edit any file, code at any language with vim. I felt I can not live without the movement of "hjkl" keys, and just can not stand for using mouse when editing files anymore. Vim is superior, is hardcore, and the killerest.

But, I'm also a highly visual guy, and the highly visual eye of mine always said to me that her want to look at more comfortable font, more beautiful interface, when my thoughts told me that all of the comfort movements should be kept.

And then 2 sweet girls appeared : Vintage mode in Sublime and IntellJ's IdeaVim. I will cover IntellJ IDEA in another posts soon.


Why Sublime Text is awesome ?

Sublime Text + Vintage Mode + VintageEx plugin gave me all the power of vim's movement, and Sublime itself brought up the elegant fonts, beautiful themes and the power of super-fast fuzzy search-everywhere. I can quickly split windows into views, traverse through views or even sidebar with my favourite vim's key-binding. 

And Sublime gave more: Multiple Selection save a lot of time in editing variable's name, Instant Project Switch made me easy to switch over multiple workspace, and Package Control is a just-feel-right elegant tool for install and remove bunch of plugins, which I will share in the next section of this post.



My favourite plugins

Here it is. Take a look at Package Control.sublime-settings file

{
 "auto_upgrade_last_run": null,
 "installed_packages":
 [
  "Alignment",
  "All Autocomplete",
  "BracketHighlighter",
  "Case Conversion",
  "CSV",
  "CTags",
  "DocBlockr",
  "Emmet",
  "Git",
  "GitGutter",
  "Goto Documentation",
  "Jedi - Python autocompletion",
  "IMESupport",
  "Laravel Color Scheme",
  "LiveStyle",
  "MoveTab",
  "Package Control",
  "PHPUnit Completions",
  "PlainTasks",
  "Prefixr",
  "Quick File Creator",
  "Search Stack Overflow",
  "SideBarEnhancements",
  "SimpleClone",
  "Smarty",
  "sublime-github",
  "SublimeLinter",
  "SublimeREPL",
  "Super Ant",
  "TrailingSpaces",
  "Twitter Bootstrap ClassNames Completions",
  "View In Browser",
  "VintageEx",
  "WordHighlight"
 ],
 "repositories":
 [
  "https://github.com/koizuss/sublime-gradle"
 ]
}

As you can see, MoveTab, SimpleClone is for arranging view groups and moving files around. SideBarEnhancements provide notable basic file operator. Ctags is must-have plugin for jumping up and down inside source base. These essential things gave me everything I want in navigating through development.


WordHighlight, SublimeLinterAll Autocomplete or BracketHighlighter made Sublime much like a really IDE, while Git, GitGutter, sublime-github or Search Stack Overflow brought up direct interaction with your version control system or your browser as well. 


If you are a front-end engineer, you also may take a look to the Emmet, Prefixr and LiveStyle. They save me much time in developing HTML and CSS. 


Finally SublimeREPL is a really interesting one. It can run a interactive interpreter inside Sublime view tab at various languages below


Clisp
Clojure
CoffeeScript
Elixir
Execnet Python
Erlang
F#
Groovy
Haskell
Lua
Matlab
MozRepl
NodeJS
OCaml
Octave
Perl
PHP interactive mode
PowerShell
Python
R
Racket
Ruby
Scala
Scheme
Shell (Windows, Linux and OS X)
SML
Sublime internal REPL (?)
Tower (CoffeeScript)

I really think it's author did a great job.



Conclusion

Sublime Text became my most favourite editor because of two main reason. One is that it can serve me just like Vim, and another one is its beautiful interface, highly customizable settings. 

As a Java or Scala developers, maybe i need a giant like IntellJ IDEA, but when I develop in Python, Ruby or PHP, Sublime is just-feel-right comfortable zone even better than vim. 




Saturday, March 22, 2014

Python IRC bot


Why I needed an IRC bot ?

I have been in a group of nerdy developers which was college friends, now working in different country and culture. We use a private IRC room on #freenode as a place to share our thoughts, to discuss or just chat with each other. But our different GMT lead to different online time of each member, which causes misunderstanding of discussion and time wasted in re-explaining. 

So we just created a system which have a bot to log all our chat text, and a tiny website to display log chat, also let us browse the log for date or time easily. I did the part of the bot, and would like to share all it's source as well as its ability of chatting, calculating or logging chat text.


Batmanbot can talk with you

I named my bot is "BatmanBot". BatmanBot can greet when you join channel, reply to some basic chat, calculate math problems exactly like Python interpreter and, most important purpose, log all out chat to a database. Here is some examples ...



... and this is chat log in private message. We can also view all logs at our web interface.




Roll your own robot

First, take a look at BatmanBot's source at My Github Repo. You will need Python 2.6+ and sqlite3 command available to summon the BatmanBot :-) Once you are ready, let's clone, build your config and database as below.



Next, create a config file "config.py" include IRC server, channel and password if needed. Note that all constant name should be exactly as below or you have to modify "BatmanBot.py" file to adapt to your own configuration. 

SERVER = ["irc.freenode.org"]
CHANNEL = "#gotham"
PASSWORD = ""
NICKNAME = ["BatmanBot"]
MAIN_SERVER = "irc.freenode.org"

Your bot is ready right now. Wake him up!

python BatmanBot.py


Conclusion

Write an IRC bot is easy and straightforward in Python. But if you want more intelligence, more functionality or some other other cool features, you maybe need to look at Hubot - an excellent robot commissioned by Github. 

BatmanBot was just a homework for me, and a play around for us trying attack and defend the Python eval function. Try to "import sys" and do some crazy thing like "rm -rf" when talking to BatmanBot :-) 




Friday, March 21, 2014

RSS Reader wrote by Python Flask


Why RSS Reader ?

It's 2013 July that Google stopped the Google RSS Reader service. My company held a small programming contest for creating similar service at any programming language, which winners can claim for about 50000yen.

I would like to share with you my work done at that time as well as it's prize :D I used about 45000yen to buy a Happy Hacking Keyboard Professional JP, an 128Gb SD card and 4Gb memory RAM :-)



The Project

My Project called PyRSSReader, wrote in Python Flask and deployed at Heroku. You can check Github repo and test everything out at Heroku public link.




Flask is a microframework for Python based on Werkzeug, Jinja 2, etc. It is fun, easy to setup and supports ORM, routing or rendering template as well. You can find it really easy to bootstrap a small app with built-in web server. Check out a very clear Quick start guide.

I followed another guide about how to create more large app wrote by Mitsuhiko. PyRSSReader was built in top of Flask, SQLAlchemy, Twitter Bootstrap and feedparser.


As you can see, this is project's requirements.txt file


Flask==0.10
Flask-OAuth==0.12
Flask-SQLAlchemy==0.16
Flask-WTF==0.8.2
Jinja2==2.6
SQLAlchemy==0.8.0
WTForms==1.0.3
Werkzeug==0.8.3
distribute==0.6.31
facepy==0.8.4
feedparser==5.1.3
httplib2==0.7.7
oauth2==1.5.211
pysqlite==2.6.3
requests==1.1.0
tweepy==2.0
wsgiref==0.1.2

Flask-OAuth, facepy or tweepy was my attempt to let users register with their twitter or facebook account, but I didn't have that much time for such an expensive feature :-) Jinja2 is the rendering template, and SQLAlchemy is for sqlite database. User can submit a new RSS link, view details, star some favourite entries as well as see other user's starred entries. Register one nick and check the "Trend" tab, you can see my favourited entries :-)



Clone and build

In this section I will show you some basic steps to build your own application. Ensure that following prerequisites are met:
  • Basic Python knowledge
  • Installed python version 2.6+ and pip in a unix-style environment
First we will install virtualenv on your environment. The virtualenv kit provides the ability to create virtual Python environments that do not interfere with either each other, or the main Python installation. Installing virtualenv before begin coding will lead to a completely clean Python environments for only this project.

sudo pip install virtualenv

Once installed, you can start to clone the repo as well as install all required packages inside your own virtual environment

git clone git@github.com:DTVD/PyRSSReader.git
cd PyRSSReader
virtualenv venv
source venv/bin/active
pip install -r requirements.txt

If everything went fine, you are now in a shell with (venv) at start and have all needed packages installed. Next step is about initialize the sqlite database

(venv)DTVD@DTVD-Air :: ~/PyRSSReader (master )-->> python shell.py
>>> db.create_all()

Above command will generate an "app.db" file in current directory, which is simply a sqlite database with table structure imported from model classes inside the "app" folder. You are always free to use sqlite3 to verify tables and it's content.

(venv)DTVD@DTVD-Air :: ~/PyRSSReader (master )-->> sqlite3 app.db
SQLite version 3.7.13 2012-07-17 17:46:21
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> .tables
facebook  feed      feeduser  star      twitter   user
sqlite>

That's it! The application is ready to start!

(venv)DTVD@DTVD-Air :: ~/PyRSSReader (master)-->> python run.py
 * Running on http://0.0.0.0:5000/
 * Restarting with reloader

Congratz! Open your browser at 127.0.0.1:5000 and test everything out.



Last words

It will takes more few steps to make all things work as a public service over the internet Take a look at Heroku-python-flask tutorial for a cloud-based solutions. 

Hope that you enjoyed the idea and feel free to fork or leave comments.