Sunday, February 22, 2015

Fixing puppet "Exiting; no certificate found and waitforcert is disabled" error

While debugging and setting up Puppet I am still running the agent and master from CLI in --no-daemonize mode.  I kept getting an error on my agent - ""Exiting; no certificate found and waitforcert is disabled".

The fix was quite simple and a little embarrassing.  Firstly I forgot to run my puppet master with root privileges which meant that it was unable to write incoming certificate requests to disk.  That's the embarrassing part and after I looked at my shell prompt and noticed this issue fixing it was quite simple.

Firstly I got the puppet ssl path by running the command puppet agent --configprint ssldir

Then I removed that directory so that my agent no longer had any certificates or requests.

On my master side I cleaned the old certificate by running puppet cert clean --all (this would remove all my agent certificates but for now I have just the one so its quicker than tagging it).

I started my agent up with the command puppet agent --test which regenerated the certificate and sent the request to my puppet master.  Because my puppet master was now running with root privileges (*cough*) it was able to write to its ssl directory and store the request.

I could then sign the request on my puppet master by running puppet cert sign --all

When running normally the puppet master will run as the puppet user so I'm not overly worried about running it as root in CLI while I debug it.

Thursday, February 19, 2015

Installing the Postgres extension for Hiphop

PostGres is rapidly gaining traction as a solid relational database manager.  It provides transaction reliability (ACID), standards compliance, and has a better reputation for handling large datasets than MariaDB / mySQL.

Luckily installing it for Hiphop is painless and there is an externally provided extension for it.

There was not a prebuilt package available for the PocketRent/hhvm-pgsql extension for my version of Hiphop so I built it following the advice in the project readme.  Their instructions worked first time.

I did not have success with using the "ini" method of including the extension so had to create a Hiphop configuration file.

 DynamicExtensionPath = /path/to/hhvm/extensions  
 DynamicExtensions {  
   * = pgsql.so  
 }  

I placed the snippet they provide (above) into a file called config.hdf in my Hiphop location.  I'm using Mint on my dev box so this was /etc/hhvm/config.hdf for me.

Then I edited /etc/default/hhvm and set Hiphop up to use the config file.  This snippet shows the change:

 ## Add additional arguments to the hhvm service start up that you can't put in CONFIG_FILE for some reason.  
 ## Default: ""  
 ## Examples:  
 ##  "-vLog.Level=Debug"        Enable debug log level  
 ##  "-vServer.DefaultDocument=app.php" Change the default document  
 ADDITIONAL_ARGS="-c /etc/hhvm/config.hdf"  

After that I restarted my hhvm service and was able to use Postgres in my Laravel project.  Hurray :)

New site in nginx is downloading PHP instead of executing it

I've just set up a new nginx host and was having problems with visiting it.

Whenever I loaded the page it would download a copy of my index file.

In my case the problem was two-fold.  Firstly I had mistyped my server name so it was falling back to an alternative catchall.

Secondly Chromium has a "won't fix" bug whereby redirects are cached.  Interestingly the cache also persisted into Incognito mode.

Clearing my cache with the "Clear Browsing History" menu function didn't work to clear out the redirect cache.  This gave the symptom of stopping nginx, stopping varnish, and even stopping hhvm but still having Chromium download the file when I tried to visit it.

What did work was to affix a get variable to the link.  So instead of visiting http://mysite.local I tried http://mysite.local?foo=1 which invalidated the cache.

This wasn't an entirely satisfactory solution so the next thing I tried was to use the developer tools (ctrl shift I) and then right clicking on the file in the network log.  Using the "Clear Browser Cache" option from that popup also worked.

So the TL;DR is:
1) Make sure that your config file is correct
2) Clear out your redirect cache or invalidate it

Wednesday, January 7, 2015

Compressing Apache output with mod_deflate on Centos

Apache on Centos ships with mod_deflate installed and enabled by default.  To check this you can grep your config file and make sure the line which loads it is not commented out.

 cat /etc/httpd/conf/httpd.conf | grep LoadModule deflate_module  

When Apache loads it reads all the config files (ending in .conf) in /etc/httpd/conf.d so we'll add configuration options for mod_deflate into this directory. Lets use a file called deflate.conf to specify the config:

 <IfModule mod_deflate.c>  
  AddOutputFilterByType DEFLATE text/plain  
  AddOutputFilterByType DEFLATE text/html  
  AddOutputFilterByType DEFLATE text/xml  
  AddOutputFilterByType DEFLATE text/css  
  AddOutputFilterByType DEFLATE text/javascript  
  AddOutputFilterByType DEFLATE image/svg+xml  
  AddOutputFilterByType DEFLATE image/x-icon  
  AddOutputFilterByType DEFLATE application/xml  
  AddOutputFilterByType DEFLATE application/xhtml+xml  
  AddOutputFilterByType DEFLATE application/rss+xml  
  AddOutputFilterByType DEFLATE application/javascript  
  AddOutputFilterByType DEFLATE application/x-javascript  
  DeflateCompressionLevel 9  
 # Browser specific settings  
  BrowserMatch ^Mozilla/4 gzip-only-text/html  
  BrowserMatch ^Mozilla/4\.0[678] no-gzip  
  BrowserMatch \bMSIE !no-gzip !gzip-only-text/html  
  BrowserMatch \bOpera !no-gzip   
 </IfModule>  

You can check it is working by noticing your YSlow report now shows, by using an online tool, or by just checking the headers with Chrome or Firefox's developer tools.

If you're using Varnish and Apache does not have mod_deflate then you can enable gzip in your vcl as per the Varnish manual.  The page linked at the bottom of the Varnish manual ( How GZIP, and GZIP+ESI works in Varnish ) explains how the response from the backend is stored in a compressed state.

  sub vcl_fetch {  
    if (beresp.http.content-type ~ "text") {  
        set beresp.do_gzip = true;  
    }  
 }  

Tuesday, January 6, 2015

Separating business logic from persistence layer in Laravel

There are several reasons to separate business logic from your persistence layer.  Perhaps the biggest advantage is that the parts of your application which are unique are not coupled to how data are persisted.  This makes the code easier to port and maintain.

I'm going to use Doctrine to replace the Eloquent ORM in Laravel.  A thorough comparison of the patterns is available here.

By using Doctrine I am also hoping to mitigate the risk of a major version upgrade on the underlying framework.  It can be expected for the ORM to change between major versions of a framework and upgrading to a new release can be quite costly.

Another advantage to this approach is to limit the access that objects have to the database.  Unless a developer is aware of the business rules in place on an Eloquent model there is a chance they will mistakenly ignore them by calling the ActiveRecord save method directly.

I'm not implementing the repository pattern in all its glory in this demo.  For a more purist approach to the pattern you can read this.  The reason that I'm choosing this approach is to cut down on the number of classes, cut down on the composer autoload, and to ensure my code is portable.  I also do not have to make any changes to my app aliases in config or create new Laravel services.

I'm going to use three objects to refer to my user table:
  • The UserEntity is a Doctrine entity to be accessed via the Doctrine entity manager by the Repository.  
  • The UserRepository is an intermediary layer akin to the data access object of other languages.  It uses the Entity to gather information that the Service layer needs.  
  • The UserService implements business logic and exposes methods to the controller.  It is the "fat model" in the "fat model / skinny controller" paradigm.
In order to be able to use Doctrine within Laravel I'm using the mitchellvanw/laravel-doctrine package.  I also use "raveren/kint" for access to the debugging "dd" shortcut.

The decision to use constructor injection when instantiating the user service is to make it easier to use a mock object when testing.  

I've deviated from the common practice of using private properties on a Doctrine entity and rather exposing getters and setters.  This is primarily so that in my user service layer I can conveniently reference properties of the entity in a manner that is not likely to change if I swap to another ORM.

I decided against marking the entity private and using reflection to retrieve the private properties in my repository.  I felt it was an unnecessary complication and not worth the processing cycles to ensure compatibility between methods of accessing a model property between ORMs.

The Doctrine entity class is incapable of persisting itself so if a developer instantiates it and modifies properties in the controller they won't be able to persist it unless they call the EntityManager class.  Hopefully this is more PT than calling save() on an ActiveRecord object.  Our design philosophy of avoiding doing this in controllers should also help to discourage mistakes here.

The files I created are listed below.  The drawback of avoiding any Laravel specific code is the rather ugly way of instantiating the service in the controller.  I believe, however, that my code will be easier to port to another framework than if I were to declare a Laravel service provider to make a static call to user.

You must include "app/models/user" into your composer autoload section and run the composer dump-autoload command from your shell once you've set up the directory.

app/models/user/UserEntity.php

 namespace User;  
 use Doctrine\ORM\Mapping AS ORM;  
 /**  
  * @ORM\Entity  
  * @ORM\Table(name="users")  
  */  
 class UserEntity  
 {  
      // see http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/basic-mapping.html for info on mapping  
      /**  
       * @ORM\Id  
       * @ORM\GeneratedValue  
       * @ORM\Column(type="integer")  
       */  
      private $id;  
      /**  
       * @ORM\Column(type="string")  
       */  
      private $name;  
      /**  
       * @ORM\Column(type="string")  
       */  
      private $password;  
      /**  
       * @ORM\Column(type="datetime")  
       */  
      private $created;  
      /**  
       * @ORM\Column(type="datetime")  
       */  
      private $modified;  
      public function getId()  
      {  
           return $this->id;  
      }  
      public function getName()  
      {  
           return $this->name;  
      }  
      public function setName($name)  
      {  
           $this->name = $name;  
      }  
      public function setPassword($password)  
      {  
           $this->password = $password;  
      }  
 }  

app/models/user/UserRepository.php

 <?php namespace User;  
 class UserRepository  
 {  
   protected $entity;  
   public function __construct( UserEntity $userEntity )  
   {  
     $this->entity = $userEntity;  
   }  
   public function getUserById($userId)  
   {  
     $user = \EntityManager::find('User\UserEntity', $userId);  
     return $user;  
   }  
   public function getUserByName($userName)  
   {  
     $user = \EntityManager::getRepository( 'User\UserEntity' )->findBy( [ 'name' => $userName ] );  
     if( !is_array( $user ) || empty( $user ) )  
     {  
       return false;  
     }  
     return $user[0];  
   }  
   public function setPassword( $userDetails, $password )  
   {  
     // If user variable is numeric, assume ID  
     if ( is_numeric( $userDetails ) )  
     {  
       // Get user based on ID  
       $user = $this->getuserById( $userDetails );  
     }  
     else  
     {  
       // Since not numeric, lets try get the user based on Name  
       $user = $this->getuserByName( $userDetails );  
     }  
     $user->setPassword( $password );  
     $this->persist( $user );  
     return true;  
   }  
   public function persist( UserEntity $user )  
   {  
     // do any last moment validations here  
     \EntityManager::persist( $user );  
     \EntityManager::flush();  
   }  
 }  

app/models/user/UserService.php

 <?php namespace User;  
 /**  
  * Our UserService, containing all useful methods for business logic around Users  
  * Do not reference the entity in here.  
  */  
 class UserService  
 {  
   // Containing our user repository to make all our database calls to  
   protected $userRepo;  
   /**  
    * Loads our $userRepo with the supplied userRepository  
    *  
    * We use constructor injection to make it easier to unit test.  
    *  
    * @param userInterface $userRepo  
    * @return userService  
    */  
   public function __construct( $userRepository )  
   {  
     $this->userRepo = $userRepository;  
   }  
   /**  
    * Method to get user based either on name or ID  
    *  
    * @param mixed $user  
    * @return string  
    */  
   public function getUserName($user)  
   {  
     // If user variable is numeric, assume ID  
     if (is_numeric($user))  
     {  
       // Get user based on ID  
       $user = $this->userRepo->getuserById($user);  
     }  
     else  
     {  
       // Since not numeric, lets try get the user based on Name  
       $user = $this->userRepo->getuserByName($user);  
     }  
     // If user entity returned (rather than null) return the name of the user  
     if ($user != null)  
     {  
       return $user->getName();  
     }  
     // If nothing found, return this simple string  
     return 'user Not Found';  
   }  
   public function setPassword( $user, $password )  
   {  
     // perform any validations  
     if( strlen( $password ) < 6 )  
     {  
       throw new \ValidationException( 'Password may not be shorter than 6 characters' );  
     }  
     // perform any hashing on the password  
     $password = \Hash::make($password);  
     return $this->userRepo->setPassword( $user, $password );  
   }  
 }  

app/controllers/HomeController.php

 <?php  
 class HomeController extends BaseController {  
      public function showWelcome()  
      {  
           return View::make('hello');  
      }  
      public function setPassword( $userDetails, $password )  
      {  
           $userService = new User\UserService( new User\UserRepository( new User\UserEntity ) );  
           try  
           {  
                $response = $userService->setPassword( $userDetails, $password );  
           }  
           catch ( ValidationException $e )  
           {  
                // set error message for frontend  
                echo 'An exception was thrown ('.$e->getMessage().') - this will result in a frontend message';  
           }  
           dd( $response );  
      }  
 }  

app/routes.php

 Route::put('/password/{user}/{password}', 'HomeController@setPassword');  

Monday, December 22, 2014

Using multiple accounts with Github

If you're like me and have a personal Github account but work for a company that also uses Github you will probably want to be able to set up multiple accounts on Github.

It's pretty simple to do so:

Firstly you need to create a new key for your company account.

Make sure that you save it to a file other than the default id_rsa otherwise you'll overwrite your default ssh key.

For illustration lets save it to ~/.ssh/id_rsa_alternate

 ssh-keygen -t rsa -C "your-email-address"  

Now open up your company account on Github and navigate through the settings to manage your ssh keys.  Use the following command:

 cat ~/.ssh/id_rsa_alternate.pub  

Copy and paste the output into a new key on your company Github account.

Next we add the new key to our identity:

 ssh-add ~/.ssh/id_rsa_alternate

Edit (or touch) your ssh config file at ~/.ssh/config and include a new option for authenticating using your company account:

  Host github-COMPANY   
  HostName github.com  
  User git  
  IdentityFile ~/.ssh/id_rsa_alternate  

Now you can add a remote like this one:

 git remote add origin git@github-COMPANY:Company/testing.git  

Note that instead of using git@github.com you're using the host name you set up in your ssh config.

Monday, December 15, 2014

Caching Laravel with Varnish


PHP Framework popularity as at 2013 - Sitepoint
After having a very good experience with using Varnish to cache a Wordpress site we decided to look at caching Laravel.

Laravel always generates cookies regardless of whether a person is logged in or not.  This interferes with Varnish which by default will pass all requests with a cookie to the backend and skip the cache.

In our particular case our site supported the ability for users to login and would then present them with custom content.  This means that cookies are not restricted to a particular path so we can't discard cookies based on the request as we did for Wordpress when discarding everything except /wp-admin/* requests.

My solution was to use a package called session-monster ( Packagist ) which sets a response header if the data in the Laravel session can be ignored.  Varnish can detect this header and prevent the cookie from being set since we don't really need it.  This together with the varnish config below handily caches pages for all the users who are not logged in.

Unfortunately we're doing this as an afterthought to add value to a client and caching was not part of our original project design.  This means that there is not development time available to make use of edge side includes which would allow caching the parts of a page that are static even for logged in users.  Early proof of concept tests show that implementing ESI is not particularly difficult.  Here's a useful looking blog post on how to implement it.  Luckily in our case we don't expect there to be many logged in users compared to non.

So assuming that you've gotten your nginx, hhvm, and varnish up and running here is an example configuration file:

 backend default {  
  .host = "127.0.0.1";  
  .port = "8080";  
 }  
 acl purge {  
  "127.0.0.1";  
  "localhost";  
 }  
 sub vcl_recv {  
   # handle purge requests  
   if (req.request == "PURGE") {  
     if (!client.ip ~ purge) {  
       error 405 "Not allowed.";  
     }  
     ban("req.url ~ "+req.url+" && req.http.host == "+req.http.host);  
     error 200 "OK";  
   }  
   if (req.url ~ "\.(png|gif|jpg|swf|css|js)$") {  
     return(lookup);  
   }
    # the cookie will persist until it expires (see your laravel session config)
    if (req.http.Cookie ~ "laravel_session") {
     return(pass);
    }   
    # else ok to fetch a cached page  
   return (lookup);  
 }  
 sub vcl_fetch {  
   # strip the cookie before the static file is inserted into cache.  
   if (req.url ~ "\.(png|gif|jpg|swf|css|js)$") {  
     unset beresp.http.set-cookie;  
   }  
   # remove some headers we never want to see  
   unset beresp.http.Server;  
   unset beresp.http.X-Powered-By;  
   unset beresp.http.X-Pingback;  
   set beresp.do_esi = true; /* Do ESI processing */  
   set beresp.ttl = 10m;  
   # don't cache response to posted requests or those with basic auth  
   if ( req.request == "POST" || req.http.Authorization ) {  
      return (hit_for_pass);  
   }  
   # Laravel always adds a session cookie - we remove it with session monster and check it here  
   # Do this before checking the page state but after post  
   if (beresp.http.X-No-Session ~ "yeah") {  
     unset beresp.http.set-cookie;  
   }  
   else
   {
       # do not cache responses which are for logged in users
       return (hit_for_pass);
   }
   # only cache status ok  
   if ( beresp.status != 200 ) {  
     return (hit_for_pass);  
   }  
   # else ok to cache the response  
   return (deliver);  
 }  
 sub vcl_deliver {  
   if (obj.hits > 0) {  
     set resp.http.X-Cache = "HIT";  
   }  
   else {  
     set resp.http.X-Cache = "MISS";  
   }  
   unset resp.http.Via;  
   unset resp.http.X-Varnish;  
 }  
 sub vcl_hit {  
  if (req.request == "PURGE") {  
   purge;  
   error 200 "OK";  
  }  
 }  
 sub vcl_miss {  
  if (req.request == "PURGE") {  
   purge;  
   error 404 "Not cached";  
  }  
 }  

Installing the Laravel side of things is simple:
  1. Add a require for session monster to your composer file ( "haifanghui/session-monster": "dev-master" )
  2. Edit your application config and include the provider as in the snippet below
  3. Edit app/config/session.php and set the session lifetime to a number you feel comfortable with
   'providers' => array(  
     'HaiFangHui\SessionMonster\SessionMonsterServiceProvider'  
   ),  
You need to set the session timeout so that the cookie expires sometime after the user logs out.  Even though Laravel will stop emitting the cookie when the user logs out the browser will keep sending it and breaking the cache.