Phing: Sync a complete website to remote server quickly

Submitted by Peter Majmesku on Mon, 08/29/2016 - 21:04

The famous Phing build tool (written in PHP) offers automation for repetitive tasks, which need to be executed by the Linux terminal. So you can sync an entire website (files and database) to the remote server. Here is an example of a Phing target to do the job:

    <target name="syncWebsiteToLive"
            description="Sync folder and files via Rsync to live server.">
        <echo msg="Syncing data to live server." />
        <exec command="rsync -a --progress --delete /home/peter/Websites/example.com/ --exclude=.git/ --exclude=.idea/ --exclude=sites/default/settings.php user@example.com:/var/www/example.com/"
              passthru="true"
              checkreturn="true" />
        <echo msg="Delete old database" />
        <exec command="ssh example.com 'mysql -uroot -pPASSWORD -e &quot;drop database example_com;&quot;'"
              passthru="true"
              checkreturn="true" />
        <echo msg="Create new empty database" />
        <exec command="ssh example.com 'mysql -uroot -pPASSWORD -e &quot;create database example_com;&quot;'"
              passthru="true"
              checkreturn="true" />
        <echo msg="Import database dump" />
        <exec command="ssh example.com 'mysql -uroot -pPASSWORD example_com &lt; /var/www/example.com/db_dumps/latest.sql'"
              passthru="true"
              checkreturn="true" />
    </target>

This target is later contained in your build.xml file. Afterwards you can execute the following command:

phing syncWebsiteToLive

The output will be the following:

Buildfile: /home/peter/build.xml
DevOps > syncWebsiteToLive:
     [echo] Syncing data to live server.
sending incremental file list
db_dumps/latest.sql
      7,515,285 100%  111.94MB/s    0:00:00 (xfr#1, ir-chk=1009/1040)
     [echo] Delete old database
     [echo] Create new empty database
     [echo] Import database dump
BUILD FINISHED
Total time: 5.7706 seconds

The website has been single by one single console command in an absolute minimum of time. This approach is good for a website in development stadium or a website which database can be just replaced. Then this way can save a lot of time compared to other approaches like ftp and manual database dump import. 

Installing free of charge SSL certificate from Letsencrypt.org in Apache 2 webserver on Ubuntu 14.04

Submitted by Peter Majmesku on Mon, 08/29/2016 - 20:57

SSL certificates are cost-free nowadays. Letsencrypt.org is providing a service which lets you get SSL certificates that are free and work in the common web browsers. The following commands must be executed on the same machine on which your webserver is running. Because letsencrypt.org must be allowed to access it. Download the certbot from letsencrypt.org and make the binary executable:

wget https://dl.eff.org/certbot-auto
chmod a+x certbot-auto

Now temporary stop your Apache server, to download the certificates.

sudo service apache2 stop

Now run the certbot program to download the certificates:

./certbot-auto certonly

This program leads you trough a short form process within your terminal. Asking you for the domain you want to secure.

Mind the SSL certificate renewal

The SSL certificate is valid for 3 months. You will get notified about this via the terminal output from the certbot. It looks like this, if you have executed the "./certbot-auto certonly" command at 23.07.2016:

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at
   /etc/letsencrypt/live/my-domain.com/fullchain.pem. Your cert will
   expire on 2016-10-21. To obtain a new or tweaked version of this
   certificate in the future, simply run certbot-auto again. To
   non-interactively renew *all* of your certificates, run
   "certbot-auto renew"
 - If you like Certbot, please consider supporting our work by:
   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

There's an automatic way to accomplish the renewal. You can add this line to your crontab, to automatically re-new the certificates:

30 4 1 * * sudo service apache2 stop && ./certbot-auto renew --standalone && sudo service apache2 start

Enable the SSL certificate on your web server

Enable SSL in your Apache setup:

sudo a2enmod ssl

Now as you have the certificates, you can integrate them in your Apache 2 VHost configuration at /etc/apache2/sites-available/. Change "example.com" to your domain name.
 

 SSLEngine on
   SSLCertificateFile    /etc/letsencrypt/live/example.com/cert.pem
   SSLCertificateKeyFile /etc/letsencrypt/live/example.com/privkey.pem
   SSLCertificateChainFile /etc/letsencrypt/live/example.com/fullchain.pem 

Switch the virtual host port from 80 to 443. For this set the following in your Vhost configuration file: 

<VirtualHost *:80>

to

 <VirtualHost *:443>

Small tweak for SEO

Search engines expect that a web page can be accessed by only one address on your website. To redirect all requests from http to https and ensure that only http://your-domain.com will be requested and not http://your-domain.com "and" http://www.your-domain.com, you can setup the redirect within your VHost file. Add this to the top of the configuration file for your virtual host: 

# Redirect all requests to SSL and ensure www. subdomain is re-directed, too.

<VirtualHost *:80>

ServerName your-domain.com

Redirect / https://your-domain.com/

</VirtualHost>

<VirtualHost *:80>

ServerName www.your-domain.com

Redirect / https://your-domain.com/

</VirtualHost>

<VirtualHost *:443>

...

Now restart Apache and happy SSL usage: 

sudo service apache2 restart

A complete Apache2 Vhost config file example:

<VirtualHost *:80>
ServerName example.com
Redirect / https://example.com
</VirtualHost>

<VirtualHost *:80>
ServerName www.example.com
Redirect / https://example.com
</VirtualHost>


<VirtualHost *:443>
ServerName example.com

ServerAdmin webmaster@localhost
DocumentRoot /var/www/example.com/web

SSLEngine on
SSLCertificateFile /etc/ssl-cert/live/example.com/cert.pem
SSLCertificateKeyFile /etc/ssl-cert/live/example.com/privkey.pem
SSLCertificateChainFile /etc/ssl-cert/live/example.com/fullchain.pem

<Directory /var/www/example.com/web>
Options Indexes FollowSymLinks
AllowOverride All
Require all granted
</Directory>

ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined

</VirtualHost>
Tags

Drupal 8: Execute automated tests and debug them in PhpStorm

Submitted by Peter Majmesku on Mon, 08/29/2016 - 20:51

To test Drupal 8 tests by the command line, you need to do the following. You need to export two environment variables first:

export SIMPLETEST_DB=mysql://root:password@localhost/database;
export SIMPLETEST_BASE_URL=http://host;

Then you can execute the test by the following command, being in the Drupal root folder:

vendor/bin/phpunit -c core modules/my_module/tests/src/Kernel/MyTest.php

Make sure to set the environment variable for PhpStorm:

export XDEBUG_CONFIG="remote_enable=1 remote_mode=req remote_port=9000 remote_host=127.0.0.1 remote_connect_back=0"

phpunit.xml 

There's a phpunit.xml.dist file in PHP projects. That's a template for your PHPUnit configuration. You can specifiy there some PHP ini settings. By this method you do not need to export the variables like above. Copy it to phpunit.xml and set there your settings like follows:

 

<phpunit bootstrap="tests/bootstrap.php" colors="true"

beStrictAboutTestsThatDoNotTestAnything="true"

beStrictAboutOutputDuringTests="true"

beStrictAboutChangesToGlobalState="true"

checkForUnintentionallyCoveredCode="false"

printerClass="\Drupal\Tests\Listeners\HtmlOutputPrinter">

<php>

<!-- Set error reporting to E_ALL. -->

<ini name="error_reporting" value="32767"/>

<!-- Do not limit the amount of memory tests take to run. -->

<ini name="memory_limit" value="-1"/>

<!-- Example SIMPLETEST_BASE_URL value: http://localhost -->

<env name="SIMPLETEST_BASE_URL" value="http://pbt-d8"/>

<!-- Example SIMPLETEST_DB value: mysql://username:password@localhost/databasename#table_prefix -->

<env name="SIMPLETEST_DB" value="mysql://root:password@localhost/pbt-d8"/>

<!-- Example BROWSERTEST_OUTPUT_DIRECTORY value: /path/to/webroot/sites/simpletest/browser_output -->

<env name="BROWSERTEST_OUTPUT_DIRECTORY" value="/home/peter/d8-tests"/>

</php>

</phpunit>

Afterwards you are able to run your test without the manual environment variables setting, just by specifying your configuration directory by the "-c" parameter. Example:

vendor/bin/phpunit -c modules/permissions_by_term modules/permissions_by_term/tests/src/Kernel/SelectTermTest.php

This command cannot execute SimpleTest tests, which inherit from the WebTestBase class. Only unit tests and tests which inherits from the KernelTestBase class can be run.

Runs a SimpleTest

// --- SimpleTests
php core/scripts/run-tests.sh --url http://d8-paragraphs:8888 --browser --verbose --color --concurrency 4 --class 'Drupal\paragraphs\Tests\ParagraphsPreviewTest'

Troubleshooting

If the debug mode is switched on in PhpStorm and the debugger does not stop at the breakpoint and you are getting the following output in the debugger window:

Remote file path '/home/user/Websites/my-project/-' is not mapped to any file path in project

 If you have already setup path mapping, then there's probably any error in your code. Chances are, that it is in the first line of the setUp() method of your test class, which inherits from KernelTestBase. Then you probably do not have Drupal 8.2.x but Drupal 8.1.x. If it's in the first line, you can insert any code line or insert

xdebug_break(); 

With this function the debugger will be forced to stop on a specific line.


PhpStorm IDE settings (version 2016.1.2)

If you are executing the tests from the command line (and probably also via the IDE itself) it's important that you setup the path mappings, so that Xdebug can recognize your test's location. You can to this in PhpStorm's settings at "Languages & Frameworks > PHP > Servers". Create a host for your Drupal website host and set the path mapping to your tests directory. An example you can find in the screenshot below: 

PhpStorm Path Mapping

PHP XPath and DOMDocument class: Encode URLs to valid HTML standard from a piece of HTML markup

Submitted by Peter Majmesku on Mon, 08/29/2016 - 20:48

The following code example shows how it works:

<?php   $dom = new \DOMDocument('1.0''UTF-8');
  
// PHP will output warnings about non-standard HTML. Suppress it by "@".
  
@$dom->loadHTML($source);
  
// Iterate over all link-elements.
  
foreach ($dom->getElementsByTagName('link') as $node) {
    
// Copy the element to be able to replace it.
    
$updated_element $node;
    
$href_value $updated_element->getAttribute'href' );
    
// Checks if the value contains a standard violating character.
    
if (is_int(strpos($href_value']=within'))) {
      
// Encodes the URL to valid href value.
      
$href_value drupal_urlencode($href_value);
      
$updated_element->setAttribute('href'$href_value);
      
// Replace the wrong html markup.
      
$node->parentNode->replaceChild($updated_element$node);
    }
  }
  
// Get the HTML markup.
  
$html_markup_with_wrappers $dom->saveHtml();
  
// Remove the unnecessary wrappers.
  
$my_html_markup preg_replace('~<(?:!DOCTYPE|/?(?:html|head|body))[^>]*>\s*~i''',
    
$html_markup_with_wrappers); 

XDebug: Debug PHP scripts which are executed by Drush within your virtual machine

Submitted by Peter Majmesku on Mon, 08/29/2016 - 20:39

If you have PhpStorm (2016.1.1) installed on your host system and you want to debug PHP code in your virtual machine, which is executed by Drush (PHP command line application), so you need some configuration which is described below.

Settings in your PhpStorm IDE:

Map your project root folder to the absolute path on the server. Make sure that your webapplication's PHP code (e.g. Drupal) is located in the same folder as Drush. Like in the screenshot below.

PhpStorm Xdebug

Uncheck "Force break at the first line when a script is outside the project":

PhpStorm Xdebug

PhpStorm needs to listen to XDebug connections:

PhpStorm

Afterwards you are able to Debug the code within Drush itself. But it is not necessarily needed to have the Drush folder mapped, when you want to only debug your Drupal code.

Install Drush via composer (PHP package manager)

composer require "drush/drush:7.3"

I have choosen a specific Drush version to suit in my webapplication's requirements. Run this command in the folder, which contains also your webapplication, to meet the path mappings:

/project-folder (run the command here)
  /webapplication
  /vendors

 Drush is then located at "vendor/drush/drush". Make sure to run the Drush file which is located there by creating an alias in your .bash_profile file:

alias drush='php /var/www/vendor/drush/drush/drush.php'

Configure your PHP CLI (command line) settings, to work with XDebug

xdebug.remote_enable=true
xdebug.profiler_enable=0
xdebug.idekey=PHPSTORM
xdebug.max_nesting_level=256
xdebug.remote_autostart=true (THIS SETTING ENABLES/DISABLES XDEBUG)
xdebug.remote_connect_back=on
xdebug.remote_host=YOUR HOST IP. - E.G: 192.168.0.14 

Force XDebug to break, so that you can set the path mapping

Use the following function in your code, to make XDebug connecting back to your IDE. If that works, you need only to set the path mapping within your PhpStorm IDE to be able to debug properly.

<?php xdebug_break(); 

If everything works

Your IDE looks like below, if you have setup everything properly:

PhpStorm

Now you can use the following Drush command to debug your code.

drush php-eval 'my_nice_function()'

Drupal will be booted up and you can inspect your code during the application runtime. This approach is faster then locating a certain page on your Drupal site for finding the entry point to your code or using the Devel module with it's webpage based PHP execution functionality at http://my-website.dev/devel/php.

Tip: Check out the Phing build tool which is written in PHP

I recommend to create a command with your favorite build tool (I recommend https://www.phing.info/) to quickly disable and enable XDebug. So that you execute "xd" for disabling and "xe" for enabling to save time.

Tags

Screen: Run processes in the background on a Linux server

Submitted by Peter Majmesku on Mon, 08/29/2016 - 20:35

The program "Screen" can run processes in background on a Linux server.

This starts an new background session with name "your-screen-name"

screen -S your-screen-name

To save the screen and return to the terminal, type Ctrl+A+D (Ctrl+A is the hint for screen that you want to do something and "D" then "d"etaches from the session without stopping it).

Restore your screen :

screen -d -r your-screen-name

List all available screen sessions:

screen -ls

Type screen -list or -ls to identify the detached screen session.

$ screen -list 
    There are screens on: 
         20751.Melvin_Peter_V42  (Detached)  

Note: 20751.Melvin_Peter_V42 is your session id.

Quit a screen session: 

screen -X -S [session id you want to kill] quit

Or just type

exit

if you are attached in your screen session.