10 steps to install WordPress with LAMP stack on Ubuntu 16.04.

Hi everyone,

Lets try to host Wordpress on our Ubuntu 16.04 using LAMP stack. For these steps, I have referred the following website. Since I have found few issues while trying to establish my wordpress server so tried to recreate one single and ready-to-follow post for your reference based upon my experience. This is tried and tested process which you can follow directly. Here I am using latest Ubuntu 16.04 with all packages updated as on dated 9/05/18.

Source: https://www.digitalocean.com/community/tutorials/how-to-install-wordpress-with-lamp-on-ubuntu-16-04

Step 1: Install Apache and Allow in Firewall

To remove apache2

server@ubuntu:/home/abhi$ sudo apt-get purge apache2 

server@ubuntu:/home/abhi$ sudo apt-get update

To reinstall

server@ubuntu:/home/abhi$ sudo apt-get install apache2

Set Global ServerName to Suppress Syntax Warnings

Add a single line to the /etc/apache2/apache2.conf file at the end to suppress a warning message: “ServerName 192.168.43.43”
While harmless, if you do not set ServerName globally, you will receive the following warning when checking your Apache configuration for syntax errors:

server@ubuntu:/home/abhi$ sudo apache2ctl configtest

Output 

AH00558: apache2: Could not reliably determine the server's fully
qualified domain name, using 127.0.1.1. Set the 'ServerName'  directive globally to suppress this message Syntax OK At the bottom of the file, add a ServerName directive, pointing  to your primary domain name or IP address.

Next, check for syntax errors by typing:

server@ubuntu:/home/abhi$ sudo apache2ctl configtest 

[sudo] password for server: 

 Syntax OK

Since we added the global ServerName directive, all you should see is:

Syntax OK

Restart Apache to implement your changes:

server@ubuntu:/home/abhi$ sudo service apache2 restart

Adjust the Firewall to Allow Web Traffic

Next, assuming that you have followed the initial server setup instructions to enable the UFW firewall, make sure that your firewall allows HTTP and HTTPS traffic

server@ubuntu:/home/abhi$ sudo ufw app list

Available applications:
Apache
Apache Full
Apache Secure
CUPS

If you look at the Apache Full profile, it should show that it enables traffic to ports 80 and 443:

server@ubuntu:/home/abhi$ sudo ufw app info "Apache Full" 

Profile: Apache Full 

Title: Web Server (HTTP,HTTPS) 

Description: Apache v2 is the next generation of the omnipresent Apache web server. 

Ports: 80,443/tcp

Allow incoming traffic for this profile:

server@ubuntu:/home/abhi$ sudo ufw allow in "Apache Full"

You can do a spot check right away to verify that everything went as planned by visiting your server's public IP address in your web browser 

  http://192.168.43.43 (use your own IP address here)

You will see the default Ubuntu 16.04 Apache web page, which is there for informational and testing purposes. It should look something like this:

                                      

If you see this page, then your web server is now correctly installed and accessible through your firewall.

Step 2: Install MySQL

Now that we have our web server up and running, it is time to install MySQL. MySQL is a database management system. Basically, it will organize and provide access to databases where our site can store information.

server@ubuntu:/home/abhi$ sudo apt-get install mysql-server

During the installation, your server will ask you to select and confirm a password for the MySQL "root" user. This is an administrative account in MySQL that has increased privileges. Think of it as being similar to the root account for the server itself (the one you are configuring now is a MySQL-specific account, however). Make sure this is a strong, unique password, and do not leave it blank.

When the installation is complete, we want to run a simple security script that will remove some dangerous defaults and lock down access to our database system a little bit. Start the interactive script by running:

server@ubuntu:/home/abhi$ mysql_secure_installation

You will be asked to enter the password you set for the MySQL root account. Next, you will be asked if you want to configure the VALIDATE PASSWORD PLUGIN.
Answer y for yes, or anything else to continue without enabling.

If you enabled password validation, you'll be shown a password strength for the existing root password, and asked you if you want to change that password. If you are happy with your current password, enter n for "no" at the prompt:

For the rest of the questions, you should press Y and hit the Enter key at each prompt. This will remove some anonymous users and the test database, disable remote root logins, and load these new rules so that MySQL immediately respects the changes we have made.
At this point, your database system is now set up and we can move on.

Step 3: Install PHP

PHP is the component of our setup that will process code to display dynamic content. It can run scripts, connect to our MySQL databases to get information, and hand the processed content over to our web server to display.

We're going to include some helper packages as well, so that PHP code can run under the Apache server and talk to our MySQL database:

server@ubuntu:/home/abhi$ sudo apt-get install php libapache2-mod-php php-mcrypt php-mysql

In most cases, we'll want to modify the way that Apache serves files when a directory is requested. Currently, if a user requests a directory from the server, Apache will first look for a file called index.html. We want to tell our web server to prefer PHP files, so we'll make Apache look for an index.ph file first.

                                

To do this, type this command to open the dir.conf file in a text editor with root privileges:

server@ubuntu:/home/abhi$ sudo nano /etc/apache2/mods-enabled/dir.conf 

<IfModule mod_dir.c> 

 DirectoryIndex index.html index.cgi index.pl index.php index.xhtml index.htm

</IfModule>

# vim: syntax=apache ts=4 sw=4 sts=4 sr noet

We want to move the PHP index file highlighted above to the first position after the Directory Index specification, like this:

<IfModule mod_dir.c>
DirectoryIndex index.php index.html index.cgi index.pl index.xhtml inde$
</IfModule>

# vim: syntax=apache ts=4 sw=4 sts=4 sr noet

When you are finished, save and close the file

After this, we need to restart the Apache web server in order for our changes to be recognized. You can do this by typing this:

server@ubuntu:/home/abhi$ sudo service apache2 restart
server@ubuntu:/home/abhi$ sudo service apache2 status 

● apache2.service - LSB: Apache2 web server
   Loaded: loaded (/etc/init.d/apache2; bad; vendor preset: enabled)
  Drop-In: /lib/systemd/system/apache2.service.d
           └─apache2-systemd.conf
   Active: active (running) since Thu 2018-05-10 13:54:48 IST; 1h 27min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 1128 ExecStart=/etc/init.d/apache2 start (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/apache2.service
           ├─1220 /usr/sbin/apache2 -k start
           ├─1224 /usr/sbin/apache2 -k start
           ├─1225 /usr/sbin/apache2 -k start
           ├─1226 /usr/sbin/apache2 -k start
           ├─1227 /usr/sbin/apache2 -k start
           └─1228 /usr/sbin/apache2 -k start

May 10 13:54:46 ubuntu systemd[1]: Starting LSB: Apache2 web server...
May 10 13:54:46 ubuntu apache2[1128]:  * Starting Apache httpd web server apache2
May 10 13:54:48 ubuntu apache2[1128]:  *
May 10 13:54:48 ubuntu systemd[1]: Started LSB: Apache2 web server.

If we decided that php-cli is something that we need, we could type:

server@ubuntu:/home/abhi$ sudo apt-get install php-cli

Step 4: Test PHP Processing on your Web Server

In order to test that our system is configured properly for PHP, we can create a very basic PHP script.

We will call this script info.php. In order for Apache to find the file and serve it correctly, it must be saved to a very specific directory, which is called the "web root".
In Ubuntu 16.04, this directory is located at /var/www/html/. We can create the file at that location by typing:

server@ubuntu:/home/abhi$ sudo nano /var/www/html/info.php

This will open a blank file. We want to put the following text, which is valid PHP code, inside the file:

<?php 

phpinfo(); 

?>

When you are finished, save and close the file.

Now we can test whether our web server can correctly display content generated by a PHP script. To try this out, we just have to visit this page in our web browser.

The address you want to visit will be:

http://192.168.43.43/info.php   (Use your own IP address)

                           


This page basically gives you information about your server from the perspective of PHP. It is useful for debugging and to ensure that your settings are being applied correctly.
If this was successful, then your PHP is working as expected.

Step 5: Create a MySQL Database and User for WordPress

WordPress uses MySQL to manage and store site and user information. We have MySQL installed already, but we need to make a database and a user for WordPress to use.
To get started, log into the MySQL root (administrative) account by issuing this command:

server@ubuntu:/home/abhi$ mysql -u root -p

You will be prompted for the password you set for the MySQL root account when you installed the software.

First, we can create a separate database that WordPress can control. You can call this whatever you would like, but we will be using “wordpressdatabase” in this guide to keep it simple. You can create the database for WordPress by typing:

mysql > CREATE DATABASE wordpressdatabase DEFAULT CHARACTER SET utf8 COLLATE utf8_unicode_ci;

Next, we are going to create a separate MySQL user account that we will use exclusively to operate on our new database. Creating one-function databases and accounts is a good idea from a management and security standpoint. We will use the name wordpressuser in this guide. Feel free to change this if you'd like.

We are going to create this account, set a password, and grant access to the database we created. We can do this by typing the following command. Remember to choose a strong password here for your database user:

mysql > GRANT ALL ON wordpressdatabase.* TO 'wordpressuser'@'localhost' IDENTIFIED BY 'password'; (Use strong password here)

You now have a database and user account, each made specifically for WordPress. We need to flush the privileges so that the current instance of MySQL knows about the recent changes we've made:

mysql > FLUSH PRIVILEGES;

Exit out of MySQL by typing:

mysql > EXIT;


Step 6: Install Additional PHP Extensions

When setting up our LAMP stack, we only required a very minimal set of extensions in order to get PHP to communicate with MySQL. WordPress and many of its plugins leverage additional PHP extensions.

We can download and install some of the most popular PHP extensions for use with WordPress by typing:

server@ubuntu:/home/abhi$ sudo apt-get update 

server@ubuntu:/home/abhi$ sudo apt-get install php-curl php-gd php-mbstring php-mcrypt php-xml php-xmlrpc

We will restart Apache to leverage these new extensions in the next section. If you are returning here to install additional plugins, you can restart Apache now by typing:

server@ubuntu:/home/abhi$ sudo service apache2 restart

Step 7: Adjust Apache's Configuration to Allow for .htaccess Overrides and Rewrites

Next, we will be making a few minor adjustments to our Apache configuration. Currently, the use of .htaccess files is disabled. WordPress and many WordPress plugins use these files extensively for in-directory tweaks to the web server's behavior.

Additionally, we will enable mod_rewrite, which will be needed in order to get WordPress permalinks to function correctly. 

Enable .htaccess Overrides

Open the primary Apache configuration file to make our first change:

server@ubuntu:/home/abhi$ sudo nano /etc/apache2/apache2.conf

To allow .htaccess files, we need to set the AllowOverride directive within a Directory block pointing to our document root. Find out the

<Directory /var/www/html> and make following changes “AllowOverride All”

. . .

<Directory /var/www/html/> 

                 AllowOverride All 

</Directory> 

. . .

When you are finished, save and close the file.

Enable the Rewrite Module

Next, we can enable mod_rewrite so that we can utilize the WordPress permalink feature:

server@ubuntu:/home/abhi$ sudo a2enmod rewrite

Enable the Changes

Before we implement the changes we've made, check to make sure we haven't made any syntax errors:

server@ubuntu:/home/abhi$ sudo apache2ctl configtest

Restart Apache to implement the changes:

server@ubuntu:/home/abhi$ sudo service apache2 restart


Step 8: Download WordPress

Now that our server software is configured, we can download and set up WordPress. For security reasons in particular, it is always recommended to get the latest version of WordPress from their site.

Change into a writable directory and then download the compressed release by typing:

server@ubuntu:/home/abhi$ cd /tmp 

server@ubuntu:/home/abhi$ wget https://wordpress.org/latest.tar.gz

Extract the compressed file to create the WordPress directory structure:

server@ubuntu:/home/abhi$ tar -xzvf latest.tar.gz

We will be moving these files into our document root momentarily. Before we do, we can add a dummy .htaccess file and set its permissions so that this will be available for WordPress to use later.

Create the file and set the permissions by typing:

server@ubuntu:/home/abhi$ touch /tmp/wordpress/.htaccess server@ubuntu:/home/abhi$ chmod 660 /tmp/wordpress/.htaccess


We'll also copy over the sample configuration file to the filename that WordPress actually reads:

server@ubuntu:/home/abhi$ cp /tmp/wordpress/wp-config-sample.php /tmp/wordpress/wp-config.php


We can also create the upgrade directory, so that WordPress won't run into permissions issues when trying to do this on its own following an update to its software:

server@ubuntu:/home/abhi$ mkdir /tmp/wordpress/wp-content/upgrade

Now, we can copy the entire contents of the directory into our
document root. We are using the -a flag to make sure our
permissions are maintained. 

server@ubuntu:/home/abhi$ sudo cp -a /tmp/wordpress/. /var/www/html

Step 9: Configure the WordPress Directory

Before we do the web-based WordPress setup, we need to adjust some items in our WordPress directory.

Adjusting the Ownership and Permissions

One of the big things we need to accomplish is setting up reasonable file permissions and ownership. We need to be able to write to these files as a regular user, and we need the web server to also be able to access and adjust certain files and directories in order to function correctly.

We'll start by assigning ownership over all of the files in our document root to our username. We will use server as our username in this guide, but you should change this to match whatever your sudo user is called. We will assign group ownership to the www-data group:

server@ubuntu:/home/abhi$ sudo chown -R server:www-data /var/www/html

Next, we will set the setgid bit on each of the directories within the document root. This causes new files created within these directories to inherit the group of the parent directory (which we just set to www-data) instead of the creating user's primary group. This just makes sure that whenever we create a file in the directory on the command line, the web server will still have group ownership over it.

We can set the setgid bit on every directory in our WordPress installation by typing:

server@ubuntu:/home/abhi$ sudo find /var/www/html -type d -exec chmod g+s {} \;


There are a few other fine-grained permissions we'll adjust. First, we'll give group write access to the wp-content directory so that the web interface can make theme and plugin changes:

server@ubuntu:/home/abhi$ sudo chmod g+w /var/www/html/wp-content

As part of this process, we will give the web server write access to all of the content in these two directories:

server@ubuntu:/home/abhi$ sudo chmod -R g+w /var/www/html/wp-content/themes 

server@ubuntu:/home/abhi$ sudo chmod -R g+w /var/www/html/wp-content/plugins

This should be a reasonable permissions set to start with.

Setting up the WordPress Configuration File

Now, we need to make some changes to the main WordPress configuration file.

When we open the file, our first order of business will be to adjust some secret keys to provide some security for our installation. WordPress provides a secure generator for these values so that you do not have to try to come up with good values on your own. These are only used internally, so it won't hurt usability to have complex, secure values here.


To grab secure values from the WordPress secret key generator, type:

server@ubuntu:/home/abhi$ wget https://api.wordpress.org/secret-key/1.1/salt/

You will get back unique values that look something like this:

define('AUTH_KEY',  'nE.--7T`#b6:||P!=x@.rD4.dBx+95OlPH=Co;PAovNp2rK$-0H[.G|y0KlewD/Y');

define('SECURE_AUTH_KEY',  '+)[!da$)vYG.oC=m}q{A{z0~)_ -m]NyMT|!goYCoXDwA-Jvzk)Su@Cx-+i~X+.J');

define('LOGGED_IN_KEY',  'z//ZWYa@<9SYmMGvUPJ#-AACz2aD?ku2,F-jK<W+}|4BOT!Msg7(mN:%LQ9YUa-N');

define('NONCE_KEY', '>|r_Aiu5$W+5y#MD]EUc+]3$fbxw9^MaWSn7*#kxf6EJjIm#1#|A(D*$YAB+f4c');

define('AUTH_SALT', 'wn@FsGFY|:/8qFGkE,WOgaZ1|,.~4~-i@cW7zw!XKSXIy>.ib).VQ~W= C~PYu`2');

define('SECURE_AUTH_SALT',  '@xqXjA{+rpxUhD9}d RYe;4-L${Jaq-fFPgaKgx!Y-?_RB=BtSb!@7_Q*d7X=K|c');

define('LOGGED_IN_SALT', 'chYB7NysY]1:XDQ{c-/k:h0!<CzW#D`K=}!.0[U+bu Bv[{w|dT3_P]n,42z3RYg');
define('NONCE_SALT', '|X=)Rm:6&1Yq`sr7XeVOW)p#i8?]y--;@)W-+_qH:a2)[J*f7)U+sfMD++JyPi*V');

NOTE: It is important that you request unique values each time.
Do NOT copy the values shown below!

These are configuration lines that we can paste directly in our configuration file to set secure keys. Copy the output you received now.
Now, open the WordPress configuration file:

server@ubuntu:/home/abhi$ nano /var/www/html/wp-config.php

Find the section that contains the dummy values for those settings. Delete those lines and paste in the values you copied just now.

Next, we need to modify some of the database connection settings at the beginning of the file. You need to adjust the database name, the database user, and the associated password that we configured within MySQL.

The other change we need to make is to set the method that WordPress should use to write to the filesystem. Since we've given the web server permission to write where it needs to, we can explicitly set the filesystem method to "direct". Failure to set this with our current settings would result in WordPress prompting for FTP credentials when we perform some actions.

server@ubuntu:/home/abhi$ /var/www/html/wp-config.php

. . . 

define('DB_NAME', 'wordpressdatabase');           (Use your database name) 

/** MySQL database username */ 

define('DB_USER', 'wordpressuser');                   (Use your wordpress username) 

/** MySQL database password */ 

define('DB_PASSWORD', 'password');                (Use your wordpress password) 

. . . 

define('FS_METHOD', 'direct'); 

( Filesystem method as DIRECT for wordpress webserver to write  directly wherever it wants)

Save and close the file when you are finished.

Step 10: Complete the Installation Through the Web Interface

Now that the server configuration is complete, we can complete the installation through the web interface.

In your web browser, navigate to your server's domain name or public IP address:

http://192.168.43.43 (USE your own IP address here)

Select the language you would like to use:

                                  

Next, you will come to the main setup page.

Select a name for your WordPress site and choose a username (it is recommended not to choose something like "admin" for security purposes). Choose a strong password.

When you click ahead, you will be taken to a page that prompts you to log in:

Once you log in, you will be taken to the WordPress administration dashboard:

                            

Hope you have leaned lot following given steps and now hosting your wordpress based website on local serve run on Ubuntu 16.04

If you ever want to make your website global, you can do it using free "ngrok" service in Ubuntu 16.04.

Enjoy! Keep Learning :-)

Using AWS cloud: Launch a Linux Virtual Machine with Amazon EC2 and Execute simple shell script

Hi guys
I am going to use the following reference link for this article:

Reference: https://aws.amazon.com/getting-started/tutorials/launch-a-virtual-machine/

Amazon Elastic Compute Cloud (EC2) is the Amazon Web Service you use to create and run virtual machines in the cloud. AWS calls these virtual machines 'instances'. Note, I am going to use Free tier account from AWS for this article.

Step 1: Login to your Amazon aws account. From “All services”, select EC2 to launch EC2 Dashboard. Click Launch Instance to create and configure your virtual machine.

Step 2: Configure your Instance

choose an Amazon Machine Image (AMI). AMIs are preconfigured server templates you can use to launch an instance. Each AMI includes an operating system, and can also include applications and application servers.

You will now choose an instance type. Instance types comprise of varying combinations of CPU, memory, storage, and networking capacity so you can choose the appropriate mix for your applications. The default option of t2.micro should already be checked.  This instance type is covered within the Free Tier and offers enough compute capacity to tackle simple workloads. Click Review and Launch at the bottom of the page.

Click Launch

• On the next screen you will be asked to choose an existing key pair or create a new key pair. A key pair is used to securely access your Linux instance using SSH. AWS stores the public part of the key pair which is just like a house lock. You download and use the private part of the key pair which is just like a house key.
• Select Create a new key pair and give it the name MyKeyPair. Next click the Download Key Pair button.
• After you download the MyKeyPair key, you will want to store your key in a secure location. If you lose your key, you won't be able to access your instance. If someone else gets access to your key, they will be able to access your instance.

For Windows:

• We recommend saving your key pair in your user directory in a sub-directory called .ssh (ex. C:\user\{yourusername}\.ssh\MyKeyPair.pem).

NOTE: Create your ssh directory name like (.ssh.), the final period is removed automatically.

After you have stored your key pair, click Launch Instance to start your instance.

Click View Instances on the next screen to view your instances and see the status of the instance you have just started.

• In a few minutes, the Instance State column on your instance will change to "running" and a IPv4 Public IP address will be shown. Copy the IPv4 Public IP address of your AWS instance, so you can use it when we connect to the instance using SSH in Step 3.

Step 3: Connect to your Instance

• After launching your instance, it's time to connect to it using SSH.
• Windows users:  Select Windows below to see instructions for installing Git Bash which includes SSH.
• Download Git for windows from https://git-scm.com/download/win and install Git.
• Right click on your desktop (not on an icon or file) and select Git Bash Here to open a Git Bash command prompt.

• Use SSH to connect to your instance. In this case the user name is ec2-user, the SSH key is stored in the .ssh directory with name MyKeyPair, and the IP address are IPv4 Public IP address that we saved from previous step. Put all these details in given format

                  ssh -i {full path of your .pem file} ec2-user@{instance IP address}

• We can create programs like shell script or python program using nano editor in VM. Here I am creating a simple shell script 

• You can notice that when I execute my simple shell script, it responds me if I am running a Linux machine but actually right now I am using a Windows machine. That’s the advantage of using VM over AWS.

Step 4: Terminate Your Instance

You can easily terminate the instance from the EC2 console. In fact, it is a best practice to terminate instances you are no longer using so you don’t keep getting charged for them.

I hope this article would help some one who is interested to make quick start on AWS services such as EC2 using Free tier account.

Keep learning and do comment your feedback!

Export data from SQL database into Microsoft Excel 2016

Hi all

This post is in continuation of previous post where sensor data is stored in SQL database over Microsoft Azure. Sometimes, we want to process our data offline for which we may need to fetch the data from SQL server and store in Excel or CSV format. So, in that situation, we need to export data stored in SQL database into Microsoft Excel 2016 format for offline processing. 

Here are the steps to do it:

·       Open New Excel Sheet

·       Click on Data Header, Get Data—From Database—From SQL server database

·       It will ask from SQL server address. This address is available in Microsoft Azure portal, in sql database—overview—server name. For my project, address is: azuredisonsqlserver.database.windows.net

·       For the first time, it may give error for connection. You have to allow it in Azure firewall.

·       In Sql database—overview option, Click on Set Server Firewall option—Click Add Client IP. Make sure that Microsoft Azure and Microsoft Excel is on same network.

·       Now try to open SQL database in Microsoft Excel.

·       It would open Navigator page, which would have Display Options like this:

·       Select AzureEdison Table and click Load option available at bottom. This step would import your data from SQL server available in Microsoft Azure into Microsoft Excel.

·       Now you can do any processing in your data offline also. 

     

      Happy learning and please post your comments below:-)

Reading Sensor data and uploading it on Microsoft Azure

Reading Sensor data via Intel Edison and storing data in SQL database over Microsoft Azure 

• In this post, I have explained the entire process in bullet points. In this project, using Ambient light sensor, the sensor value is displayed on terminal console, which is actually sent at Microsoft Azure. 
• This value can be verified using Device Explorer tool, Later Using Stream Analytics Job, the data is streamed and saved in SQL server, which is created online, in Microsoft Azure in form of SQL table. At this stage, this data can be used for analysis purpose and data visualization using suitable tools such as Power BI.

-- Open portal.azure.com
-- Click New—Internet of Things—IoT Hub
-- Give unique name to your IoT hub –Select pricing to Free –resource group (create new and give any name) and create it.
-- Now go to Dashboard and click on you newly create IoT Hub.
-- Go to Settings—Shared Access Policies—select policy “IoT Hub Owner”—Go to Shared access key—Select “Connection String—primary key” and copy the key. And paste it in notepad. This is Key 1.
-- In Iot hub, search for IoT Devices, click on “Add”, give device id, authentication type—symmetric key, select auto generate key, Enable with connect device to IoT Hub and Save. 
-- After Save, reopen the device and copy Connection String--Primary key in notepad. This is Key 2.

-- Go to this page to download “Device Explorer” https://github.com/Azure/azure-iot-sdk-csharp/releases and then install it.
-- Paste the copied connection string (Key 1) in box under configuration tab –IoT Hub Connection String and Update.
-- Click on Management tab and click create tab—give name to Device ID.
-- Once it is created, a device id with given name is create with multiple tabs.
-- Click under Connection string Tab, right click on content—copy connection string for selected device and paste it again in notepad. This is the same key 2 you got earlier also.

Paste this key in program against connectionString variable.
-- This key is very essential for sending data to Microsoft Azure IoT Hub.

Actual program which would run in Edison, written in JavaScript (This program will fetch light intensity and post that data via MQTT protocol usingg Microsoft Azure IoT Hub)

var mraa = require("mraa") ;

var sense= new mraa.Aio(0);

var Protocol = require('azure-iot-device-mqtt').Mqtt;

var Client = require('azure-iot-device').Client;

var Message = require('azure-iot-device').Message;

//

//Azure methods

//

var connectionString=”HostName=azuredison.azure-devices.net;DeviceId=azuredison_iot;SharedAccessKey=xxxxxxxxxxxxxxxxxxxxxx”;

var client = Client.fromConnectionString(connectionString, Protocol);

var connectCallback = function (err) {

  if (err) {

    console.error('Could not connect: ' + err.message);

  } else {

    console.log('Client connected');

    client.on('message', function (msg) {

      console.log('Id: ' + msg.messageId + ' Body: ' + msg.data);

      client.complete(msg, printResultFor('completed'));

    });

  }

};

client.open(connectCallback);

function lightRead()

{

    var lightVal= sense.read();

    //console.log(lightVal);

    var obj= {"lightVal":lightVal};

    var json= JSON.stringify(obj);

    var message = new Message(json);

    client.sendEvent(message, printResultFor('send'));

    console.log(json);

    setTimeout(lightRead,10000);

}

lightRead();

function printResultFor(op) {

  return function printResult(err, res) {

    if (err) console.log(op + ' error: ' + err.toString());

    if (res) console.log(op + ' status: ' + res.constructor.name);

  };

}

·       Access Intel Edison using Putty/TeraTerm and install following packages in Intel Edison (Yocto Linux)
#                              npm install azure-iot-device-mqtt
                                npm install azure-iot-device
  -- Paste connection String copied in notebook in connection string variable. 
  -- Run this code and it will through light intensity values on console
  -- Go to device Explorer tool—Data Tab—Select Device ID as created—Click Monitor Tab.
  -- You should be able to see the light intensity data in this tool. This means that data is transferred to Microsoft Azure But we are able to check its value in this tool.
  -- Click on New—SQL Database—Create
  -- Give Name to SQL database—Subscription—Free Trial—Resource Group—Use existing—Blank Database—Create new sql server—give server name, usename and password for sql server.
  -- After sql database is created, Goto Data explorer (preview) option, login to sql server, and run Query. This query would create sql table where variable LightVal data would be stored:
                    create table AzureIoTEdison(LightVal varchar(255));   
   
  -- Run another query. This query would fetch all data stored in sql table AzureIoTEdison. Since at this stage no value is available (program is not running in edison), it would return 0:
                     Select * from AzureIoTEdison
  
  -- Click New—Search Stream Analytics Job—Create. Click on overview window and look in Job Topology.
  -- There three blocks are available: Inputs, Query, Output.
  -- Click on Input Block—Add—Input alias—“Edison” or any name of input device. Source—IoT Hub.
  -- IoT hub—selected automatically, Event serialization format—JSON, Encoding—UTF-8
  -- Click Create
  -- Click on Output Block—Add—Give output alias name—Edisonoutputdata. Sink—SQL database, Database—selected automatically, give username & password for SQL database, Table—AzureIoTEdison (name of the table created)--Click Create
  
  -- Click on Query Block. Paste this code :

          SELECT

    lightVal
INTO

    [edisonoutputdata]
FROM

    [edison]

Save

-- Start Stream Job
-- Run the code in Edison using Putty #node main.js
-- Open Device Explorer tool

-- Open new tab and open portal.azure.com here again

-- Open dashboard, sql database, Data Explorer(preview)—login to sql server and type query and run:
                    Select * from AzureEdison

-- Now in the result window of sql sever, you can see the data coming up from sensor.  
-- Same data is also visible in Data Explorer tool. This data also get stored in SQL Database. Make sure your program is already running in Edison.
-- In Sql database, in DatExplore(preview) option, login to sql server. You will see multiple options as: Tables, Views, Stored Procedures.
-- Click on Tables—dbo.AzureIoTEdison—LightVal(varchar,null). This is the table in which sensor data is stored in SQL server. TO view this data, Select dbo.AzureIoTEdison—Click Edit Data(Preview). You can map these values with the values generated by sensors in Putty Console or Data Explorer. This verifies that Data actually get stored in SQL server in Microsoft Azure cloud.
-- Once you are done, you can Stop Stream Analytics Job.

Hope you have got a fair insight of how to save sensor data in SQL database over Microsoft Azure. Steps mentioned in this post are well verified and tested,still, if you face any issue, feel free to post your query. I would be happy to help you.

I will come up with few more interesting posts on Microsoft Azure in coming future.

Stay tuned and post your comments!

Python Programming for Data Analytics: Pandas library Basics

Pandas Library

pandas is an open source, BSD-licensed library providing high-performance, easy-to-use data structures and data analysis tools for the Python programming language.

Importing panda library

import pandas as pd

Convert list elements into series of elements ranging from 10—50

import numpy as np

import pandas as pd

my_data=[10,20,30,40,50]

pd.Series(data=my_data)           

# convert element lists into series of elements, which have index from 0—5

Convert dictionary values into series of elements ranging from 10—50

import numpy as np

import pandas as pd

d={'a':10,'b':20,'c':30,'d':40}     

#dictionary keys act as index and values with every key act as series values

pd.Series(d)

Addition of two series

import numpy as np

import pandas as pd

ser1=pd.Series([1,2,3,4],['USA','Germany','Japan','USSR’])  

#create series from 1—4 with index as country names

ser2=pd.Series([1,2,5,4],['USA','Germany','Italy','USSR’])

ser1+ser2 

#addition of 2 series; values get added where index match but if same index is not available in either of series, it generates NaN value

Generate 5x4 table filled with random values having labels for rows and columns (similar to dataset) and pick a particular columns for processing.

from numpy.random import randn

import pandas as pd

np.random.seed(101)

df=pd.DataFrame(randn(5,4),['A','B','C','D','E'],['W','X','Y','Z’]) 

#generate random number for 5 rows and 4 columns, row labels: A—E, columns labels: W—Z. 

df

df[‘W’]

df[[‘W’,’Z’]]

Modifying datasets using panda library

df.drop('W',axis=1,inplace=True)                          

#Droping selected column from dataset; inplace=True, permanently accepts modification

df.loc['A']                          # rows are also series; fetch particular row from dataset having index ‘A’

df.iloc[3]                           # fetch 3^rd row from dataset

df.loc[['A','C'],['X','Z’]]                  #fetch a subset of data from given dataset; [[row][column]]

df > 0                                 # all those locations in dataset which are less than threshold is taken False

df[df>0]                            #all those positions in datasets which are less than threshold is taken as NaN

df=pd.DataFrame(randn(5,4),['A','B','C','D','E'],['W','X','Y','Z’]) 

df[df['W']>0]                   

#Select the row where True is available in W column and fetch other elements in same row

df[df['W']>0][['X','Y']]                  

# fetch out desired frame of X & Y from dataset, for those rows where value is more than 0 in ‘W’ column;

df.reset_index()   #assign natural index

df.set_index(‘Z’)    #set ‘Z’ column as index value

Drop missing elements from dataset

import pandas as pd

d={'A':[1,2,np.NaN], 'B':[1,np.NaN,np.NaN],'C':[1,2,3]}    

# np.NaN is the missing element in DataFrame

df=pd.DataFrame(d)

df.dropna()                                     #pandas would drop any row with missing value

df.dropna(axis=1)                          #drop column with NULL value

df.dropna(thresh=2)      

#thresh is the variable for threshold which says that it would not drop row until unless it has atleast 2 NonNA values else it would drop row.

Filling suitable value instead of NaN in dataset

df.fillna(value='FILL VALUE')                      #NaN is replaced by value=FILL VALUE

df['A'].fillna(value=df['A'].mean())    

#Select column "A" and fill the missing value with mean value of the column A

Calculating mean and standard deviation on values in given dataset

data = {'Company':['GOOG','GOOG','MSFT','MSFT','FB','FB'],        #create temporary dataset

'Person':['Sam','Charlie','Amy','Vanessa','Carl','Sarah'],

'Sales':[200,120,340,124,243,350]}

df=pd.DataFrame(data)               #connect data as dataframe which have row and column label

df

comp=df.groupby("Company").mean() 
# data in dataset is grouped by label “Company” i.e. elements with similar company names are grouped together and after grouping data, aggregate ‘mean’ function is applied on its value; mean could not be applied on second column "person name" due to string type

comp

comp1=df.groupby("Company")              #grouping done using label name “Company”

comp1.std()                                                  #apply standard deviation on grouped data

Fetching particular data from dataset after applying aggregate function

data = {'Company':['GOOG','GOOG','MSFT','MSFT','FB','FB'],        #create temporary dataset

'Person':['Sam','Charlie','Amy','Vanessa','Carl','Sarah'],

'Sales':[200,120,340,124,243,350]}

df=pd.DataFrame(data)               #connect data as dataframe which have row and column label

df.groupby("Company").sum().loc["FB"]              

#group data by ‘company’ label, apply sum function such that all data of same company gets added and then fetch Company “FB” value after summation

Finding maximum value in each label in dataset

data = {'Company':['GOOG','GOOG','MSFT','MSFT','FB','FB'],        #create temporary dataset

'Person':['Sam','Charlie','Amy','Vanessa','Carl','Sarah'],

'Sales':[200,120,340,124,243,350]}

df=pd.DataFrame(data)               #connect data as dataframe which have row and column label

df.groupby("Company").max()  

#group dataset based on ‘company’ label and pick maximum value in each label

Find unique value and number of occurrence of values in dataset

df = pd.DataFrame({'col1':[1,2,3,4],'col2':[444,555,666,444],'col3':['abc','def','ghi','xyz’]})

# col1, col2 & col3 are column labels, each column have their own values

df['col2'].unique()           #fetches the unique values available in column ‘col2’

df['col2'].value_counts()             

# count number of occurance of every value in column and display its count

Save Data frame into CSV file or Excel file

df.to_csv('example.csv',index=False)

df.to_excel('Excel_Sample.xlsx',sheet_name='Sheet1')

Hope you have enjoyed doing basic Data Munging with Pandas Libraries. 

Stay Calm and learn. Please post your comments below :-) 

Python Programming for Data Analytics: Matplotlib library Basics

Matplotlib Tool

Matplotlib is a plotting library for the Python programming language and its numerical mathematics extension NumPy. It provides an object-oriented API for embedding plots into applications using general-purpose GUI toolkits like TKinter, wxPython, Qt, or GTK+.

Source: Wikipedia

Importing matplotlib and numpy to plot line graph

import matplotlib.pyplot as plt

%matplotlib inline

import numpy as np

x = np.linspace(0, 5, 11)               #generate 1D array starting from 0—5, total 11 elements

y = x ** 2                                         # x²

plt.plot(x, y, 'r’)                              # 'r' is the color red

plt.xlabel('X Axis Title Here')                      #label x asis

plt.ylabel('Y Axis Title Here')                      # label y axis

plt.title('String Title Here')                         # give title to plot

plt.show()                                                     # show plot

Create subplot in single plot (like MATLAB)

plt.subplot(1,2,1)                          # 1 row, 2 columns, 1^st figure in row

plt.plot(x, y, 'r--')                            # r—red color, -- for dashed graph lines

plt.subplot(1,2,2)                          # 1 row, 2 columns, 2^nd figure in row

plt.plot(y, x, 'g*-');                         # g—green color, *- for graph lines

Use subplot functions to create MATLAB style line color

import matplotlib.pyplot as plt

%matplotlib inline

import numpy as np

x = np.linspace(0, 5, 11)

fig, ax = plt.subplots()                              #generate 2 objects, fig and ax

ax.plot(x, x**2, 'b.-')                               # blue line with dots

ax.plot(x, x**3, 'g--')                              # green dashed line

ax.set_xlabel('x axis’)

ax.set_ylabel('y axis')

ax.set_title(' plot title')

Plot various types of plots (scatter, step, bar and fill) in same figure

import numpy as np

import matplotlib.pyplot as plt

%matplotlib inline

n = np.array([0,1,2,3,4,5])

xx = np.linspace(-0.75, 1., 100)

fig, axes = plt.subplots(1, 4, figsize=(12,3))        #create figure of dimension 12x3 inch, with 4 subplots

axes[0].scatter(xx, xx + 0.25*np.random.randn(len(xx)))

axes[0].set_title("scatter")

axes[1].step(n, n**2, lw=2)                       #lw=line width of 2

axes[1].set_title("step")

axes[2].bar(n, n**2, align="center", width=0.5, alpha=0.5)     #alpha – transparency level

axes[2].set_title("bar")

axes[3].fill_between(xx, xx**2, xx**3, color="green", alpha=0.5);

axes[3].set_title("fill_between");

Saving plot into file format

fig.savefig("filename.png")

fig.savefig("filename.png", dpi=200)

Hope you have enjoyed working with Matplotlib library and plotted beautiful graphs of your data :-)

In next post, I would come up with another popular and important library used for Data Analytics purpose. 

Keep learning.Keep Growing :-) Please post your comments below !

Python Programming for Data Analytics: NumPy Library Basics

NumPy Library

NumPy is the core library for scientific computing in Python. It provides a high-performance multidimensional array object, and tools for working with these arrays.

Create 2-D array using numpy from list

import numpy as np

mylist=[[1,2,3],[4,5,6],[7,8,9]]

np.array(mylist)

Create 1-D and 2-D array using random values

np.random.rand(5)             #uniform distribution of random values

np.random.rand(4,4)          # create 2-D array of random values

np.random.randn(4)        #std. normal distribution, centered around 0

Fetch Max on Min value from array

arr1=np.random.randint(0,100,10)    #10 random integer elements

arr1.max()    or arr1.min()

Playing with the data using numpy library

arr=np.arange(0,11)                            #create array of 10 elements between 1—10

arr[:5]                                                 #fetch 1^st 5 elements of array arr

arr[5:]                                                  #fetch elements from 5^th positions till last elements

arr[:2]=100                                        # replace the 1^st two elements of array with value 100

arr=np.arange(0,25)

arr.reshape(5,5)                                             #reshape method which reshapes 1-D array into 2-D array

arr_2d=np.array([[1,2,3],[4,5,6],[7,8,9]])     #create 2-D array

arr_2d[:2,1:]                                              # select elements in a particular row and column in 2D array

arr=np.arange(0,11)

arr > 5                                                        #return True for elements position which is more than 5

arr[arr<5]                        

# return True for elements position which is more than 5 and pick array values for True positions

arr+arr                                                       # element by element array addition

mat = np.arange(1,26).reshape(5,5)    #generate 1D array of 15 elements and convert it into 2D array

mat.std()                                     #finding standard deviation on elements

mat.sum(axis=0)                        #summing elements column wise in given array mat

Hope you had enjoyed working with these basics of NumPy library. in next post, I would come up with another popular library used in Data Analytics.

Happy Learning! Please post your comments below :-)