Author Archives: admin

MILK – A New Language for Memory Management

It was always known that memory management was a challenge in the traditional systems where a few data points are in the play slowing down the process; this challenge becomes insurmountable when one deals with Big Data. MIT has recently announced a new programming language – MILK, which increases the speed of the common algorithms by 4x. This is believed to help in memory management and especially when we have Big Data playing a prominent part in the technologies of today.

The major reason for the clog in the performance is the need to retrieve data from the main memory of today’s chip and having to do this repeatedly slows down execution. Since memory management works on the principle of locality which assumes a program not only requires a particular portion of data stored in distant memory but may also require the neighboring portions as well. Thus slowing the process considerably. In Big Data however this does not play, it is usually the need to retrieve few datapoints scattered over huge data sets.

The new language MILK developed by MIT’s Computer Science and Artificial Intelligence Laboratory helps application developers to manage memory more efficiently. Application developer, using MILK, adds a few lines of code to any instructions written to retrieve a few data points scattered across huge data sources. MILK compilers then resolve to manage the memory management accordingly.

According to MIT, the programs written in MILK were four times faster than those written in existing languages, and ran on some common algorithm and promised that this will only get better as lot more research is envisaged to fine tune the technology further.

For more on this please read this article by Katherine Noyes @ http://www.infoworld.com/article/3120689/big-data/new- programming-language- promises-a-4x-speed- boost-on- big-data.html

Tags: MILK, Big Data, Memory Management

Container versus Virtual Machines – Few Basic Differences

Container technology is making rapid strides with some recent reports putting the adoption rate as high as 5x times in one year. But surprisingly not many are aware of Container technology and how exactly it is different from virtual machines.

In the last decade, due to the rapid advancement in server technology the bare metal applications i.e. applications programmed to work on specific hardware in a system was unable to fully utilize the processing power and capacity of servers, leaving resources underutilized necessitating the birth of virtual machines.

Virtual machines are designed by the emulation of particular hardware system by running software on top of a physical server. Many VMs with different operating systems may be run on the same physical server with a unique OS inside each VM. So a Linux VM may easily co-exist with a Unix VM. The benefits of this technology was, it ushered in a system of consolidation of applications on to a single system instead of the prevalent single application/system accruing massive cost benefits and helping in optimal utilization of processing power and capacity of physical systems.

Container is a complete package consisting of entire runtime environment: an application, plus all its dependencies, its libraries, binaries and configuration files. This way the differences in OS distribution and the underlying infrastructure may be overcome to make the application portable on to any platform or OS.

In short, three VMs with three separate OSes running on top of it may sit on a single physical system with a hypervisor, whereas three containerized application runs on a single OS in a single server.

The size of a VM and a Container varies widely, sometime a Container may be in megabytes, the VMs because of its operation systems will usually be in gigabytes. So obviously a single server can host more containers than it can host VMs. And for this same reason a container can boot almost instantly whereas a VM may take several minutes.

But as several containers share the same OS on the kernel level, it cannot provide the same degree of isolation as a VM does and so is lot more vulnerable to security threats.

Who moved my Apps?

Progressive Web Apps (PWA) are making a silent march onto users mobile with space constricting for native mobile apps.

Even with a worldwide download of 102,062m apps in 2016, is mobile web making a comeback? Probably yes, if you believe this story by BI Intelligence Apps and Platforms briefing.

Surprisingly, the foremost reason given by users for not using mobile apps is, they like using website better (53%), security concerns (31%) and too much space on the mobile (28%). Reinforcing this feeling is another study which tells, almost 20% of user’s drop-off from the stage of download to using the app.

Taking advantage of this is Google’s PWA (Progressive Web Apps) open-source initiative. Announced in 2015, PWA is a sort of hybrid of app and web page, offering look and feel of an app on web. This ensures the user immediately starts using the app instead of spending time/bandwidth on downloading and installing the app.

This might have stemmed from the fact that Google wants users to spend more of their time on mobile web than on app, even though the fact that Google receives more web searches from mobile than desktop. The fact that many apps have metamorphosed themselves into a platform has not helped Google.

So, what does this mean for business apps, most welcome this shift. Even though time spent on mobile has increased, business have conceded that there are limitations on number of brands a user can have on his mobile, so the message is “Mobile – Yes, Mobile app – No”. PWA provides a more open field for companies to displace brands and switch loyalties.

From user’s perspective, when would a user prefer a mobile app to PWA and vice-versa? An ecommerce site or an online grocery store which he uses frequently, almost daily, can be a native mobile app as compared to an airline check-in app, where the user uses it a few times a month.

So next time when companies sit to plan their mobile strategy, they would want to consider PWA rather than being limited to building just mobile apps.

10 years of Amazon Web Services, lessons learnt so far.

Writing on the occasion of Amazon web services completing 10 years this March, Werner Vogels, CTO of Amazon.com, shared few important lessons Amazon learnt on its way to build a secure, scalable and reliable services. Amazon Web Services, a cloud services platform which today is the foundation of IT infrastructure of majority of business, be it a startup, SME or a government agency, is according to a reputed mainstream magazine “the piece of infrastructure which has enabled the current tech boom”.

 Vogels writes that there is no dearth of opportunities to learn new lessons when you are in the process of building a service which is adding million active customers per month, who in turn may serve hundreds of millions of their own customers. Below are the 10 lessons he shortlists for the 10 years completed in the hope that this may be of use to others.
Lesson 1 – Build evolvable systems
Build flexible software which are open to receive new components in its architecture as often you need to scale as AWS understood, it would not be possible to upgrade the system during maintenance, since enterprises are dependent on its service 24×7.
Lesson 2 – Expect the unexpected
Failure is inevitable especially over time and when systems are made to scale where least probability of error will become realistic. One needs to build systems that envelop failure, as AWS developed a skill of managing the “Blast radius “of the failure so the overall health of the system is maintained.
Lesson 3 – Primitives and not Frameworks
AWS quickly realized that they need to be very agile since the way their customers would like to use their services was changing constantly. This made AWS to offer customers a wide collection of templates and tools to choose from instead of forcing them to use a single framework.
Lesson 4 – Automation is key
Operating a software service as AWS does especially at that scale requires a different mindset as it is very important to assure customer of reliability, performance, and scalability. This AWS achieves by automating the management process as much as possible.
Lesson 5 – APIs are forever
As AWS is a API- centric business where customers build their applications and systems using AWS ‘s APIs, AWS has to get the APIs right the first time around as changing that subsequently would have impacted their customer’s businesses.
Lesson 6 – Know your resource usage
Good data about the cost of operations and service especially when at very high volumes is very necessary to settle on a robust charging model, at AWS, the charging model accommodates for all dimensions of resource usage to make it a sustainable business model.
Lesson 7 – Building security from ground-up
Securing your customers business should be of critical priority as it is for AWS, the security is very much integrated at the design of the service model for AWS.
Lesson 8 – Encryption is the key
Empowering customers to have full access to their data is of very critical importance. At AWS they make it easier for customers to use encryption and its mechanism is integrated at a design phase itself.
Lesson 9 – The importance of network
In light of the fact that AWS supports varying workloads, it has developed a unique skill to build data center layout and design so that customers need to meet varying workloads is achieved by providing a flexible network infrastructure
Lesson 10 – No gate keepers
Very Important aspect of AWS is there are no gate keepers to tell what one can do or not do, which keeps it very open to innovation, as a result it has different customers building platforms on AWS themselves to meet very specific needs of a business vertical.
Coming from AWS, this probably becomes A Gospel for companies looking to build a cloud-based service. For more on this, please visit http://www.allthingsdistributed.com/2016/03/10-lessons- from- 10-years- of-aws.html
Keywords – Amazon Web Services, AWS, cloud-services, Werner Vogels

Microsoft Azure releases new competencies for Container Technology

At the recently concluded Docker 16 event at Seattle WA, Microsoft, which in the last couple of years has been working closely with container technology leaders like Docker and Mesosphere, demonstrated new capabilities to enhance the running and managing of containers in cloud.

Mark Russinovich – CTO, Microsoft Azure in a blog published shortly after demonstrating these new capabilities, dealt in depth on how Microsoft is spearheading innovations to container technology both in Windows and Linux ecosystems to help companies to build or scale applications at great speed irrespective of their platform. Sharing the journey of Microsoft Azure with container technology in the last one year, Mark Russinovich writes how Microsoft with its partnership with Dockers introduced Windows server containers to broadest set up of developers.

With enterprises betting big on cloud computing, containers are exploding into application development, as with this technology, one big lacunae of cloud computing namely portability owing to the proprietary nature of some public clouds are squarely addressed. With this, an application can be moved from one cloud infrastructure to another without much hassle, since container technology abstracts applications into virtual containers. The benefits in terms of cost savings and business agility the container technology delivers has encouraged 76% of 1400 global companies plan to use container technology in 2016.

Taking account of the priorities of the enterprise, Microsoft Azure has released new competencies in running and managing containers in cloud;

 Docker Datacenter to manage hybrid, container-based application spread across Azure and for the first time, Azure Stack on premise

 Operations Management Suite (OMS) managing containers across Microsoft Azure and Azure Stack on-premises.

 Preview of Azure Container Service

 SQL Server running on Linux, in a Docker container

With enterprises using container technology for complex scenarios, Microsoft is trying to entice them with its hybrid capabilities and commitment to open technology.

Want to know more? Read this blog from Mark Russinovich – https://azure.microsoft.com/en-us/blog/microsoft-brings- container-innovation- to-the- enterprise-at- dockercon-2016/

Keywords – Microsoft Azure, Docker, Mesosphere, Container Technology, Docker Datacenter, Operations Management Suite (OMS)

Access and manipulate your database within Microsoft Excel (MySQL for Excel 1.3.6 has been released)

MySQL for Excel is an application plug-in enabling data analysts to very easily access and manipulate MySQL data within Microsoft Excel. It enables you to directly work with a MySQL database from within Microsoft Excel so you can easily do tasks such as:

* Importing MySQL Data into Excel

* Exporting Excel data directly into MySQL to a new or existing table

* Editing MySQL data directly within Excel

MySQL for Excel is installed using the MySQL Installer for Windows.

The MySQL Installer comes in 2 versions

- Full (150 MB) which includes a complete set of MySQL products with

their binaries included in the download.

- Web (1.5 MB – a network install) which will just pull the MySQL for

Excel over the web and install it when run.

You can download MySQL Installer from our official Downloads page at https://urldefense.proofpoint.com/v2/url?u=http-3A__dev.mysql.com_downloads_installer_&d=CwIFaQ&c=IV_clAzoPDE253xZdHuilRgztyh_RiV3wUrLrDQYWSI&r=b3rm4E2nx7aGlNTnMaOjlzWmTXv_sJTyehHR_J-0TT4&m=MNKqrM4wOBaJFL1sLBDcd_51hRakxL9J36IXe6W5Xao&s=biWElijgNfFAEB5zTpXOEzKFv8VnHeOK4NTfmjeP7HY&e=

The MySQL for Excel product can also be downloaded by using the product standalone installer found at this linkhttps://urldefense.proofpoint.com/v2/url?u=http-3A__dev.mysql.com_downloads_windows_excel_&d=CwIFaQ&c=IV_clAzoPDE253xZdHuilRgztyh_RiV3wUrLrDQYWSI&r=b3rm4E2nx7aGlNTnMaOjlzWmTXv_sJTyehHR_J-0TT4&m=MNKqrM4wOBaJFL1sLBDcd_51hRakxL9J36IXe6W5Xao&s=QTrLZCERkFq1f5vnvv3zVm6OZv8_YD2DWfcRpfUcmEM&e=

Changes in MySQL for Excel 1.3.6 (2016-01-07)

* Functionality Added or Changed

* Bugs Fixed

Functionality Added or Changed

* A new Generate an INSERT statement for each data row

option was added to the Advanced Options dialog for

Append and Export Data operations. This is disabled by

default. A single INSERT statement is generated that

inserts all affected rows, unless this option is enabled.

When checked, an INSERT statement for each data row being

appended or exported is generated in the resulting SQL

query sent to the MySQL server. When unchecked, a single

INSERT statement is generated as a bulk data operation,

which performs better than multiple INSERT statements.

While row-by-row inserts are slower, for debugging errors

it could help determine which row causes an error, since

the MySQL Server will process them row by row.

* Added column properties to the Export Data dialog to

support the definition of Auto Increment and Default

Value for new MySQL tables where the data will be

exported.

Added a context menu to declare a numeric field to be

Unsigned or to have Zero-Fill, accessible by

right-clicking the Data Type combo box.

Added the TIMESTAMP data type to the full list of valid

data types, as it was not being shown before. Added help

tool tips to fields in the Export Data dialog that

explain in detail how the column properties work in a

MySQL database.

Bugs Fixed

* The Pivot Table feature was updated to work with Excel

2016. (Bug #22387425)

* Closing the plugin via the Close button in the panel

would not toggle the MySQL for Excel checkbox in the Data

ribbon. (Bug #22374026, Bug #79655)

* Fixed the content detector for the active Excel cell

after importing data and after loading database objects

when a database is opened. Now, the data options are not

enabled until a table is selected. (Bug #22372915)

* Append and Export operations would not append/export

boolean data. (Bug #22138966)

* The MySQL for Excel Refresh All function would throw an

unhandled exception. (Bug #22110627, Bug #78941)

* With Office 2007, data would not auto-save after closing

Excel. (Bug #22081313, Bug #78910)

* Numeric data stored in text columns is now correctly

formatted by Excel as text, and not as the “General”

format that automatically formats cells as numbers when

the data is stored in a text object. This fixes the issue

where text columns were formatted as numbers, which

stripped leading zeros. (Bug #20905284, Bug #76738)

* Changed settings for containers and forms to let all UI

components automatically scale on different DPI settings.

Also removed the restriction to resize the MySQL for

Excel’s pane, so in case a different DPI setting is used

and the fonts look too big, the pane can be resized to

see contents as best as possible. (Bug #18099454, Bug

#71389)

 

Machine Learning – A Trend or a Fad

Big Data and Machine Learning are the two buzz words in the industry today. The way companies in various sectors are adopting data science approach to improve their top line and bottom line numbers has been phenomenal.

With the huge popularity that the data science has achieved in the recent past, it’s high time to take a step back and understand how different organizations fared when they employed ML or Advanced techniques to improve their business operations. Towards the later half of 2015, there was an article published by Gartner where they showed the hype cycle of the emerging technologies.

http://www.gartner.com/newsroom/id/3114217

What is interesting over here is to find that “Machine Learning” falls in the “Peak of Inflated Expectations” in the Hype Cycle which shows that there have been successes (often owing to the initial publicity) accompanied by scores of failures. This may look scary at first since Big Data and Advanced Analytics are supposed to lift up the business. We definitely wanted this to be in the “Plateau of Productivity” on the Hype Cycle. Technologies falling in this plateau region are the ones that are adopted by all the mainstream businesses and have positive ROI. It’s promising to see that both “Machine Learning” and “Advanced Analytics” are expected to reach the plateau in 2-5 years’ time.

Gartner’s Hype Cycle shows that benefits of Machine Learning and Advanced Analytics have not yet reached its true potential. This also shows that it is the best time to be in the space. As shown through the article, the companies who can adopt these technologies in this time and space will get the benefits of it within the next five years.

Neverware – Revive your old Windows PC as a Chromebook

Do you have an old Windows laptop lying around that you refuse to touch because of how slow it runs Windows (Vista/7) today? If so, a new product called Neverware CloudReady may be able to help. CloudReady is FREE for Home use.

Neverware’s CloudReady will help you repurpose that laptop into a Chromebook that your young child can use. They have a list of models below that are certified to run CloudReady without any issues here: http://go.neverware.com/certifiedmodels. 

The Verge has a great article below on how Neverware is helping resuscitate these old PCs for schools and other educational institutions.
http://www.theverge.com/2016/2/17/11030406/neverware-google-chromebook-chromium-os-education-microsoft

Have you tried CloudReady already? If so, chime in with your comments and experience. I am planning on trying it out soon myself!

SSIS DATA PROFILING

SQL Server Integration Services offers a useful tool to analyze data before you bring it into your Data Warehouse.  The Profile Task will store the analysis in an XML file, which you can view using the Data Profile Viewer.  Before we review how to use the Profile Task, let’s take a look at the eight types of profiles that can be generated by this control.

  • Candidate Key Profile Request
    • Use this profile to identify the columns that make up a key in your data
  • Column Length Distribution Profile Request
    • This profile reports the distinct lengths of string values in selected columns and the percentage of rows in the table that each length represents. Use this profile to identify invalid data, for example a United States state code column with more than two characters.
  • Column Null Ratio Profile Request
    • As the name implies, this profile will report the percentage of null values in selected columns. Use this profile to identify unexpectedly high ratios of null values.
  • Column Pattern Profile Request
    • Reports a set of regular expressions that cover the specified percentage of values in a string column. Use this profile to identify invalid strings in your data, such as Zip Code/Postal Code that do not fit a specific format.
  • Column Statistics Profile Request
    • Reports statistics such as minimum, maximum, average and standard deviation for numeric columns, and minimum and maximum for datetime columns. Use this profile to look for out of range values, like a column of historical dates with a maximum date in the future.
  • Column Value Distribution Profile Request
    • This profile reports all the distinct values in selected columns and the percentage of rows in the table that each value represents. It can also report values that represent more than a specified percentage in the table.  This profile can help you identify problems in your data such as an incorrect number of distinct values in a column.  For example, it can tell you if you have more than 50 distinct values in a column that contains United States state codes.
  • Functional Dependency Profile Request
    • The Functional Dependency Profile reports the extent to which the values in one column (the dependent column) depend on the values in another column or set of columns (the determinant column). This profile can also help you identify problems in your data, such as values that are not valid. For example, you profile the dependency between a column of United States Zip Codes and a column of states in the United States. The same Zip Code should always have the same state, but the profile discovers violations of this dependency.
  • Value Inclusion Profile Request
    • The Value Inclusion Profile computes the overlap in the values between two columns or sets of columns. This profile can also determine whether a column or set of columns is appropriate to serve as a foreign key between the selected tables. This profile can also help you identify problems in your data such as values that are not valid. For example, you profile the ProductID column of a Sales table and discover that the column contains values that are not found in the ProductID column of the Products table.

SQL SERVER DATABASE ENCRYPTION STATUS

Securing your data is very important and database encryption is just part of that landscape. Taking advantage SQL Server database encryption can be a daunting task but once it is configured it is also important to monitor it.

This query is intended to allow the database administrator to gain information about the status of encryption on their systems. This script can be very handy for audits or to ensure that your databases are in an a state you expect them to be in.

For those databases in your environment that require Transparent Data Encryption (TDE) this script will be invaluable for monitoring the encryption states of your databases.

 

SELECT
[db].name,
[db].is_encrypted,
[dek].encryption_state,
CASE [dek].encryption_state
WHEN 0 THEN 'Not Encrypted'
WHEN 1 THEN 'Unencrypted'
WHEN 2 THEN 'Encryption in progress'
WHEN 3 THEN 'Encrypted'
WHEN 4 THEN 'Key change in progress'
WHEN 5 THEN 'Decryption in progress'
WHEN 6 THEN 'Protection change in progress '
ELSE 'Not Encrypted'
END AS 'Desc'
FROM
sys.dm_database_encryption_keys [dek]
RIGHT JOIN
sys.databases [db] ON [dek].database_id = [db].database_id