Hadoop India Summit 2011

Online event registration for Hadoop India Summit 2011

Security Checklist before moving into IaaS type of Cloud

The following security questions should help you make proper decision while looking for an IaaS or a PaaS type of Cloud Computing Vendor.

General

What security is built to protect the physical access to computing premises?
Is the storage co-located as the computing resources? If no then what security measure are taken to protect the physical access to the storage resources.
Who have access to host machines and storage (both physical and networked)? How is the access to host machines managed?
• What happens when an authorized person leaves the organization or is transferred to different a role? How soon the privileges are revoked? What happens if that person leaves an open session?
• Are there different levels of access rights? Who manages these rights? Is this activity logged?
Are all the activities that administrators perform on any host servers logged? How are the logs protected (time-stamped / signed)? Can the administrator delete (or modify) the log after performing any actions?
• What kind of access do the administrators of the host machine have on the guest machines and guest session? Is it possible for the logged in administrator to perform memory scan of the guest machine?
Can a virtual machine do hard disk scanning and read any left-over data from the previous virtual machine session?

Data Security

• Can the host machine (or an administrator) perform scan of virtual machine's storage (hard disk?)
What are privacy policies about data stored in the cloud storage?
What are privacy policies about customer information and application?
Once a customer has closed the account with the provider does the provider still retain the customer data?
• If the cloud computing vendor shuts down the business for any reason, how will the customer be able to acquire all his data?
Is it possible to use application level encryption to protect the data in the cloud?
Can the data for multiple customers be physically co-located on the same storage rack? In case of government agency demanding to cease all physical storage rack belonging to certain customer, how do you guarantee that the business of other customers is not affected? If you cannot guarantee then what is your policy to repair the business loss because of the above incident?
What is the security model for accessing data stored in the cloud? This security model should be verified with a security expert.
Is it possible to define access protection on the stored data?
Is it possible to log the storage access activities?
What facilities are provided to keep business records of Information Lifecycle Management (ILM). The ILM business records are is defined in terms of the following:
• Creation and Receipt
• Distribution
• Use
• Maintenance
• Disposition
Are there any tools provided to users for monitoring data access log?

Virtual Environment and Network Security

• How are the Virtual machines protected from each other? (i.e., protection against neighbor attack).
• Can virtual machines for different customers discover each other? Can they communicate with each other over the private network? Can a Virtual machine monitor all the network traffic for other virtual machines on the same network (or the same host machine)?
What kind of protection is built to stop virtual machines from doing IP spoofing?
Are there any tools provided to users for monitoring the protection of their virtual machines.
What measures are taken to protect the kernel level security on the host machine? Can a malicious program gain the access to hypervisor and monitor the activities of virtual machines?
Can the host computer do sniffing on the network traffic of Virtual machines?
• Is the data between computer processor (and memory) and disk storage (primary drive) local or network based? Generally the data transfer between computer RAM and hard-disk is not protected. If it is network based, it will raise many more security questions.
• How is the data transfer between computer processes and the persistent storage (SAN) (similar to Amazon’s EBS) protected?
How is the XEN hypervisor protected against attacks? As the paper Subverting the Xen Hypervisor shows, it is possible to insert malicious c code to gain access to DMA and insert malicious code into XEN hypervisor. The malicious code can perform memory scan of virtual machines.

In future I will post about SLA, Terms & Conditions and policy related questions that must be considered before you can choose a cloud computing provider.

Private Cloud vs Internal Cloud

Most people today use the terms Private cloud and Internal cloud interchangeably. I see them differently.

I consider an "internal cloud" to a cloud computing setup on on-premise data-center of an Organization. In this case the organization owns the infrastructure and uses the cloud computing platform for the optimal usage of its infrastructure in its data-center.
The motivation behind an internal cloud is data-center consolidation using virtualization and dynamic provisioning technologies to allow
a) optimal usage of infrastructure resources
b) near real-time response to varying need of application capacity (elastic scalability).
An internal cloud is almost always in form of IaaS.

A Private Cloud does not necessarily refer to an internal data-center consolidation. It may refer to set of services provided exclusively to a customer. In case of private cloud a vendor will provide a cloud computing platform to a customer that has been tailored to custom needs of the customer. This customer has exclusive access to the cloud. A private cloud is generally not multi-tenants in the sense that the resources will not be shared with other customers.
A private cloud may also refer to cloud computing platform that has been tailored to an exclusive community of users (such as law firms, universities etc.). In this type of setup the cloud can be multi-tenant.

Private clouds generally offer higher level of compliance and governance than a public cloud.

A private cloud can in the form of IaaS, PaaS or even SaaS.

Private cloud and Internal cloud share some of the characteristics. The resources are not shared among multiple organizations. Provide higher degree of control over data governance and regulation compliance.

Cloud Computing is not Housing Market Bubble

I read somewhere that NIST says that cloud computing bubble will burst like housing market did.

I don't agree with the analogy. Housing market bubble was artificially created. There was no science or a natural evolution behind the crazy Housing Market bubble. It was set to be burst.

The same principle can not be applied to cloud computing. Even after cutting all the hype around it, the cloud computing is a part of computing industry evolution (if not revolution) to better utilize the resources that we have.

It is like providing water or electricity. Not every company has its own Electricity generation plant. It is not economical and is also waste of resources. Cloud Computing is similar. Not every company need to build/own a data center.

Predictions about Sun's products after Oracle aquires it

Prediction 1
MySQL : Initially 'the database expert' company buying MySQL will look promising. However, soon this company (Oracle) will start offering MySQL enterprise edition and force customers to buy only Enterprise edition.
This will create lot of noise in the FSF world (as it will violate the GPL license). Since, open source community of MySQL is huge, there will be fork of MySQL in the open source world.
Actual MySQL will eventually be dead because of lack of support from Oracle. However its forked re-incarnation will continue to live and flourish (different name).
Summary : MySQL will live and keep evolving under different name

Prediction 2:
Glassfish: Oracle will now own three major Java Enterprise Servers (OC4J, Weblogic and Glassfish). Oracle is currently consolidating their OC4J and Weblogic product lines. It is phasing out OC4J and forcing customers to migrate the Weblogic.
As far as Glassfish is concerned, Oracle will fire most engineers from Glassfish development team. The Glassfish will still live because Oracle will still need to develop an RI that demonstrates the latest and greatest of JavaEE. However, it will no longer be a production quality software. The open source community of Glassfish is not big enough to keep it alive so Glassfish will not live more than 2 years.
Eventually Apache Gernimo will be used as latest and greatest RI of JavaEE.
Summary : Glassfish may live for few years (may be 2) but in form of non-usable, non-production quality RI

Prediction 3:
Solaris: Will continue to live and evolve but more in form of Proprietary Operating System

Prediction 4:
VirtualBox: Will be forked. Oracle will also continue the development but its basic version will be available as Free software (not an open source software). Eventually Oracle's version and Open source version will differ significantly.

Prediction 5:
Sparc Chip : I don't know. Can go in any direction

Prediction 6:
Open Office : It will live and flourish. However, Oracle's contribution will be mostly limited to the development for StarOffice rather than for OpenOffice.
There will be better marketing and sale of StarOffice. You will see more products like SharePoint that integrate enterprise Document management into StarOffice. StarOffice will shine. (So will OpenOffice since more people will be using it)

Prediction 7:
Java: The innovation in Java will slowdown significantly will be business driven rather than technological driven. Oracle will not share the idea of Sun that Java should be everywhere (big server to smallest possible devices). As a result more and more people will migrate the Microsoft's .Net or some other platform.

Prediction 8:
Java mobile (and FX) : Innovation will cease. Because of lack of innovation, the platform will become deprecated and eventually die.

Prediction 9:
Overall Java Adaption: Will decrease. It will mostly be limited to Server side Platform (and SOA)

Prediction 10:
Netbeans : Will continue to live. Oracle will migrate its JDeveloper to use NetBeans platform.

Maven vs Ant+Ivy

Choosing a right build management tool is important for any software development process.

The most important thing a build management tool should provide is improved productivity of the development team. Developers have to deal with the build management tool on day to day basis. If the build management tool is not right for the team, it will create frustration in the team members which may result in not so good position for the project.

One tool can not fit every team and every project. The right tool must be chosen based on its audience and the project requirements.

I think most developers from Open Source and Java community prefer command line based tools such as ANT or Maven. These tools are flexible (easy to enhance), scriptable (and hence can be automated). These avoid dependency on any IDE and can be used from a remote terminal.

In some cases, however, depending on the type of project and team, IDE can be a better build tool. It offers click and build features. Debugging is, I think one of the biggest feature of using an IDE. This will be a natural choice for people from Microsoft background. If the team has most of such developers, don't try to push command line based tools on them. The migration should be gradual.

I prefer to use non-ui based build tools.

I had been using ANT for past many years and simply loved it. ANT scripts are very simple to develop and use, and develop. It is also very simple to understand and follow its execution. It has very good documentation and consistent way of doing things. Despite its simplicity ANT is very flexible.

However, everytime I start a new project, I had to create a new script (not fun). I had to download all the libraries that I will need in the project, put them in some folder. I also need to understand the dependencies between these different libraries. Now I need to modify ANT script to manually manage the dependencies and blah blah blah....too much pain.

Doing the above was fun for first few times but when I had to do it again and again, I started to dislike it.

Moreover, all IDEs, despite their claims, don't have good integration with ANT. People will say, hey Eclipse, NetBeans, IntelliJ etc. can import ant based projects and can build using ANT script. But none of these IDE can understand the library dependency defined in the ANT script. As a result, I can't trust IDE to resolve classes and code completion. What a shame.

Last year, I was introduced to the wonderland of Maven. After some initial learning curve, I became comfortable with it and I really started to like it.

Maven gives you library dependency management (the biggest pain in ANT). It can automatically create default project directory structure and initial pom.xml based on the type of project you create. It has concept of project build life-cycle and even without modifying the default script, you can build and release the project.

You can have multi-module project and create dependencies between them. When doing a build on multi-module project, it can automatically identify the correct sequence of building module.

Wow Maven is a wonderful tool. It does so much for you. I don't need to download libraries that I use (at least most of them). I can simply define dependency in the maven script and all the required libraries (including transitive dependencies) are downloaded during Maven build.

Eclipse and Netbeans also support Maven and understand library dependencies defined in the Maven script.

And then...

As I worked more with Maven, I realized that Maven is not flexible at all. Although, it offers so many features but does not allow me to anything unconventionally.

For example everything part of the script has to fit within the Maven project lifecycle. Let's say if I want to start http tunnel for monitoring the traffic. How do I do that? Starting an http tunnel is not part of project life cycle. May be it is...some people will argument...but some things don't have to be perfect....they have to be simpler and flexible...At the end of the day getting things done and done faster is all that matters.

I can not use any library outside of Maven repository. If some library is not available in any Maven repository, Maven guide tells me to put that library in local Maven repository manually. That is not cool. In this case I can not distribute my code among my team members. Assuming that there is no local central repository, everybody in my team will have to add the library manually. I don't like that.

As far as writing a script is concerned, I complained about ANT before. However, I should have also mentioned that the ANT script notations and structure are so simple and consistent that I can almost always write the initial ANT script (compile, run, clean javadoc deploy etc) without referring to any documentation.

I can not say the same thing for Maven. Even if all I need to do is enable Java 5 compatibility in my Project, I have to refer to it's documentation (who remembers name of all the plugins, group id, version numbers blah blah blah).

(I should also say....what the hack....Java 7 is due pretty soon and Maven assumes that the new projects are going to use Java 1.3 ???????..very annoying)

As other people described about Maven, it is like a black magic. It magically does too much stuff for you...How....you don't know....You have no idea what it did underneath but it was done. May be it is good thing for some people but I found it very annoying. When it does not work, it is very difficult to follow through its execution and discover problems.

Maven tries to force you to do things like the Maven way, while ANT allows you to do things your way.

With Maven, you are like a lion in a cage. You are enjoying tasty food everyday but are not free to go anywhere. With ANT you are like a lion in a jungle. You are free; you can do anything but you have to hunt for your own food and sometimes you may not even find one.

Each project is different. The environment and the constraints are always different in each project and there is no single solution to all of them.

Each project may have different needs and may not fit into perfect life cycle and the project structure of Maven.

Ant provides the flexibility. With Ant you really need are plugins that provide Maven features without Maven restrictions.

One of the biggest feature of Maven IMHO is dependency management. Apache Ivy adds this feature to ANT.

Apache Ivy is very similar to Maven in dependency management. You define project's library dependencies in ivy configuration file, add its reference in the ANT script and that's pretty much it.

Apache Ivy can also understand Maven repository.

The ANT + Ivy allows you to mix repository based libraries and non-repository based libraries. I can put non-repository based libraries in a special folder in the project and reference it in ANT.

The good thing about this solution is that it provides you dependencies management while enjoying the freedom of ANT as opposed to caze of Maven.

It gives you controlled, predictable, transparent solution with no surprises. And that is exactly what I want.

However you should also remember that there are still certain nice features about Maven that are not provided by ANT +Ivy solution (such as project life cycle management, integration with central repository+version control system, ability to publish/release projects just to name a few).

These features are, though easily implementable, have to be developed in-house.

One more thing, Eclipse has good integration with Ivy. Netbeans lacks such integration. Well..I think we will see this soon.

Because of the flexibility ANT ( + Ivy) will be my first choice for the time being.

I would also like to see a plugins/features can auto-create IDE independent ANT script and project directory structure for well known projects (should be pluggable so that people can contribute more project types)...similar to artifact-type in Maven.

VMWare accuses Microsoft for playing dirty tricks trying to prevent customers from using virtualized machines.
http://www.vmware.com/solutions/whitepapers/msoft_licensing_wp.html

Let us see how long will the Microsoft be able to deny the evolution in hardware Virtualization.