Distributed computing instance usage with starcluster Ipython parallel plugin
I am using starcluster with Ipython plugin . When i run a Kmeans clustering from Ipython notebook with load balance mode. Its always the Master with 100% CPU usage constantly. And the other EC2 instances never takes the load.
I tried with large datasets and 20 nodes . The result is same all the Load is on the Master. I tried direct view with the node001 but even then the master is having all the load.
Am i configuring any thing wrong. Do i need to make the Disable Queue true in the config?. How can i distribute the load on all the Instances.
I cannot able to post the image because i am a starter in stackoverflow. Here is the htop for the master and node001
Link for the image: https://drive.google.com/file/d/0BzfdmaY9JuagT0lmX29xY1RiUmM/view?usp=sharing
Any help would be highly appreciated.
Regards, Tej.
My Template file :
[cluster iptemplate]
KEYNAME = ********
CLUSTER_SIZE = 2
CLUSTER_USER = ipuser
CLUSTER_SHELL = bash
REGION = us-west-2
NODE_IMAGE_ID = ami-04bedf34
NODE_INSTANCE_TYPE = m3.medium
#DISABLE_QUEUE = True
PLUGINS = pypackages,ipcluster
[plugin ipcluster]
SETUP_CLASS = starcluster.plugins.ipcluster.IPCluster
ENABLE_NOTEBOOK = True
NOTEBOOK_PASSWD = *****
[plugin ipclusterstop]
SETUP_CLASS = starcluster.plugins.ipcluster.IPClusterStop
[plugin ipclusterrestart]
SETUP_CLASS = starcluster.plugins.ipcluster.IPClusterRestartEngines
[plugin pypackages]
setup_class = starcluster.plugins.pypkginstaller.PyPkgInstaller
packages = scikit-learn, psutil, scikit-image, numpy, pyzmq
[plugin opencvinstaller]
setup_class = ubuntu.PackageInstaller
pkg_to_install = cmake
[plugin pkginstaller]
SETUP_CLASS = starcluster.plugins.pkginstaller.PackageInstaller
# list of apt-get installable packages
PACKAGES = python-mysqldb
Code:
from IPython import parallel
clients = parallel.Client()
rc = clients.load_balanced_view()
def clustering(X_digits):
from sklearn.cluster import KMeans
kmeans = KMeans(20)
mu_digits = kmeans.fit(X_digits).cluster_centers_
return mu_digits
rc.block = True
rc.apply(clustering, X_digits)
I am just learning about starcluster/ipython myself, but this gist seems to jive with @thomas-k's comment, namely you need to structure your code to be able to be passed to a load-balanced map:
https://gist.github.com/pprett/3989337
cv = KFold(X.shape[0], K, shuffle=True, random_state=0)
# instantiate the tasks - K times the number of grid cells
# FIXME use generator to limit memory consumption or do fancy
# indexing in _parallel_grid_search.
tasks = [(i, k, estimator, params, X[train], y[train], X[test], y[test])
for i, params in enumerate(grid) for k, (train, test)
in enumerate(cv)]
# distribute tasks on ipcluster
rc = parallel.Client()
lview = rc.load_balanced_view()
results = lview.map(_parallel_grid_search, tasks)