Build home mini PC with Cubietruck and Linux

Four months ago, I decided to build a home mini PC. Its main purpose is to act as NAS server, but it should also support additional light services too.

Home mini PC requirements:

  • ARM board (low power consumption and very good performance)
  • SATA 3.5″ hard disk(much more stable than 2.5″)
  • GNU/Linux (as always)

Home mini PC services:

  1. NAS server (1st level requirement)
  2. Light server-side music player (2nd level requirement)
  3. Light BitTorrent client (2nd level requirement)
  4. No need for X window (not a requirement)
  5. Apache HTTP server (3rd level requirement)
  6. Light wiki server (3rd level requirement)
  7. Light database (3rd level requirement)

Candidates:

  • Raspberry Pi: Unbelievable community support and hundreds of applied examples, but has limited RAM and mainly does not support SATA.
  • BeagleBone Black: Stable and much more Open Hardware than Raspberry Pi, but much less features (CPU, RAM, …)
  • Cubietruck (aka Cubieboard 3): Much more features than all the above, but with much smaller community. Specifications summary here.

As you understand, my requirements are not very light for a small board :)
So, I decided to use Cubietruck (CT), which is the strongest board, but getting the risk of small community (as I am not a hardware or Linux expert).

I received my Cubietruck a few weeks ago!!!

cubietruck_details

Final hardware:

  • Cubietruck
  • Cubieboard HDD Addon (to support SATA 3.5″, which needs external power input 12V)
  • Power Adaptor 5V 3A (for CT)
  • Power Adaptor 12V 4A (for HDD addon)
  • Western Digital Red 3TB
  • SD Card 16 GB

ATTENION: You have to use different power adapters to power Cubietruck and HDD Addon. Output USB connection of HDD Addon (which work as power adapters) cannot supply enough Amperes to your Cubietruck, when are needed. Extra details here.

Final Software:

Final view…

cubietruck hdd sata details

Cubietruck with HDD SATA and all cables

Details…

cubietruck hdd addon base

Cubietruck and HDD Addon base

In next blog post, I will try to write about Debian and any extra hints I found.
If you need extra details please leave a comment to answer (or to include this info in next post).

Regards,
Adrianos Dadis.

Real Democracy requires Free Software

Posted in GNU/Linux, Personal | Tagged , , , | Leave a comment

Storm event processor – GC log file per worker

In the last three months, I am working with a new team building a product for Big Data analytics on Telecom domain.

Storm event processor is one of the main frameworks we use and it is really great. You can read more details on its official documentation (which has been improved).

Storm uses Workers to do your job, where each of them is a single JVM and is administrated internally by Storm (start, restart if no responsive, move Worker to another node of cluster, etc.). For a single job you can run many Workers on your cluster (Storm decides how to distribute your Workers in cluster nodes). As “node” I mean a running OS, either running on VM or on a physical machine.

The tricky point here is that all Workers in a node read the same configuration file (STORM_HOME/conf/storm.yaml) even they are running/processing a different kind of job. Additionally, there is a single parameter (worker.childopts) in this file, which is used for all Workers (of the same node) to initialize theirs JVMs (how to set JVM Options).

As we want to know how GC performs in each worker we need to monitor GC log of each Worker/JVM.

As I said, the problem is that as all Workers, in a node, read the same parameter from the same configuration file in order to initialize theirs JVMs, so it is not trivial to use a different GC logging file for each Worker/JVM.

Fortunately, Storm developers have expose a “variable” that solves this problem. This variable is named “ID” and it is unique for each Worker on each node (same Worker ID could exist in different nodes).

For Workers JVM Options, we use this entry in our “storm.yaml” file:

worker.childopts: "-Xmx1024m -XX:MaxPermSize=256m -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -verbose:gc -Xloggc:/opt/storm/logs/gc-storm-worker-%ID%.log"

Be aware, that you have to add “%” before and after “ID” string (in order to be identified as an internal Storm variable).

Additionally, for Supervisor JVM Options (one process on each node), we use this entry in our “storm.yaml” file:

supervisor.childopts: "-Xmx512m -XX:MaxPermSize=256m -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -verbose:gc -Xloggc:/opt/storm/logs/gc-storm-supervisor.log"

I have also included a kind of memory settings (“-Xmx” and “-XX:MaxPermSize”) too, but it is just an example.

Please keep in mind that Storm requires Oracle Hotspot JDK 6 (JDK 7/8 is not yet supported). This is a strong drawback, but we hope it will be fixed soon.

Hope it helps,
Adrianos Dadis.

Democracy Requires Free Software

Posted in Big Data, Java, Software Development | Tagged , | 1 Comment

Set WildFly binding address and shutdown using CLI

It’s very easy to bind WildFly on a hostname/IP just using command line parameters.
I have a simple GNU/Linux box that I use it to play with various things, one of them is WildFly.

I start WildFly listening on a specific IP using this commands:

$> cd /opt/wildfly/wildfly-8.0.0.Beta1/bin
$> ./standalone.sh -c standalone-full.xml -b=192.168.1.10 -bmanagement=192.168.1.10  

As you can see, I use my IP in 2 parameters/points. The first one (-b) is the regular address that server listens to serve traffic and the second one (-bmanagement) is the management address. The first parameter (-b) is the old known parameter that is used on JBoss 4.x too.

In order to shutdown server, I use the new CLI interface of JBoss/WildFly:

$> cd /opt/wildfly/wildfly-8.0.0.Beta1/bin
$> ./jboss-cli.sh --connect controller=192.168.1.10 command=:shutdown  

JBoss/WildFly CLI is really useful tool, as you can use it to create/change/view various resources on server (Datasource, JMS destinations, etc.) and deploy/undeploy applications.
JBoss/WildFly CLI is very similar to WebLogic Scripting Tool (WLST), but JBoss CLI is Open Source and the knowledge is on public space and does not belong to a company or to a closed group. This is the vital virtue of Open Source Software and Free Software, to learn people to share and innovate on the same time.

Hope it helps,
Adrianos Dadis.

Democracy Requires Free Software

Posted in Administration, JBoss | Tagged , | Leave a comment

Add Apache Camel and Spring as jboss modules in WildFly

These days I am playing with Wildfly and Apache Camel and Spring.

As Panagiotis suggests, a simple way to communicate between EARs / WARs is using direct-vm component of Camel. There are many ways to achieve this with or without Camel. Camel works like a charm in WildFly without any need for extra configurations. Camel is great!!!

In order to avoid pack all required JARs of Spring and Camel with my applications, I create two modules, using the great JBoss Modules framework (which is already used by WildFly). Then I can reference these two frameworks, without the need to pack all these JARs inside my applications (EAR/WAR).

Create Spring module

  • Go to WildFly home dir: $> cd /home/torun/jboss/wildfly/wildfly-8.0.0.Beta1
  • Create Spring module directory structure:
    • $> mkdir -p modules/org/springframework/3.2.5.RELEASE
  • Inside this new directory, create module.xml file with this content:
<module xmlns="urn:jboss:module:1.3" name="org.springframework" slot="3.2.5.RELEASE">
  <resources>
    <resource-root path="aopalliance-1.0.jar"/>
    <resource-root path="aspectjrt-1.7.4.jar"/>
    <resource-root path="aspectjtools-1.7.4.jar"/>
    <resource-root path="aspectjweaver-1.7.4.jar"/>
    <resource-root path="org.aspectj.matcher-1.7.4.jar"/>
    <resource-root path="spring-aop-3.2.5.RELEASE.jar"/>
    <resource-root path="spring-aspects-3.2.5.RELEASE.jar"/>
    <resource-root path="spring-beans-3.2.5.RELEASE.jar"/>
    <resource-root path="spring-context-3.2.5.RELEASE.jar"/>
    <resource-root path="spring-context-support-3.2.5.RELEASE.jar"/>
    <resource-root path="spring-core-3.2.5.RELEASE.jar"/>
    <resource-root path="spring-expression-3.2.5.RELEASE.jar"/>
    <resource-root path="spring-jdbc-3.2.5.RELEASE.jar"/>
    <resource-root path="spring-orm-3.2.5.RELEASE.jar"/>
    <resource-root path="spring-oxm-3.2.5.RELEASE.jar"/>
    <resource-root path="spring-tx-3.2.5.RELEASE.jar"/>
    <resource-root path="spring-web-3.2.5.RELEASE.jar"/>
    <resource-root path="spring-webmvc-3.2.5.RELEASE.jar"/>
    <resource-root path="spring-webmvc-portlet-3.2.5.RELEASE.jar"/>
  </resources>

  <dependencies>
    <module name="javaee.api"/>
    <module name="org.apache.commons.logging"/>
    <module name="org.jboss.vfs"/>
    <module name="org.hibernate"/>
    <module name="javax.el.api" export="true"/>
    <module name="com.sun.xml.bind" export="true"/>
  </dependencies>
</module>
  • Then add all the JARs mentioned as “resource-root” inside this new directory
  • You are DONE with Spring module!!!
  • Now you can reference spring module using the next line in your “jboss-deployment-structure.xml”, from your EAR/WAR:
    • <module name=”org.springframework” slot=”3.2.5.RELEASE”/>

Create Camel module

  • Create Camel module directory structure:
    • $> mkdir -p modules/org/apache/camel/2.12.1
  • Inside this new directory, create module.xml file with this content:
<module xmlns="urn:jboss:module:1.3" name="org.apache.camel" slot="2.12.1">
  <resources>
    <resource-root path="camel-core-2.12.1.jar"/>
    <resource-root path="camel-spring-2.12.1.jar"/>
    <resource-root path="jaxb-impl-2.2.6.jar"/>
  </resources>
  <dependencies>
    <module name="org.springframework" slot="3.2.5.RELEASE" />
    <module name="org.slf4j"/>
    <module name="javax.xml.bind.api"/>
    <module name="javax.api"/>
    <module name="sun.jdk" />
  </dependencies>
</module>
  • Then add all the JARs mentioned as “resource-root” inside this new directory
  • You are DONE with Camel module!!!
  • Now you can reference camel module using the next line in your “jboss-deployment-structure.xml”, from your EAR/WAR:
    • <module name=”org.apache.camel” slot=”2.12.1″ />

You may cut out a few jar dependencies from Spring or Camel module, but these are just my current settings and I know it works :)

One more important note. While I was trying to found the correct JARs for these modules, I met a few exceptions… So, if you forget any JAR, then you may see any of these exceptions:

Caused by: java.lang.NoClassDefFoundError: sun/misc/Unsafe
    at org.apache.camel.com.googlecode.concurrentlinkedhashmap.ConcurrentHashMapV8.getUnsafe(ConcurrentHashMapV8.java:4136) [camel-core-2.12.1.jar:2.12.1]
    at org.apache.camel.com.googlecode.concurrentlinkedhashmap.ConcurrentHashMapV8.<clinit>(ConcurrentHashMapV8.java:4101) [camel-core-2.12.1.jar:2.12.1]
    at org.apache.camel.com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap.<init>(ConcurrentLinkedHashMap.java:221) [camel-core-2.12.1.jar:2.12.1]
    at org.apache.camel.com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap.<init>(ConcurrentLinkedHashMap.java:104) [camel-core-2.12.1.jar:2.12.1]
    at org.apache.camel.com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Builder.build(ConcurrentLinkedHashMap.java:1634) [camel-core-2.12.1.jar:2.12.1]
    at org.apache.camel.util.LRUCache.<init>(LRUCache.java:83) [camel-core-2.12.1.jar:2.12.1]
    at org.apache.camel.util.LRUSoftCache.<init>(LRUSoftCache.java:68) [camel-core-2.12.1.jar:2.12.1]
    at org.apache.camel.impl.EndpointRegistry.<init>(EndpointRegistry.java:39) [camel-core-2.12.1.jar:2.12.1]
    at org.apache.camel.impl.DefaultCamelContext.<init>(DefaultCamelContext.java:234) [camel-core-2.12.1.jar:2.12.1]
    at org.apache.camel.spring.SpringCamelContext.<init>(SpringCamelContext.java:67) [camel-spring-2.12.1.jar:2.12.1]
    at org.apache.camel.spring.CamelContextFactoryBean.newCamelContext(CamelContextFactoryBean.java:356) [camel-spring-2.12.1.jar:2.12.1]
    at org.apache.camel.spring.CamelContextFactoryBean.createContext(CamelContextFactoryBean.java:350) [camel-spring-2.12.1.jar:2.12.1]
    at org.apache.camel.spring.CamelContextFactoryBean.getContext(CamelContextFactoryBean.java:361) [camel-spring-2.12.1.jar:2.12.1]
    at org.apache.camel.spring.CamelContextFactoryBean.getContext(CamelContextFactoryBean.java:80) [camel-spring-2.12.1.jar:2.12.1]
    at org.apache.camel.core.xml.AbstractCamelContextFactoryBean.getContext(AbstractCamelContextFactoryBean.java:518) [camel-spring-2.12.1.jar:2.12.1]
    at org.apache.camel.core.xml.AbstractCamelContextFactoryBean.afterPropertiesSet(AbstractCamelContextFactoryBean.java:160) [camel-spring-2.12.1.jar:2.12.1]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1571) [spring-beans-3.2.5.RELEASE.jar:3.2.5.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1509) [spring-beans-3.2.5.RELEASE.jar:3.2.5.RELEASE]
    ... 23 more
Caused by: java.lang.ClassNotFoundException: sun.misc.Unsafe from [Module "org.apache.camel:2.12.1" from local module loader @1a6e5d5 (finder: local module finder @3b3402 (roots: /home/torun/jboss/wildfly/wildfly-8.0.0.Beta1/modules,/home/torun/jboss/wildfly/wildfly-8.0.0.Beta1/modules/system/layers/base))]
    at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:197) [jboss-modules.jar:1.3.0.Final]
    ...

OR

Caused by: java.lang.NoClassDefFoundError: org/w3c/dom/Node
    at java.lang.Class.getDeclaredConstructors0(Native Method) [rt.jar:1.7.0_40]
    at java.lang.Class.privateGetDeclaredConstructors(Class.java:2483) [rt.jar:1.7.0_40]
    at java.lang.Class.getConstructor0(Class.java:2793) [rt.jar:1.7.0_40]
    at java.lang.Class.getDeclaredConstructor(Class.java:2043) [rt.jar:1.7.0_40]
    at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:105)
    at org.springframework.beans.factory.xml.DefaultNamespaceHandlerResolver.resolve(DefaultNamespaceHandlerResolver.java:129)
    ... 29 more
Caused by: java.lang.ClassNotFoundException: org.w3c.dom.Node from [Module "org.apache.camel:2.12.1" from local module loader @1a6e5d5 (finder: local module finder @3b3402 (roots: /home/torun/jboss/wildfly/wildfly-8.0.0.Beta1/modules,/home/torun/jboss/wildfly/wildfly-8.0.0.Beta1/modules/system/layers/base))]
    at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:197) [jboss-modules.jar:1.3.0.Final]
    ...

OR

Caused by: java.lang.NoClassDefFoundError: javax/xml/bind/JAXBException
    at java.lang.Class.getDeclaredConstructors0(Native Method) [rt.jar:1.7.0_40]
    at java.lang.Class.privateGetDeclaredConstructors(Class.java:2483) [rt.jar:1.7.0_40]
    at java.lang.Class.getConstructor0(Class.java:2793) [rt.jar:1.7.0_40]
    at java.lang.Class.getDeclaredConstructor(Class.java:2043) [rt.jar:1.7.0_40]
    at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:105)
    at org.springframework.beans.factory.xml.DefaultNamespaceHandlerResolver.resolve(DefaultNamespaceHandlerResolver.java:129)
    ... 29 more
Caused by: java.lang.ClassNotFoundException: javax.xml.bind.JAXBException from [Module "org.apache.camel:2.12.1" from local module loader @1a6e5d5 (finder: local module finder @3b3402 (roots: /home/torun/jboss/wildfly/wildfly-8.0.0.Beta1/modules,/home/torun/jboss/wildfly/wildfly-8.0.0.Beta1/modules/system/layers/base))]
    at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:197) [jboss-modules.jar:1.3.0.Final]
    ...

Hope it helps,
Adrianos Dadis.

Democracy Requires Free Software

Posted in Java, Java EE, JBoss, Software Development | Tagged , , , | Leave a comment

Java bytecode verification is always REQUIRED

Many Java performance tuning articles propose to disable bytecode verification when running a Java program (like a Java application server or web container like Tomcat).
This is WRONG and you must NOT apply it on your installations.

OK then, but what is bytecode verification in Java?
The full information is at JVM Specification. In short, it is the procedure to check that the program is type-safe in all program points.

In order to run your program faster, many optimization guides/articles recommend to use one of the following parameters:

  • -Xverify:none
  • -noverify

You must NOT use any of the above parameters, as they may lead you to security problems!!!

It is highly recommended to remove all the above parameters from your startup parameters. If you do so, then the default value “-Xverify:remote” becomes active, which is an acceptable solution.
Alternatively, you can use parameter “-Xverify:all“, to apply full bytecode verification.

In case you need to investigate the problem in depth, please check CERT advisory “Do not disable bytecode verification“.

Regards,
Adrianos Dadis.

Democracy requires Free Software

Posted in Administration, Java, Java EE | Tagged , , , , , , , | Leave a comment